Introduce a rewritten picojpeg that is able to operate statefully and
rasterize into an internal line ringbuffer, emitting a line of pixels
at a time to the caller. This is the JPEG equivalent of the lws
PNG decoder.
JPEG is based around 8- or 16- line height MCU blocks, depending on
the chroma coding, mandating a corresponding internal line buffer
requirement.
Example total heap requirement for various kinds of 600px width jpeg
decoding:
Grayscale: 6.5KB
RGB 4:4:4: 16.4KB
RGB 4:2:2v: 16.4KB
RGB 4:4:2h: 31KB
RGB 4:4:0: 31KB
No other allocations occur during decode.
Stateful stream parsing means decode can be paused for lack of input
at any time and resumed seamlessly when more input becomes available.
The adapted upng has a very compact fully-stateful lws-aligned
implementation already.
Adapt it to also be buildable and operable standalone, and to
understand gzip headers.
Provide some apis to inflate gzip simply reusing opaque inflator
contexts from upng.
Provide an api test that inflates gzip files from stdin -> stdout
Add a rewritten version of upng that decodes statefully line by line, and so
does not require a bitmap buffer for the output. This compares to original
upng approach that needs heap allocations for the input, the whole output
and intermediate allocations.
Instead of buffers for input, decompression and output, it only allocates
2 x lines of RGBA pixels (ie, a few KB), and 32KB of decompressed data for
backward references in the decoder, and decodes as needed into the 2-line
buffer to produce line rasterized results. For a 600px width PNG, this is
just 40KB heap for the duration.
This adds apis that enable usage of compressed backtraces in heap
instrumentation.
A decompressor tool is also provided that emits a textual
call stack suitable for use with addr2line.
This allows handling of the unusal but foreseeable situation were
the client provides a zero length output buffer, for example, if
the input buffer was also zero length.
Leave the http minilex as it is, and add an improved version in lib/misc
- get a list of strings from stdin and emit C header to stdout
- support ambiguous terminals (xxx and xxx-something) correctly regardless
of introduction order
- add generic parser in lib/misc
minilex doesn't build as part of lws since it's only needed by developers,
there's a one-line build documented at a comment at the top of
lib/misc/minilex.c
For plugins that handle PROTOCOL_INIT, we have to iterate any PLUGINS_BUILTIN
plugins as if we had just discovered and loaded them from plugin files, so
they bind to vhosts appropriately.
Add some private helpers to keep the guts from duplication in plat.
This adds an optional feature LEJP_FLAG_FEAT_OBJECT_INDEXES that changes
lejp to treat { } items as indexable in ctx->i[] / ctx->ipos, since they
also can take commas.
This may break existing uses so it requires the default-off feature flag to
enable it. The flags context field is zeroed by lejp_construct(), so any
flags should be set on ctx->flags after alling that.
There's also a flag LEJP_FLAG_LATEST available as an alias to enable any
desirable but not-backwards-compatible behaviour, including this.
Add the info to the README and adapt the unit test to do it both with and
without the FEAT_OBJECT_INDEXES flag.
This is a NOP for existing usecases.
At the moment the only implemented transport for serialized SS is wsi, it's
typically used with Unix Domain Sockets, but it also works over tcp the
same.
It generalizes the interface between serialized chunks and the
transport, separately for client and proxy. The wsi transport is migrated
to use the new transport ops structs.
It will then be possible to "bring your own transport", so long as it is
reliable, and in-order, both for proxy and client / sspc.
We also adapt minimal-secure-streams-binance to build the -client variant
via SS proxy as well.
LWS_ONLY_SSPC is added so libwebsockets can be produced with just sspc
client support even for tiny targets.
A new embedded minimal example for rpi pico is also provided that
demonstrates using Serialized SS over a UART to an SS proxy, to implement
the SS Binance example on the pico, even though it has no networking itself.
This provides very memory-efficient CBOR stream parsing
and writing.
The parser converts pieces of CBOR into callbacks that define
the structure and collate string and blobs into buffer chunks
for extensible and easy access.
It is fragementation-safe and does not need all the CBOR in
the same place at one time, chunks of CBOR are parsed and
discarded as provided.
It does not allocate and just needs a few hundred bytes of
stack for even huge CBOR objects. Huge strings and blobs
are handled without needing memory to hold them atomically.
Includes ./minimal-examples/api-tests/api-test-lecp that
unit tests it against 82 official example CBORs and
26 additional test vectors from COSE (just checking the CBOR
parsing).
The writing apis allow printf style semantics with a variety
of CBOR-aware %-formats. The apis write into a context that
manages output buffer usage, if the output buffer fills,
then the apis return with an AGAIN code that lets you issue
and reset the output buffer and repeat the api all to issue
more output. The subsequent calls can occur much later or
from a different function context, so this is perfect for
WRITEABLE-mediated output from the network parts of lws.
See ./READMEs/README.cbor-lecp.md
Also prioritize LD_LIBRARY_PATH check for plugins first
Iterate through paths in LD_LIBRARY_PATH in order
Warn on failed plugins init but continue protocol init
This is a huge patch that should be a global NOP.
For unix type platforms it enables -Wconversion to issue warnings (-> error)
for all automatic casts that seem less than ideal but are normally concealed
by the toolchain.
This is things like passing an int to a size_t argument. Once enabled, I
went through all args on my default build (which build most things) and
tried to make the removed default cast explicit.
With that approach it neither change nor bloat the code, since it compiles
to whatever it was doing before, just with the casts made explicit... in a
few cases I changed some length args from int to size_t but largely left
the causes alone.
From now on, new code that is relying on less than ideal casting
will complain and nudge me to improve it by warnings.
On an error path, we might not have a valid spa / lejp to destroy.
In that case, don't hurl ourselves into calling a
function pointer that can be NULL for the destroy cb.
This adds some new objects and helpers for keeping and logging
info on grouped allocations, a group is, eg, SS handles or client
wsis.
Allocated objects get a context-unique "tag" string intended to replace
%p / wsi pointers etc. Pointers quickly become confusing when
allocations are freed and reused, the tag string won't repeat
until you produce 2^64 objects in a context.
In addition the tag string documents the object group, with prefixes
like "wsi-" or "vh-" and contain object-specific additional
information like the vhost name, address / port or the role of the wsi.
At creation time the lws code can use a format string and args
to add whatever group-specific info makes sense, eg, a wsi bound
to a secure stream can also append the guid of the secure stream,
it's copied into the new object tag and so is still available
cleanly after the stream is destroyed if the wsi outlives it.
C++ APIs wrapping SS client
These are intended to provide an experimental protocol-independent c++
api even more abstracted than secure streams, along the lines of
"wget -Omyfile https://example.com/thing"
WIP
This is complicated by the fact extern on a function declaration implies
visibility... we have to make LWS_EXTERN empty when building static.
And, setting target_compile_definitions() doesn't work inside macros,
so it has to be set explicitly for the plugins.
Checking the symbol status needs nm -C -D as per
https://stackoverflow.com/questions/37934388/symbol-visibility-not-working-as-expected
after this patch, libwebsockets.a shows no symbols when checked like that and
the static-linked minimal examples only show -U for their other dynamic
imports.
In a handful of cases we use LWS_EXTERN on extern data declarations,
those then need to change to explicit extern.
Event lib support as it has been isn't scaling well, at the low level
libevent and libev headers have a namespace conflict so they can't
both be built into the same image, and at the distro level, binding
all the event libs to libwebsockets.so makes a bloaty situation for
packaging, lws will drag in all the event libs every time.
This patch implements the plan discussed here
https://github.com/warmcat/libwebsockets/issues/1980
and refactors the event lib support so they are built into isolated
plugins and bound at runtime according to what the application says
it wants to use. The event lib plugins can be packaged individually
so that only the needed sets of support are installed (perhaps none
of them if the user code is OK with the default poll() loop). And
dependent user code can mark the specific event loop plugin package
as required so pieces are added as needed.
The eventlib-foreign example is also refactored to build the selected
lib support isolated.
A readme is added detailing the changes and how to use them.
https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.event-libs.md
Move the common plugin scanning dir stuff to be based on lws_dir, which
already builds for windows. Previously this was done via dirent for unix
and libuv for windows.
Reduce the dl plat stuff to just wrap instantiation and destruction of
dynlibs, establish common code in lib/misc/dir.c for plugin scanning
itself.
Migrate the libuv windows dl stuff to windows-plugins.c, so that he's
available even if later libuv loop support becomes and event lib plugin.
Remove the existing api exports scheme for plugins, just export a const struct
now which has a fixed header type but then whatever you want afterwards depending
on the class / purpose of the plugin. Place a "class" string in the header so
there can be different kinds of plugins implying different types exported.
Make the plugin apis public and add support for filter by class string, and
per instantation / destruction callbacks so the subclassed header type can
do its thing for the plugin class. The user provides a linked-list base
for his class of plugins, so he can manage them completely separately and
in user code / user export types.
Rip out some last hangers-on from generic sessions / tables.
This is all aimed at making the plugins support general enough so it can
provide event lib plugins later.
For some patterns of JSON we return to parse at the outermost level and
meet a situation path_match is 0. In some places we're looking at things
from perspective of path_match - 1... that does not seem to cause trouble on
x86_64 but can on aarch64, which is how it got noticed.
This logically protects those accesses by checking !!path_match.
There's a good pattern that's encouraged by using lws_struct pieces, that
we have an lws_dll2 owner with an array of objects listed in it that exist
in an lwsac. And because it came from JSON, there is tending to be a
logical name for the objects.
This adds a typed helper and wrapper to scan the owner list looking for
a specific name (of a specified length, not NUL terminated) in a specific
member of the listed objects, which must be a NUL-terminated const char *.
Again this is a good pattern that's encouraged by use of lws_tokenize
to recover the name we're looking for.
So it leads to the helper that can cleanly search for a listed object of the
right name from an owner, and return the typed object pointer or NULL, from a
length-specified string.
