Add initial support for defining servers using Secure Streams
policy and api semantics.
Serving h1, h2 and ws should be functional, the new minimal
example shows a combined http + SS server with an incrementing
ws message shown in the browser over tls, in around 200 lines
of user code.
NOP out anything to do with plugins, they're not currently used.
Update the docs correspondingly.
You may use separate rx or tx handlers to neatly isolate different
rx or tx state handling, for example if the connection enters some
mode where you may send a variety of possibly large things, it can
be advantageous to have different code handling each of the
different things.
This allows you to change the rx, tx and / or state handlers to
different ones suitable for the user protocol state, if it's helpful.
With upcoming SS Server support, this has another use when SS
indicates that the underlying protocol upgraded, eg, http -> ws,
you may want to change the handlers for the different sort of
payloads expected after that, according to your user protocol.
Presently a vh is allocated per trust store at policy parsing-time, this
is no problem on a linux-class device or if you decide you need a dynamic
policy for functionality reasons.
However if you're in a constrained enough situation that the static policy
makes sense, in the case your trust stores do not have 100% duty cycle, ie,
are anyway always in use, the currently-unused vhosts and their x.509 stack
are sitting there taking up heap for no immediate benefit.
This patch modifies behaviour in ..._STATIC_POLICY_ONLY so that vhosts and
associated x.509 tls contexts are not instantiated until a secure stream using
them is created; they are refcounted, and when the last logical secure
stream using a vhost is destroyed, the vhost and its tls context is also
destroyed.
If another ss connection is created that wants to use the trust store, the
vhost and x.509 context is regenerated again as needed.
Currently the refcounting is by ss, it's also possible to move the refcounting
to be by connection. The choice is between the delay to generate the vh
being visisble at logical ss creation-time, or at connection-time. It's anyway
not preferable to have ss instantiated and taking up space with no associated
connection or connection attempt underway.
NB you will need to reprocess any static policies after this patch so they
conform to the trust_store changes.
Currently we always reserve a fakewsi per pt so events that don't have a related actual
wsi, like vhost-protocol-init or vhost cert init via protocol callback can make callbacks
that look reasonable to user protocol handler code expecting a valid wsi every time.
This patch splits out stuff that user callbacks often unconditionally expect to be in
a wsi, like context pointer, vhost pointer etc into a substructure, which is composed
into struct lws at the top of it. Internal references (struct lws is opaque, so there
are only internal references) are all updated to go via the substructre, the compiler
should make that a NOP.
Helpers are added when fakewsi is used and referenced.
If not PLAT_FREERTOS, we continue to provide a full fakewsi in the pt as before,
although the helpers improve consistency by zeroing down the substructure. There is
a huge amount of user code out there over the last 10 years that did not always have
the minimal examples to follow, some of it does some unexpected things.
If it is PLAT_FREERTOS, that is a newer thing in lws and users have the benefit of
being able to follow the minimal examples' approach. For PLAT_FREERTOS we don't
reserve the fakewsi in the pt any more, saving around 800 bytes. The helpers then
create a struct lws_a (the substructure) on the stack, zero it down (but it is only
like 4 pointers) and prepare it with whatever we know like the context.
Then we cast it to a struct lws * and use it in the user protocol handler call.
In this case, the remainder of the struct lws is undefined. However the amount of
old protocol handlers that might touch things outside of the substructure in
PLAT_FREERTOS is very limited compared to legacy lws user code and the saving is
significant on constrained devices.
User handlers should not be touching everything in a wsi every time anyway, there
are several cases where there is no valid wsi to do the call with. Dereference of
things outside the substructure should only happen when the callback reason shows
there is a valid wsi bound to the activity (as in all the minimal examples).
Callbacks can ask the caller to, eg, destroy the ss handle now. But some
callback returns are handled and produced inside other helper apis, eg
lws_ss_backoff() may have to had fulfilled the callback request to destroy
the ss... therefore it has to signal to its caller, and its callers have
to check and exit their flow accordingly.
This differentiates between client connections for retry / writeable requests
and explicit lws_ss_client_connect() api calls. The former effectively uses
retry / backoff, and the latter resets the retry / backoff.
If you receive ALL_RETRIES_FAILED due to the retry policy, you can do whatever
you need to do there and call lws_ss_client_connect() to try to connect again
with a fresh, reset retry / backoff state.
- Add low level system message distibution framework
- Add support for local Secure Streams to participate using _lws_smd streamtype
- Add apit test and minimal example
- Add SS proxy support for _lws_smd
See minimal-secure-streams-smd README.md
Allow usage of ${metadata} string substitution in more policy elements
for MQTT:
- associated subscription topic in policy
- associated publish topic in policy
- associated will topic in policy
- associated will message in policy
Tested against lws-minimal-mqtt-client-multi / mosquitto
Sometimes we need to find out the substituted length before we can
allocate and actually store it. Teach strexp that if we set the
output buffer to NULL (and the output length to something big) we
are asking for the substituted length and to not produce output.
It's not safe to destroy objects inside a callback from a parent that
still has references to the object.
Formalize what the user code can indicate by its return code from the
callback functions and provide the implementations at the parents.
- LWSSSSRET_OK: no action, OK
- LWSSSSRET_DISCONNECT_ME: disconnect the underlying connection
- LWSSSSRET_DESTROY_ME: destroy the ss object
- LWSSSSRET_TX_DONT_SEND: for tx, give up the tx opportunity since nothing to send
Some streamtypes do not pass or receive payload meaningfully. Allow them
to just leave their related cb NULL. Ditto for state, although I'm not sure
how useful such a streamtype can be.
Adapt the pt sul owner list to be an array, and define two different lists,
one that acts like before and is the default for existing users, and another
that has the ability to cooperate with systemwide suspend to restrict the
interval spent suspended so that it will wake in time for the earliest
thing on this wake-suspend sul list.
Clean the api a bit and add lws_sul_cancel() that only needs the sul as the
argument.
Add a flag for client creation info to indicate that this client connection
is important enough that, eg, validity checking it to detect silently dead
connections should go on the wake-suspend sul list. That flag is exposed in
secure streams policy so it can be added to a streamtype with
"swake_validity": true
Deprecate out the old vhost timer stuff that predates sul. Add a flag
LWS_WITH_DEPRECATED_THINGS in cmake so users can get it back temporarily
before it will be removed in a v4.2.
Adapt all remaining in-tree users of it to use explicit suls.
Establish a new distributed CMake architecture with CMake code related to
a source directory moving to be in the subdir in its own CMakeLists.txt.
In particular, there's now one in ./lib which calls through to ones
further down the directory tree like ./lib/plat/xxx, ./lib/roles/xxx etc.
This cuts the main CMakelists.txt from 98KB -> 33KB, about a 66% reduction,
and it's much easier to maintain sub-CMakeLists.txt that are in the same
directory as the sources they manage, and conceal all the details that that
level.
Child CMakelists.txt become responsible for:
- include_directories() definition (this is not supported by CMake
directly, it passes it back up via PARENT_SCOPE vars in helper
macros)
- Addition child CMakeLists.txt inclusion, for example toplevel ->
role -> role subdir
- Source file addition to the build
- Dependent library path resolution... this is now a private thing
in the child CMakeLists.txt, it just passes back any adaptations
to include_directories() and the LIB_LIST without filling the
parent namespace with the details
Esp-idf has an improved but still kind of abused cmake-
based build system now.
If we see ESP_PLATFORM coming as a cmake var, we can know we
are being built from inside the esp-idf config system.
Leave the existing esp32 arrangements alone but triggered off
ESP_PLATFORM, adapt to use the cross toolchain file and
various quirks automatically.
In this way you can build lws a part of your project in a
much cleaner way.
Prepare a minimal esp32 test app for use in Sai
Adapt .sai.json to build for esp32
When most of ss-h2 was combined into ss-h1 during development, the h2 difference
about needing HTTP_FINAL to signal h2 FIN flag was accidentally dropped. In
many cases the peer can infer it, from, eg, content-length reached. But we need
to replace explicitly doing it to cover all cases.
lejp_parse() return type is an int... but in the function, the temp
for it is a char. This leads to badness that is currently worked
around by casting the return through a signed char type.
But that leads to more badness since if there's >127 bytes of buffer
left after the end of the JSON object, we misreport it.
Bite the bullet and fix the temp type, and fix up all the guys
who were working around it at the caller return casting to use the
resulting straight int.
If you are using this api, remove any casting you may have cut-
and-pasted like this
n = (int)(signed char)lejp_parse(...);
... to just be like this...
n = lejp_parse(...);
You can disconnect the stream by returning -1 from tx(). You can
give up your chance to send anything by returning 1 from tx().
Returning 0 sends `*len` amount of the provided buffer.
Returning <0 from rx() also disconnects the stream.
GCC -fanalyzer did find a real issue (unchecked malloc return)
but it mainly reported things that weren't true due to what
was happening outside of the particular compilation unit that
it could see.
In some cases devices may be too constrained to handle JSON policies but still
want to use SS apis and methodology.
This introduces an off-by-default cmake option LWS_WITH_SECURE_STREAMS_STATIC_POLICY_ONLY,
if enabled the JSON parsing part is excluded and it's assumed the user code
provides its policy as hardcoded policy structs.
There are a few automatic things that look for streamtypes that may or
may not exist now
- captive_portal_detect
- fetch_policy
- api_amazon_com_auth
logging them as notice every startup is pretty intrusive, change to info.
