Teach lws how to deal with date: and retry-after:
Add quick selftest into apt-test-lws_tokenize
Expand lws_retry_sul_schedule_retry_wsi() to check for retry_after and
increase the backoff if a larger one found.
Finally, change SS h1 protocol to handle 503 + retry-after: as a
failure, and apply any increased backoff from retry-after
automatically.
Event lib support as it has been isn't scaling well, at the low level
libevent and libev headers have a namespace conflict so they can't
both be built into the same image, and at the distro level, binding
all the event libs to libwebsockets.so makes a bloaty situation for
packaging, lws will drag in all the event libs every time.
This patch implements the plan discussed here
https://github.com/warmcat/libwebsockets/issues/1980
and refactors the event lib support so they are built into isolated
plugins and bound at runtime according to what the application says
it wants to use. The event lib plugins can be packaged individually
so that only the needed sets of support are installed (perhaps none
of them if the user code is OK with the default poll() loop). And
dependent user code can mark the specific event loop plugin package
as required so pieces are added as needed.
The eventlib-foreign example is also refactored to build the selected
lib support isolated.
A readme is added detailing the changes and how to use them.
https://libwebsockets.org/git/libwebsockets/tree/READMEs/README.event-libs.md
PARENT_SCOPE needs adjusting in a few places for wolfssl to work, and
we need a second level export of USE_WOLFSSL through lib/CMakeLists.txt
Add noi/f32 Sai build for WOLFSSL + MINIMAL_EXAMPLES
Presently a vh is allocated per trust store at policy parsing-time, this
is no problem on a linux-class device or if you decide you need a dynamic
policy for functionality reasons.
However if you're in a constrained enough situation that the static policy
makes sense, in the case your trust stores do not have 100% duty cycle, ie,
are anyway always in use, the currently-unused vhosts and their x.509 stack
are sitting there taking up heap for no immediate benefit.
This patch modifies behaviour in ..._STATIC_POLICY_ONLY so that vhosts and
associated x.509 tls contexts are not instantiated until a secure stream using
them is created; they are refcounted, and when the last logical secure
stream using a vhost is destroyed, the vhost and its tls context is also
destroyed.
If another ss connection is created that wants to use the trust store, the
vhost and x.509 context is regenerated again as needed.
Currently the refcounting is by ss, it's also possible to move the refcounting
to be by connection. The choice is between the delay to generate the vh
being visisble at logical ss creation-time, or at connection-time. It's anyway
not preferable to have ss instantiated and taking up space with no associated
connection or connection attempt underway.
NB you will need to reprocess any static policies after this patch so they
conform to the trust_store changes.
For some patterns of JSON we return to parse at the outermost level and
meet a situation path_match is 0. In some places we're looking at things
from perspective of path_match - 1... that does not seem to cause trouble on
x86_64 but can on aarch64, which is how it got noticed.
This logically protects those accesses by checking !!path_match.
There's a good pattern that's encouraged by using lws_struct pieces, that
we have an lws_dll2 owner with an array of objects listed in it that exist
in an lwsac. And because it came from JSON, there is tending to be a
logical name for the objects.
This adds a typed helper and wrapper to scan the owner list looking for
a specific name (of a specified length, not NUL terminated) in a specific
member of the listed objects, which must be a NUL-terminated const char *.
Again this is a good pattern that's encouraged by use of lws_tokenize
to recover the name we're looking for.
So it leads to the helper that can cleanly search for a listed object of the
right name from an owner, and return the typed object pointer or NULL, from a
length-specified string.
- Add low level system message distibution framework
- Add support for local Secure Streams to participate using _lws_smd streamtype
- Add apit test and minimal example
- Add SS proxy support for _lws_smd
See minimal-secure-streams-smd README.md
String helpers for scanning non-NUL-delimited strings safely,
and very cheap simple string match based JSON parse for cases
that make sense for it... for more complex cases, do a full
JSON parse.
Sometimes we need to find out the substituted length before we can
allocate and actually store it. Teach strexp that if we set the
output buffer to NULL (and the output length to something big) we
are asking for the substituted length and to not produce output.
Establish a new distributed CMake architecture with CMake code related to
a source directory moving to be in the subdir in its own CMakeLists.txt.
In particular, there's now one in ./lib which calls through to ones
further down the directory tree like ./lib/plat/xxx, ./lib/roles/xxx etc.
This cuts the main CMakelists.txt from 98KB -> 33KB, about a 66% reduction,
and it's much easier to maintain sub-CMakeLists.txt that are in the same
directory as the sources they manage, and conceal all the details that that
level.
Child CMakelists.txt become responsible for:
- include_directories() definition (this is not supported by CMake
directly, it passes it back up via PARENT_SCOPE vars in helper
macros)
- Addition child CMakeLists.txt inclusion, for example toplevel ->
role -> role subdir
- Source file addition to the build
- Dependent library path resolution... this is now a private thing
in the child CMakeLists.txt, it just passes back any adaptations
to include_directories() and the LIB_LIST without filling the
parent namespace with the details
Replace the bash selftest plumbing with CTest.
To use the selftests, build with -DLWS_WITH_MINIMAL_EXAMPLES=1
and `CTEST_OUTPUT_ON_FAILURE=1 make test` or just
`make test`.
To disable tests that require internet access, also give
-DLWS_CTEST_INTERNET_AVAILABLE=0
Remove travis and appveyor scripts on master.
Remove travis and appveyor decals on README.md.
lejp_parse() return type is an int... but in the function, the temp
for it is a char. This leads to badness that is currently worked
around by casting the return through a signed char type.
But that leads to more badness since if there's >127 bytes of buffer
left after the end of the JSON object, we misreport it.
Bite the bullet and fix the temp type, and fix up all the guys
who were working around it at the caller return casting to use the
resulting straight int.
If you are using this api, remove any casting you may have cut-
and-pasted like this
n = (int)(signed char)lejp_parse(...);
... to just be like this...
n = lejp_parse(...);
Fix pthreads detection in the minimal examples and add it where needed.
Fix unistd.h include to be conditional on not WIN32
With this, -DLWS_WITH_MINIMAL_EXAMPLES=1 is happy and warning-free
on windows.
Continue with lws_struct, add sqlite support for one
level of lws_dll2_t lists of structs serialization and
deserialization, plus the matching api-test.
There are some minor public api type improvements rather than cast everywhere
inside lws and user code to work around them... these changed from int to
size_t
- lws_buflist_use_segment() return
- lws_tokenize_t .len and .token_len
- lws_tokenize_cstr() length
- lws_get_peer_simple() namelen
- lws_get_peer_simple_fd() namelen, int fd -> lws_sockfd_type fd
- lws_write_numeric_address() len
- lws_sa46_write_numeric_address() len
These changes are typically a NOP for user code
The %.*s is very handy to print strings where you have a length, but
there is no NUL termination. It's quite widely supported but at least
one vendor RTOS toolchain doesn't have it.
Since there aren't that many uses of it yet, audit all uses and
convert to a new helper lws_strnncpy() which uses the smaller of
two lengths.
Generic lws_system IPv4 DHCP client
- netif and route control via lib/plat apis
- linux plat pieces implemented
- Uses raw ip socket for UDP broadcast and rx
- security-aware
- usual stuff plus up to 4 x dns server
If it's enabled for build, it holds the system
state at DHCP until at least one registered interface
has acquired a set of IP / mask / router / DNS server
It uses PF_PACKET which is Linux-only atm. But those
areas are isolated into plat code.
TODOs
- lease timing and reacquire
- plat pieces for other than Linux
Old certs were getting near the end of their life and we switched the
server to use letsencrypt. The root and intermediate needed for the
mbedtls case changed accordingly
This adds the option to have lws do its own dns resolution on
the event loop, without blocking. Existing implementations get
the name resolution done by the libc, which is blocking. In
the case you are opening client connections but need to carefully
manage latency, another connection opening and doing the name
resolution becomes a big problem.
Currently it supports
- ipv4 / A records
- ipv6 / AAAA records
- ipv4-over-ipv6 ::ffff:1.2.3.4 A record promotion for ipv6
- only one server supported over UDP :53
- nameserver discovery on linux, windows, freertos
It also has some nice advantages
- lws-style paranoid response parsing
- random unique tid generation to increase difficulty of poisoning
- it's really integrated with the lws event loop, it does not spawn
threads or use the libc resolver, and of course no blocking at all
- platform-specific server address capturing (from /etc/resolv.conf
on linux, windows apis on windows)
- it has LRU caching
- piggybacking (multiple requests before the first completes go on
a list on the first request, not spawn multiple requests)
- observes TTL in cache
- TTL and timeout use lws_sul timers on the event loop
- ipv6 pieces only built if cmake LWS_IPV6 enabled
wsi timeout, wsi hrtimer, sequencer timeout and vh-protocol timer
all now participate on a single sorted us list.
The whole idea of polling wakes is thrown out, poll waits ignore the
timeout field and always use infinite timeouts.
Introduce a public api that can schedule its own callback from the event
loop with us resolution (usually ms is all the platform can do).
Upgrade timeouts and sequencer timeouts to also be able to use us resolution.
Introduce a prepared fakewsi in the pt, so we don't have to allocate
one on the heap when we need it.
Directly handle vh-protocol timer if LWS_MAX_SMP == 1
Adapt service loops and event libs to use microsecond waits
internally, for hrtimer and sequencer. Reduce granularity
according to platform / event lib wait.
Add a helper so there's a single place to extend it.
