A second chunk of ss / sspc handling did not get cleaned up
along with the other patch from a few weeks ago, it wrongly
treats sspc the same as ss. This can cause the wrong thing
to be zeroed down, 64-bit and 32-bit builds end up with
different victims.
This patch makes it understand the difference and treat them
accordingly, same as the main for_ss handling.
Some platforms need two sockets with AF_INET and AF_INET6 to listen to both
protocols.
This patch changes the single listen socket each vhost could previously
handle to become an lws_dll2 and adapts the related code to handle them as
a linked-list rather than as a singleton.
The next patch adapts the listen / server code to create multiple listen
wsi for vhosts listening on multiple ip protocols.
Until now although we can follow redirects, and they can promote the
protocol from h1->h2, we couldn't handle h2 wsi reuse since there are many
states in the wsi affected by being h2.
This wipes the related states in lws_wsi_reset() and follows the generic
wsi close flow before deviating into the redirect really close to the end,
ensuring we cleaned out evidence of our previous life properly.
h2->h2 redirects work properly after this.
The max number of redirects is increased from 3 -> 4 since this was seen in
the wild with www and then geographic-based redirects.
This provides a build option LWS_WITH_CONMON that lets user code recover
detailed connection stats on client connections with the LCCSCF_CONMON
flag.
In addition to latencies for dns, socket connection, tls and first protocol
response where possible, it also provides the user code an unfiltered list
of DNS responses that the client received, and the peer it actually
succeded to connect to.
There are a few build options that are trying to keep and report
various statistics
- DETAILED_LATENCY
- SERVER_STATUS
- WITH_STATS
remove all those and establish a generic rplacement, lws_metrics.
lws_metrics makes its stats available via an lws_system ops function
pointer that the user code can set.
Openmetrics export is supported, for, eg, prometheus scraping.
This is a huge patch that should be a global NOP.
For unix type platforms it enables -Wconversion to issue warnings (-> error)
for all automatic casts that seem less than ideal but are normally concealed
by the toolchain.
This is things like passing an int to a size_t argument. Once enabled, I
went through all args on my default build (which build most things) and
tried to make the removed default cast explicit.
With that approach it neither change nor bloat the code, since it compiles
to whatever it was doing before, just with the casts made explicit... in a
few cases I changed some length args from int to size_t but largely left
the causes alone.
From now on, new code that is relying on less than ideal casting
will complain and nudge me to improve it by warnings.
This adds some new objects and helpers for keeping and logging
info on grouped allocations, a group is, eg, SS handles or client
wsis.
Allocated objects get a context-unique "tag" string intended to replace
%p / wsi pointers etc. Pointers quickly become confusing when
allocations are freed and reused, the tag string won't repeat
until you produce 2^64 objects in a context.
In addition the tag string documents the object group, with prefixes
like "wsi-" or "vh-" and contain object-specific additional
information like the vhost name, address / port or the role of the wsi.
At creation time the lws code can use a format string and args
to add whatever group-specific info makes sense, eg, a wsi bound
to a secure stream can also append the guid of the secure stream,
it's copied into the new object tag and so is still available
cleanly after the stream is destroyed if the wsi outlives it.
role ops are usually only sparsely filled, there are currently 20
function pointers but several roles only fill in two. No single
role has more than 14 of the ops. On a 32/64 bit build this part
of the ops struct takes a fixed 80 / 160 bytes then.
First reduce the type of the callback reason part from uint16_t to
uint8_t, this saves 12 bytes unconditionally.
Change to a separate function pointer array with a nybble index
array, it costs 10 bytes for the index and a pointer to the
separate array, for 32-bit the cost is
2 + (4 x ops_used)
and for 64-bit
6 + (8 x ops_used)
for 2 x ops_used it means 32-bit: 10 vs 80 / 64-bit: 22 vs 160
For a typical system with h1 (9), h2 (14), listen (2), netlink (2),
pipe (1), raw_skt (3), ws (12), == 43 ops_used out of 140, it means
the .rodata for this reduced from 32-bit: 560 -> 174 (386 byte
saving) and 64-bit: 1120 -> 350 (770 byte saving)
This doesn't account for the changed function ops calling code, two
ways were tried, a preprocessor macro and explicit functions
For an x86_64 gcc 10 build with most options, release mode,
.text + .rodata
before patch: 553282
accessor macro: 552714 (568 byte saving)
accessor functions: 553674 (392 bytes worse than without patch)
therefore we went with the macros
RFC6724 defines an ipv6-centric DNS result sorting algorithm, that
takes route and source address route information for the results
given by the DNS resolution, and sorts them in order of preferability,
which defines the order they should be tried in.
If LWS_WITH_NETLINK, then lws takes care about collecting and monitoring
the interface, route and source address information, and uses it to
perform the RFC6724 sorting to re-sort the DNS before trying to make
the connections.
In the case http client doesn't get a response and closes, currently
it is confused, it reports it as a CLIENT_CONNECTION_ERROR but then
also a CLOSED_CLIENT_HTTP.
Adapt the logic so we only go that way for ws connection... not getting
the server headers means not reaching ESTABLISHED, which makes it a
CCE not a CLOSE.
Also make sure we never issue a CLOSE type callback if we issued a CCE.
For server, if the adoption of the incoming connection proceeds but then
fails early on, eg, tls alert due to hostname mismatch with cert, the
wsi close happens but it doesn't clean up the invalidated reference to
itself in the server ss object... if it became established, that's handled
by the ss protocol callback.
This patch helps the close path to understand there is a related ss object
and to clean up after itself.
Currently we always reserve a fakewsi per pt so events that don't have a related actual
wsi, like vhost-protocol-init or vhost cert init via protocol callback can make callbacks
that look reasonable to user protocol handler code expecting a valid wsi every time.
This patch splits out stuff that user callbacks often unconditionally expect to be in
a wsi, like context pointer, vhost pointer etc into a substructure, which is composed
into struct lws at the top of it. Internal references (struct lws is opaque, so there
are only internal references) are all updated to go via the substructre, the compiler
should make that a NOP.
Helpers are added when fakewsi is used and referenced.
If not PLAT_FREERTOS, we continue to provide a full fakewsi in the pt as before,
although the helpers improve consistency by zeroing down the substructure. There is
a huge amount of user code out there over the last 10 years that did not always have
the minimal examples to follow, some of it does some unexpected things.
If it is PLAT_FREERTOS, that is a newer thing in lws and users have the benefit of
being able to follow the minimal examples' approach. For PLAT_FREERTOS we don't
reserve the fakewsi in the pt any more, saving around 800 bytes. The helpers then
create a struct lws_a (the substructure) on the stack, zero it down (but it is only
like 4 pointers) and prepare it with whatever we know like the context.
Then we cast it to a struct lws * and use it in the user protocol handler call.
In this case, the remainder of the struct lws is undefined. However the amount of
old protocol handlers that might touch things outside of the substructure in
PLAT_FREERTOS is very limited compared to legacy lws user code and the saving is
significant on constrained devices.
User handlers should not be touching everything in a wsi every time anyway, there
are several cases where there is no valid wsi to do the call with. Dereference of
things outside the substructure should only happen when the callback reason shows
there is a valid wsi bound to the activity (as in all the minimal examples).
If the client connection attempt fails early, we report it will a NULL
return from the client connection api. If it fails later, perhaps after
more times around the event loop, we report it as a CONNECTION_ERROR.
This patch makes sure we don't do CONNECTION_ERROR flow if in fact we
are still in the client_connect_via_info() and in a position to report
the failure by returning NULL from there, without it under some
conditions we will do both a CONNECTION_ERROR and return NULL.
Replace the bash selftest plumbing with CTest.
To use the selftests, build with -DLWS_WITH_MINIMAL_EXAMPLES=1
and `CTEST_OUTPUT_ON_FAILURE=1 make test` or just
`make test`.
To disable tests that require internet access, also give
-DLWS_CTEST_INTERNET_AVAILABLE=0
Remove travis and appveyor scripts on master.
Remove travis and appveyor decals on README.md.
Secure Streams is an optional layer on top of lws that separates policy
like endpoint selection and tls cert validation into a device JSON
policy document.
Code that wants to open a client connection just specifies a streamtype name,
and no longer deals with details like the endpoint, the protocol (!) or anything
else other than payloads and optionally generic metadata; the JSON policy
contains all the details for each streamtype. h1, h2, ws and mqtt client
connections are supported.
Logical secure streams outlive any particular connection and supports "nailed-up"
connectivity regardless of underlying connection stability.
Adds client support for MQTT QoS0 and QoS1, compatible with AWS IoT
Supports stream binding where independent client connections to the
same endpoint can mux on a single tcp + tls connection with topic
routing managed internally.
The vfork optimized spawn, stdxxx and terminal handling in the cgi
implementation is quite mature and sophisticated, and useful for
other things unrelated to cgi. Break it out into its own public
api under LWS_WITH_SPAWN, off by default.
Expand it so the parent wsi is optional, and the role and protocol
bindings for stdxxx pipes can be set. Allow optional sul timeout
and external lws_dll2 owner for extant children.
Remove inline style from minimal http-server-cgi
