There are a few build options that are trying to keep and report
various statistics
- DETAILED_LATENCY
- SERVER_STATUS
- WITH_STATS
remove all those and establish a generic rplacement, lws_metrics.
lws_metrics makes its stats available via an lws_system ops function
pointer that the user code can set.
Openmetrics export is supported, for, eg, prometheus scraping.
This is a huge patch that should be a global NOP.
For unix type platforms it enables -Wconversion to issue warnings (-> error)
for all automatic casts that seem less than ideal but are normally concealed
by the toolchain.
This is things like passing an int to a size_t argument. Once enabled, I
went through all args on my default build (which build most things) and
tried to make the removed default cast explicit.
With that approach it neither change nor bloat the code, since it compiles
to whatever it was doing before, just with the casts made explicit... in a
few cases I changed some length args from int to size_t but largely left
the causes alone.
From now on, new code that is relying on less than ideal casting
will complain and nudge me to improve it by warnings.
This is complicated by the fact extern on a function declaration implies
visibility... we have to make LWS_EXTERN empty when building static.
And, setting target_compile_definitions() doesn't work inside macros,
so it has to be set explicitly for the plugins.
Checking the symbol status needs nm -C -D as per
https://stackoverflow.com/questions/37934388/symbol-visibility-not-working-as-expected
after this patch, libwebsockets.a shows no symbols when checked like that and
the static-linked minimal examples only show -U for their other dynamic
imports.
In a handful of cases we use LWS_EXTERN on extern data declarations,
those then need to change to explicit extern.
Adapt the pt sul owner list to be an array, and define two different lists,
one that acts like before and is the default for existing users, and another
that has the ability to cooperate with systemwide suspend to restrict the
interval spent suspended so that it will wake in time for the earliest
thing on this wake-suspend sul list.
Clean the api a bit and add lws_sul_cancel() that only needs the sul as the
argument.
Add a flag for client creation info to indicate that this client connection
is important enough that, eg, validity checking it to detect silently dead
connections should go on the wake-suspend sul list. That flag is exposed in
secure streams policy so it can be added to a streamtype with
"swake_validity": true
Deprecate out the old vhost timer stuff that predates sul. Add a flag
LWS_WITH_DEPRECATED_THINGS in cmake so users can get it back temporarily
before it will be removed in a v4.2.
Adapt all remaining in-tree users of it to use explicit suls.
Establish a new distributed CMake architecture with CMake code related to
a source directory moving to be in the subdir in its own CMakeLists.txt.
In particular, there's now one in ./lib which calls through to ones
further down the directory tree like ./lib/plat/xxx, ./lib/roles/xxx etc.
This cuts the main CMakelists.txt from 98KB -> 33KB, about a 66% reduction,
and it's much easier to maintain sub-CMakeLists.txt that are in the same
directory as the sources they manage, and conceal all the details that that
level.
Child CMakelists.txt become responsible for:
- include_directories() definition (this is not supported by CMake
directly, it passes it back up via PARENT_SCOPE vars in helper
macros)
- Addition child CMakeLists.txt inclusion, for example toplevel ->
role -> role subdir
- Source file addition to the build
- Dependent library path resolution... this is now a private thing
in the child CMakeLists.txt, it just passes back any adaptations
to include_directories() and the LIB_LIST without filling the
parent namespace with the details
https://libwebsockets.org/pipermail/libwebsockets/2019-April/007937.html
thanks to Bruce Perens for noting it.
This doesn't change the intention or status of the CC0 files, they were
pure CC0 before (ie, public domain) and they are pure CC0 now. It just
gets rid of the (C) part at the top of the dedication which may be read
to be a bit contradictory since the purpose is to make it public domain.
This has no effect on user code or backward compatibility.
It moves the in-tree public api header libwebsockets.h from ./lib
to ./include, and introduces a dir ./include/libwebsockets/
The single public api header is split out into 31 sub-headers
in ./include/libwebsockets. ./include/libwebsockets.h contains
some core types and platform adaptation code, but the rest of it
is now 31 #include <libwebsockets/...>
At install time, /usr/[local/]include/libwebsockets.h is installed
as before, along now with the 31 sub-headers in ...include/libwebsockets/
There's no net effect on user code.
But the api header is now much easier to maintain and study, with 31
topic-based sub headers.
This adds a plugin that interfaces to libjsongit2
https://warmcat.com/git/libjsongit2
to provide a per-vhost service for presenting bare git repos in a
web interface.
For h1 / ws, a combination of removing POLLIN wait and
stashing any unused rx lets us immediately respond to
rx flow control requests in a simple and effective way,
because the tcp socket is the stream.
But for muxed protocols like h2, that technique cannot
be used because we cannot silence the whole bundle of
streams because one can't handle any more rx dynamically.
There are control frames and content for other streams
serialized inbetween the flow controlled stream content.
We have no choice but to read to so we can see the other
things. Therefore for muxed protocols like h2 and spdy,
rx flow control boils down to tx credit manipulation
on individual streams to staunch the flow at the peer.
However that requires a round trip to take effect, any
transmitted packets that were in flight before the tx credit
reduction arrives at the remote peer are still going to come
and have to be dealt with by adding them to the stash.
This patch introduces lws_buflist scatter-gather type
buffer management for rxflow handling, so we can append
buffer segments in a linked-list to handle whatever rx
is unavoidably in flight on a stream that is trying to
assert rx flow control.
This is just an internal mass change of LWS_NO_EXTENSIONS to
LWS_WITHOUT_EXTENSIONS to match the public name and eliminate
all instances of LWS_NO_EXTENSIONS.
This enables selected things from -Wextra, can't use -Wextra because it is
fussy enough to complain about unused params on functions... they are
there for a reason.
-Wsign-compare
-Wignored-qualifiers
not -Wimplicit-fallthrough=3 ... only on gcc 7
-Wtype-limits
-Wuninitialized
not -Wclobbered ... only on gcc 7ish
fix the warnings everywhere they were found.
HTTP/2 support is now able to serve the test server, complete with
websockets, from a single vhost.
- This works the same with both OpenSSL and mbedTLS.
- POST is now wired up and works (also for file upload).
- CGI is wired up and works.
- Redirect is adapted and works
- lwsws works.
- URI urldecode, sanitation and argument parsing wired up for :path
valgrind clean (aside from openssl-style false uninit data usage in mbedtls send occasionally)
h2spec reports:
$ h2spec -h 127.0.0.1 -p 7681 -t -k -o 1
...
145 tests, 145 passed, 0 skipped, 0 failed"
Incorporates:
- "https://github.com/warmcat/libwebsockets/pull/1039
Fixes issue with -Werror=unused-variable flag
- 2c843a1395
ssl: fix infinite loop on client cert verification failure
Signed-off-by: Petar Paradzik <petar.paradzik@sartura.hr>"
Caused and fixes Coverity 184887 - 184892
For some targets like ESP32, the ah pool is mainly sitting idle wasting memory.
For HTTP/2, if the client sends a series of pipelined headers on different SIDs
that exist simultaneously, there is no way to stall the headers to wait for an
ah, because we must read the stream for stuff like WINDOW_UPDATE on the other
streams.
In both these cases having the ability to free unused ah completely and allocate
more dynamically if there is memory is useful, so this patch makes the ah pool
an initially-empty linked list that allocates on demand up to the "max pool
size" limit from the context info. When nobody wants an ah, it is freed (if
someone was waiting for it, it is directly reused).
For ESP32 it means no large, permanent alloc when lws starts and dynamic alloc
according to how many streams the client opens, which can be controlled by
SETTINGS.
Adds a new api lws_vhost_destroy(struct lws_vhost *) which allows dynamic removal of vhosts.
The external api calls two parts of internal helpers that get reused for context destroy.
The second part is called deferred by 5s... this is to ensure that event library objects
composed into structs owned by the vhost all have a chance to complete their close
asynchronously. That should happen immediately, but it requires us to return to the
event loop first.
The vhost being removed is deleted from the context vhost list by the first part, and does
not block further removals or creation during the delay for the deferred freeing of the
vhost memory.
Part 1:
- if the vhost owned a listen socket needed by other vhosts listening on same iface + port, the listen
socket is first handed off to another vhost so it stays alive
- all wsi still open on the vhost are forcibly closed (including any listen socket still attached)
- inform all active protocols on the vhost they should destroy themselves
- remove vhost from context vhost list (can no longer be found by incoming connections)
- add to a "being destroyed" context list and schedule the second part to be called in 5s
Part 2:
- remove us from the being destroyed list
- free all allocations owned by the vhost
- zero down the vhost and free the vhost itself
In libwebsockets-test-server, you can send it a SIGUSR1 to have it toggle the creation and destruction of
a second vhost on port + 1.
1) This makes lwsws run a parent process with the original permissions.
But this process is only able to respond to SIGHUP, it doesn't do anything
else.
2) You can send this parent process a SIGHUP now to cause it to
- close listening sockets in existing lwsws processes
- mark those processes as to exit when the number of active connections
on the falls to zero
- spawn a fresh child process from scratch, using latest configuration
file content, latest plugins, etc. It can now reopen listening sockets
if it chooses to, or open different listen ports or whatever.
Notes:
1) lws_context_destroy() has been split into two pieces... the reason for
the split is the first part closes the per-vhost protocols, but since
they may have created libuv objects in the per-vhost protocol storage,
these cannot be freed until after the loop has been run.
That's the purpose of the second part of the context destruction,
lws_context_destroy2().
For compatibility, if you are not using libuv, the first part calls the
second part. However if you are using libuv, you must now call the
second part from your own main.c after the first part.
This adds a context creation-time member that points to something
that should be freed when the context is destroyed.
It's in preparation for context deprecation, when a context might
be destroyed asynchronously... a related external with the
lifetime of thee context should also be freed at that time.
Adapt lwsws to use it with the context "strings" (also used for
aligned structs created by the config) allocation.
This adds a new member to the context creation info struct "ws_ping_pong_interval".
If nonzero, it sets the number of seconds that established ws connections are
allowed to be idle before a PING is forced to be sent. If zero (the default) then
tracking of idle connection is disabled for backwards compatibility.
Timeouts cover both the period between decision to send the ping and it being
sent (because it needs the socket to become writeable), and the period between
the ping being sent and the PONG coming back.
INFO debug logs are issues when the timeout stuff is operating.
You can test the server side by running the test server hacked to set ws_ping_pong_interval
and debug log mask of 15. Both the mirror protocol and the server-status protocol are
idle if nothing is happening and will trigger the PING / PONG testing. (You can also
test using lwsws and /etc/lwsws/conf with "ws-pingpong-secs": "20" in the global section)
For client, run the test client with -n -P 20 for 20s interval. -n stops the test client
writing using the mirror protocol, so it will be idle and trigger the PING / PONGs.
The timeout interval may be up to +10s late, as lws checks for affected connections every
10s.
It's going to be much easier to transition to lwsws
if the example config exactly matches what used to be
seen from the default test server.
# mkdir -p /etc/lwsws/conf.d /var/log/lwsws
# cp ./lwsws/etc-lwsws-conf-EXAMPLE /etc/lwsws/conf
# cp ./lwsws/etc-lwsws-conf.d-localhost-EXAMPLE /etc/lwsws/conf.d/test-server
# sudo lwsws
Signed-off-by: Andy Green <andy@warmcat.com>
Also add lwsws "enable-client-ssl": "1" vhost option to match.
Client cert iclient ssl is not supported in lwsws, if someone wants it, it can be added.
Signed-off-by: Andy Green <andy@warmcat.com>
Actually lwsws doesn't need his own protocol handler even for http
any more. The default http handler in lws should do everything.
Move the cgi routing into lws default http protocol handler, and
delete lwsws one. Remove all protocols from lwsws so the lws
default one gets used.
With this, and the earlier move of lejp into lws, lwsws itself
becomes 15.5KB of x86_64 (mainly conf parsing).
Signed-off-by: Andy Green <andy@warmcat.com>
