Add -Wextra (with -Wno-unused-parameter) to unix builds in addition to
-Wall -Werror.
This can successfully build everything in Sai without warnings / errors.
Really not having any logs makes it difficult to know what is really
happening, but if that's you're thing this will align debug and release
modes to just have ERR and USER if you give WITH_NO_LOGS
Also prioritize LD_LIBRARY_PATH check for plugins first
Iterate through paths in LD_LIBRARY_PATH in order
Warn on failed plugins init but continue protocol init
warmcat.com and libwebsockets.org use Let's Encrypt certificates... LE
have changed their CA signing arrangements and after 2021-01-12 (the
point I renewed the LE server certs and received one signed using the
new arrangements) it's required to trust new root certs for the examples
to connect to warmcat.com and libwebsockets.org.
https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html
This updates the in-tree CA copies, the remote policies on warmcat.com
have also been updated.
Just goes to show for real client infrastructure, you need to run your own
CA (that doesn't have to be trusted by anything outside the clients)
where you can control the CA lifetime.
This is a huge patch that should be a global NOP.
For unix type platforms it enables -Wconversion to issue warnings (-> error)
for all automatic casts that seem less than ideal but are normally concealed
by the toolchain.
This is things like passing an int to a size_t argument. Once enabled, I
went through all args on my default build (which build most things) and
tried to make the removed default cast explicit.
With that approach it neither change nor bloat the code, since it compiles
to whatever it was doing before, just with the casts made explicit... in a
few cases I changed some length args from int to size_t but largely left
the causes alone.
From now on, new code that is relying on less than ideal casting
will complain and nudge me to improve it by warnings.
RFC6724 defines an ipv6-centric DNS result sorting algorithm, that
takes route and source address route information for the results
given by the DNS resolution, and sorts them in order of preferability,
which defines the order they should be tried in.
If LWS_WITH_NETLINK, then lws takes care about collecting and monitoring
the interface, route and source address information, and uses it to
perform the RFC6724 sorting to re-sort the DNS before trying to make
the connections.
Move the common plugin scanning dir stuff to be based on lws_dir, which
already builds for windows. Previously this was done via dirent for unix
and libuv for windows.
Reduce the dl plat stuff to just wrap instantiation and destruction of
dynlibs, establish common code in lib/misc/dir.c for plugin scanning
itself.
Migrate the libuv windows dl stuff to windows-plugins.c, so that he's
available even if later libuv loop support becomes and event lib plugin.
Remove the existing api exports scheme for plugins, just export a const struct
now which has a fixed header type but then whatever you want afterwards depending
on the class / purpose of the plugin. Place a "class" string in the header so
there can be different kinds of plugins implying different types exported.
Make the plugin apis public and add support for filter by class string, and
per instantation / destruction callbacks so the subclassed header type can
do its thing for the plugin class. The user provides a linked-list base
for his class of plugins, so he can manage them completely separately and
in user code / user export types.
Rip out some last hangers-on from generic sessions / tables.
This is all aimed at making the plugins support general enough so it can
provide event lib plugins later.
Add a minimal example showing how to write a binance client using
permessage_deflate and LCCSF_PRIORITIZE_READS to minimize latency.
This is partly informed by kutoga's example on
https://github.com/warmcat/libwebsockets/issues/2019 which was in
turn based on the existing ws client minimal example.
PARENT_SCOPE needs adjusting in a few places for wolfssl to work, and
we need a second level export of USE_WOLFSSL through lib/CMakeLists.txt
Add noi/f32 Sai build for WOLFSSL + MINIMAL_EXAMPLES
Adapt the pt sul owner list to be an array, and define two different lists,
one that acts like before and is the default for existing users, and another
that has the ability to cooperate with systemwide suspend to restrict the
interval spent suspended so that it will wake in time for the earliest
thing on this wake-suspend sul list.
Clean the api a bit and add lws_sul_cancel() that only needs the sul as the
argument.
Add a flag for client creation info to indicate that this client connection
is important enough that, eg, validity checking it to detect silently dead
connections should go on the wake-suspend sul list. That flag is exposed in
secure streams policy so it can be added to a streamtype with
"swake_validity": true
Deprecate out the old vhost timer stuff that predates sul. Add a flag
LWS_WITH_DEPRECATED_THINGS in cmake so users can get it back temporarily
before it will be removed in a v4.2.
Adapt all remaining in-tree users of it to use explicit suls.
Establish a new distributed CMake architecture with CMake code related to
a source directory moving to be in the subdir in its own CMakeLists.txt.
In particular, there's now one in ./lib which calls through to ones
further down the directory tree like ./lib/plat/xxx, ./lib/roles/xxx etc.
This cuts the main CMakelists.txt from 98KB -> 33KB, about a 66% reduction,
and it's much easier to maintain sub-CMakeLists.txt that are in the same
directory as the sources they manage, and conceal all the details that that
level.
Child CMakelists.txt become responsible for:
- include_directories() definition (this is not supported by CMake
directly, it passes it back up via PARENT_SCOPE vars in helper
macros)
- Addition child CMakeLists.txt inclusion, for example toplevel ->
role -> role subdir
- Source file addition to the build
- Dependent library path resolution... this is now a private thing
in the child CMakeLists.txt, it just passes back any adaptations
to include_directories() and the LIB_LIST without filling the
parent namespace with the details
Add a member to the vh init struct allowing control of the overall
connection wait introduced in an earlier patch. Set it to 20s
by default.
The timeout_secs member controls the individual DNS result
connect timeout and is reduced to 5s by default.
Replace the bash selftest plumbing with CTest.
To use the selftests, build with -DLWS_WITH_MINIMAL_EXAMPLES=1
and `CTEST_OUTPUT_ON_FAILURE=1 make test` or just
`make test`.
To disable tests that require internet access, also give
-DLWS_CTEST_INTERNET_AVAILABLE=0
Remove travis and appveyor scripts on master.
Remove travis and appveyor decals on README.md.
Add support for external pthreads lib on windows and some docs about how to do.
It can build with LWS_WITH_THREADPOOL and LWS_WITH_MINIMAL_EXAMPLES including the
pthreads-dependent ones without warnings or errors on windows platform as well with this.
pthreads_t can be anything, including a struct - not a pointer-to-a-struct
but the struct itself. These can't be cast to a void * for printing as they can
on linux, where the base type is a pointer.
Let's fix all the usage of those to determine their own thread index in terms
of the meaning to the program rather than as a tid.
Fix pthreads detection in the minimal examples and add it where needed.
Fix unistd.h include to be conditional on not WIN32
With this, -DLWS_WITH_MINIMAL_EXAMPLES=1 is happy and warning-free
on windows.
Refactor everything around ping / pong handling in ws and h2, so there
is instead a protocol-independent validity lws_sul tracking how long it
has been since the last exchange that confirms the operation of the
network connection in both directions.
Clean out periodic role callback and replace the last two role users
with discrete lws_sul for each pt.
Old certs were getting near the end of their life and we switched the
server to use letsencrypt. The root and intermediate needed for the
mbedtls case changed accordingly
wsi timeout, wsi hrtimer, sequencer timeout and vh-protocol timer
all now participate on a single sorted us list.
The whole idea of polling wakes is thrown out, poll waits ignore the
timeout field and always use infinite timeouts.
Introduce a public api that can schedule its own callback from the event
loop with us resolution (usually ms is all the platform can do).
Upgrade timeouts and sequencer timeouts to also be able to use us resolution.
Introduce a prepared fakewsi in the pt, so we don't have to allocate
one on the heap when we need it.
Directly handle vh-protocol timer if LWS_MAX_SMP == 1
There are quite a few linked-lists of things that want events after
some period. This introduces a type binding an lws_dll2 for the
list and a lws_usec_t for the duration.
The wsi timeouts, the hrtimer and the sequencer timeouts are converted
to use these, also in the common event wait calculation.
Travis seems to be restricting the number of outgoing connections
or the rate of them... we have been using 10 concurrent and 100 connections
[2019/08/02 09:26:22:7950] USER: callback_minimal_spam: established (try 10, est 8, closed 0, err 0)
[2019/08/02 09:26:22:8041] USER: callback_minimal_spam: established (try 10, est 9, closed 0, err 0)
[2019/08/02 09:26:23:0098] USER: callback_minimal_spam: reopening (try 11, est 10, closed 1, err 0)
[2019/08/02 09:26:23:0105] USER: callback_minimal_spam: reopening (try 12, est 10, closed 2, err 0)
[2019/08/02 09:26:23:0111] USER: callback_minimal_spam: reopening (try 13, est 10, closed 3, err 0)
[2019/08/02 09:26:23:0117] USER: callback_minimalRROR: closed before established (try 25, est 14, closed 14, err 2)
[2019/08/02 09:26:44:6125] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 26, est 14, closed 14, err 3)
[2019/08/02 09:26:44:6129] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 27, est 14, closed 14, err 4)
[2019/08/02 09:26:44:6133] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 28, est 14, closed 14, err 5)
[2019/08/02 09:26:44:6137] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 29, est 14, closed 14, err 6)
[2019/08/02 09:26:45:6152] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 30, est 14, closed 14, err 7)
[2019/08/02 09:26:45:6163] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 31, est 14, closed 14, err 8)
[2019/08/02 09:26:45:6168] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 32, est 14, closed 14, err 9)
[2019/08/02 09:26:45:6174] ERR: CLIENT_CONNECTION_ERROR: closed before established (try 33, est 14, closed 14, err 10)
[2019/08/02 09:26:47:0635] USER: callback_minimal_spam: established (try 34, est 14, closed 14, err 10)
Reduce to 3 concurrent / 15 see if it helps travis get over the hump
The logic in the loops for insertion and deletion from the
mini, forced to non ulimit max fds in the pt mode was not
quite right.
It showed up in hard to reproduce problem with the ws client
spam test that uses the mini mode, on travis. This should
fix the root cause.
An lws context usually contains a processwide fd -> wsi lookup table.
This allows any possible fd returned by a *nix type OS to be immediately
converted to a wsi just by indexing an array of struct lws * the size of
the highest possible fd, as found by ulimit -n or similar.
This works modestly for Linux type systems where the default ulimit -n for
a process is 1024, it means a 4KB or 8KB lookup table for 32-bit or
64-bit systems.
However in the case your lws usage is much simpler, like one outgoing
client connection and no serving, this represents increasing waste. It's
made much worse if the system has a much larger default ulimit -n, eg 1M,
the table is occupying 4MB or 8MB, of which you will only use one.
Even so, because lws can't be sure the OS won't return a socket fd at any
number up to (ulimit -n - 1), it has to allocate the whole lookup table
at the moment.
This patch looks to see if the context creation info is setting
info->fd_limit_per_thread... if it leaves it at the default 0, then
everything is as it was before this patch. However if finds that
(info->fd_limit_per_thread * actual_number_of_service_threads) where
the default number of service threads is 1, is less than the fd limit
set by ulimit -n, lws switches to a slower lookup table scheme, which
only allocates the requested number of slots. Lookups happen then by
iterating the table and comparing rather than indexing the array
directly, which is obviously somewhat of a performance hit.
However in the case where you know lws will only have a very few wsi
maximum, this method can very usefully trade off speed to be able to
avoid the allocation sized by ulimit -n.
minimal examples for client that can make use of this are also modified
by this patch to use the smaller context allocations.
https://libwebsockets.org/pipermail/libwebsockets/2019-April/007937.html
thanks to Bruce Perens for noting it.
This doesn't change the intention or status of the CC0 files, they were
pure CC0 before (ie, public domain) and they are pure CC0 now. It just
gets rid of the (C) part at the top of the dedication which may be read
to be a bit contradictory since the purpose is to make it public domain.
