--test404 sends us instead to httpbin.org/status/404
--test404red goes to warmcat.com/noexist.html which is handled as a
redirect to a 404 page (served with a 200...), streamtype
allows redirects to be followed
--test404redref same as above but streamtype disallows redirect follow
These should all produce CONMON results for the first response, be it 404
or 302 and whether the 302 was followed or not, and whether the client
goes via the proxy or not.
Add 9 fault injection cases in SS creation flow, and 5 of those
instantiate in the minimal examples ctests. The other 4 relate
to static policy and server, I tested the server ones by hand.
These tests confirm the recent change to unpick create using
lws_ss_destroy.
Add -Wextra (with -Wno-unused-parameter) to unix builds in addition to
-Wall -Werror.
This can successfully build everything in Sai without warnings / errors.
The --blob option requires GENCRYPTO that's not on by default, to handle
the hash checks... that's going to cause a lot of confusion because it
means the simplest ss example won't build by default then.
Let's remove the blob support (and GENCRYPTO dependency) from the simplest
example and make a new minimal-secure-streams-blob example that has --blob
support and the GENCRYPTO dependency as well.
Trying to use the opaque pointer in the handle to point to the conn isn't
going to work when we need it to point to the ss handle.
Move it to have its on place in the handle.
Mbedtls does not have the same concept as openssl about preloading the
system trust store into every SSL_CTX.
This patch allows you to simulate the behaviour by passing in a context
creation-time filepath that all client SSL_CTX will be initialized from.
Currently the lws_cancel_service() api only manifests itself at lws level.
This adds a state LWSSSCS_EVENT_WAIT_CANCELLED that is broadcast to all
SS in the event loop getting the cancel service api call, and allows
SS-level user code to pick up handling events from other threads.
There's a new example minimal-secure-streams-threads which shows the
pattern for other threads to communicate with and trigger the event in the
lws service thread.
It's already the case that leaving off the "tls_trust_store" member of the
streamtype definition in the policy causes the streamtype to validate its
tls connections via the OS trust store, usually a bundle OpenSSL has been
configured to load at init automagically, but also literally the OS trust
store in windows case.
Add tests to confirm that.
This fixes the proxy rx flow by adding an lws_dsh helper to hide the
off-by-one in the "kind" array (kind 0 is reserved for tracking the
unallocated dsh blocks).
For testing, it adds a --blob option on minimal-secure-streams[-client]
which uses a streamtype "bulkproxflow" from here
https://warmcat.com/policy/minimal-proxy-v4.2-v2.json
"bulkproxflow": {
"endpoint": "warmcat.com",
"port": 443,
"protocol": "h1",
"http_method": "GET",
"http_url": "blob.bin",
"proxy_buflen": 32768,
"proxy_buflen_rxflow_on_above": 24576,
"proxy_buflen_rxflow_off_below": 8192,
"tls": true,
"retry": "default",
"tls_trust_store": "le_via_dst"
}
This downloads a 51MB blob of random data with the SHA256sum
ed5720c16830810e5829dfb9b66c96b2e24efc4f93aa5e38c7ff4150d31cfbbf
The minimal-secure-streams --blob example client delays the download by
50ms every 10KiB it sees to force rx flow usage at the proxy.
It downloads the whole thing and checks the SHA256 is as expected.
Logs about rxflow status are available at LLL_INFO log level.
This provides a way to get ahold of LWS_WITH_CONMON telemetry from Secure
Streams, it works the same with direct onward connections or via the proxy.
You can mark streamtypes with a "perf": true policy attribute... this
causes the onward connections on those streamtypes to collect information
about the connection performance, and the unsorted DNS results.
Streams with that policy attribute receive extra data in their rx callback,
with the LWSSS_FLAG_PERF_JSON flag set on it, containing JSON describing the
performance of the onward connection taken from CONMON data, in a JSON
representation. Streams without the "perf" attribute set never receive
this extra rx.
The received JSON is based on the CONMON struct info and looks like
{"peer":"46.105.127.147","dns_us":596,"sockconn_us":31382,"tls_us":28180,"txn_resp_us:23015,"dns":["2001:41d0:2:ee93::1","46.105.127.147"]}
A new minimal example minimal-secure-streams-perf is added that collects
this data on an HTTP GET from warmcat.com, and is built with a -client
version as well if LWS_WITH_SECURE_STREAMS_PROXY_API is set, that operates
via the ss proxy and produces the same result at the client.
Really not having any logs makes it difficult to know what is really
happening, but if that's you're thing this will align debug and release
modes to just have ERR and USER if you give WITH_NO_LOGS
Until now we set metadata value pointers into the onward wsi ah data
area... that's OK until we get a situation the wsi has gone away before we
have a chance to deliver the metadata over the proxy link.
Add a variant lws_ss_alloc_set_metadata() that allocates space on the heap
and takes a copy of the input metadata. Change ss-h1 to alloc copies of
its metadata so we no longer race the wsi ah lifetime.
lws_ss_set_metadata can fail... eg, due to transient OOM situation... if it does,
caller must take appropriate action like disconnect and retry.
So mark the api as requiring the result checking, and make sure all the
examples do it.
If the DNS lookup fails, we just sit out the remaining connect time.
The adapts it to reuse the wsi->sul_connect_timeout to schedule DNS lookup
retries until we're out of time.
Eventually we want to try other things as well, this is aligned with that.
Found with fault injection.
There are a few build options that are trying to keep and report
various statistics
- DETAILED_LATENCY
- SERVER_STATUS
- WITH_STATS
remove all those and establish a generic rplacement, lws_metrics.
lws_metrics makes its stats available via an lws_system ops function
pointer that the user code can set.
Openmetrics export is supported, for, eg, prometheus scraping.
warmcat.com and libwebsockets.org use Let's Encrypt certificates... LE
have changed their CA signing arrangements and after 2021-01-12 (the
point I renewed the LE server certs and received one signed using the
new arrangements) it's required to trust new root certs for the examples
to connect to warmcat.com and libwebsockets.org.
https://letsencrypt.org/2020/09/17/new-root-and-intermediates.html
This updates the in-tree CA copies, the remote policies on warmcat.com
have also been updated.
Just goes to show for real client infrastructure, you need to run your own
CA (that doesn't have to be trusted by anything outside the clients)
where you can control the CA lifetime.
Add a helper to simplify passing smd ss rx traffic into the local
smd participants, excluding the rx that received it externally to
avoid looping.
Make the smd readme clearer with three diagrams and more explanation
of how the ss proxying works.
