tls: JIT Trust

Add support for dynamically determining the CAs needed to validate server certificates. This allows you to avoid instantiating > 120 X.509 trusted CA certs and have them take up heap the whole time. Works for both openssl and mbedtls. See READMEs/README.jit-trust.md for the documentation You likely want the next patch for http redirect enhancements as well.
2021-05-21 14:32:21 +01:00 · 2021-05-21 14:32:21 +01:00 · 2f9bb7a30a
parent f8f1749be9
commit 2f9bb7a30a
36 changed files with 11450 additions and 28 deletions
--- a/.sai.json
+++ b/.sai.json
@ -282,6 +282,10 @@
 		"cookiejar": {
 			"cmake":	"-DLWS_WITH_CACHE_NSCOOKIEJAR=ON"
 		},
+		"jittrust": {
+			"cmake":	"-DLWS_WITH_TLS_JIT_TRUST=1",
+			"platforms":	"none, linux-fedora-32/x86_64-amd/gcc"
+		},
 		"smp": {
 			"cmake":	"-DLWS_MAX_SMP=32 -DLWS_WITH_MINIMAL_EXAMPLES=1"
 		},
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@ -166,6 +166,7 @@ option(LWS_SSL_CLIENT_USE_OS_CA_CERTS "SSL support should make use of the OS-ins
 option(LWS_TLS_LOG_PLAINTEXT_RX "For debugging log the received plaintext as soon as decrypted" OFF)
 option(LWS_TLS_LOG_PLAINTEXT_TX "For debugging log the transmitted plaintext just before encryption" OFF)
 option(LWS_WITH_TLS_SESSIONS "Enable persistent, resumable TLS sessions" ON)
+option(LWS_WITH_TLS_JIT_TRUST "Enable dynamically computing which trusted TLS CA is needed to be instantiated" OFF)

 #
 # Event library options (may select multiple, or none for default poll()
--- a/9
+++ b/9
@ -6,10 +6,11 @@ them.

 Original liberal license retained:

-  - lib/misc/sha-1.c          - 3-clause BSD license retained, link to original [BSD3]
-  - win32port/zlib            - ZLIB license (see zlib.h) [ZLIB]
-  - lib/tls/mbedtls/wrapper   - Apache 2.0 (only built if linked against mbedtls) {APACHE2]
-  - lib/misc/base64-decode.c  - already MIT
+  - lib/misc/sha-1.c                     - 3-clause BSD license retained, link to original [BSD3]
+  - win32port/zlib                       - ZLIB license (see zlib.h) [ZLIB]
+  - lib/tls/mbedtls/wrapper              - Apache 2.0 (only built if linked against mbedtls) {APACHE2]
+    lib/tls/mbedtls/mbedtls-extensions.c
+  - lib/misc/base64-decode.c             - already MIT

 Relicensed to MIT:

--- a/READMEs/README.jit-trust.md
+++ b/READMEs/README.jit-trust.md
@ -0,0 +1,446 @@
+# JIT trust
+
+![JIT Trust logo](../doc-assets/jit-trust-logo.png)
+
+## Background
+
+Most systems using openssl rely on a system trust bundle that openssl was
+compiled to load at library init.  This is a bit expensive, since it
+instantiates over 120 CA X.509 certs, but most modern Linux systems don't really
+notice the permanent use of 1MB or so of heap from init, the advantage is client
+connections have all the trusted root certs available in memory to perform
+validation.
+
+![Using system trust bundles](../doc-assets/jit-trust-system-trust.png)
+
+For the kind of systems that choose mbedtls, they will typically either be
+burdened by or not even have enough ram to take this approach.
+
+If the device only connects to endpoints that are signed by a specific
+CA, you can just prepare the connection with the known trusted CA, that's
+the approach the examples take.  This method should still be used for critical
+connections to the cloud, for example provide the necessary CA cert in the
+Secure Streams policy, or at vhost creation time.
+
+![Using system trust bundles](../doc-assets/jit-trust-single-trust.png)
+
+However if you also have a browser type application that could connect anywhere,
+but you don't have heap spare to preload all the CAs, you need something like
+"JIT trust".
+
+## JIT trust overview
+
+The basic approach is to connect to the server to retrieve its certificates,
+then study the certificates to determine the identity of the missing trusted
+cert we should be trying to validate with.
+
+![JIT Trust overview](../doc-assets/jit-trust-overview.png)
+
+We attempt to get the trusted cert from some local or remote store, and retry
+the connection having instantiated the missing CA cert as trusted for that
+connection, if it is one that we do actually trust.  If it lies about what CA it
+needs to validate, or we do not trust the one it asks for, subsequent
+connections will fail.
+
+If it asked for a trusted CA that we trust, and the relationship was valid, the
+tls negotiation should then complete successfully, and we can cache the CA cert
+and the host -> CA cert pre-trust requirement so future connections can work
+first time.
+
+## Subject Key Id and Authority Key Id
+
+All of the certificates publish a unique-enough personal "Subject Key ID" or
+SKID blob.  These are typically 20-byte hashes based on the cert public key.
+
+When a server certificate is issued by the CA, an entry is made first in the
+certificate noting the SKID of the certificate that will be used to sign it,
+in an "Authority Key ID", or AKID, extension.  The certificate is then signed by
+the parent certificate private key to prove it was issued by the real owner of
+the CA or intermediate certificate.
+
+![X.509 validation paths](../doc-assets/jit-trust-paths.png)
+
+Basically this AKID on a certificate is guiding the validator with
+information about which certificate it claims is next in the chain of trust
+leading back to a trusted CA.  Lying about it doesn't help an attacker,
+because we're only using the AKID to get the CA certificate and then try to do
+the full signature check using it, if it's not really signed by the AKID cert it
+told, or anything else wrong, the actual validation will just fail.
+
+A chain that terminates in a CA certificate is complete, and can undergo full
+validation using the tls library.
+
+## Converting the Mozilla trust bundle for JIT trust
+
+Lws provides a bash script `./scripts/mozilla-trust-gen.sh` that can fetch the
+latest Mozilla CA trust bundle for certs usable for tls validation, and convert
+it to three different forms to allow maintaining the trust bundle in different
+ways for different kinds of device to consume.
+
+ - as a webroot directory, so you can server trusted DERs, with
+   symlink indexes to the CA certs by SKID and issuer/serial
+
+ - as an atomic binary blob, currently about 143KB, with structure
+   at the start pointing to DER certs and indexes inside
+
+ - a C-compiler friendly `uint8_t` array version of the blob,
+   so it can be compiled into .rodata directly if necessary.
+
+Currently there are 128 certs in the trust bundle, and the whole blob is about
+143KB uncompressed.
+
+## Considerations about maintaining the trust blob
+
+Mozilla update their trust bundle at intervals, and there have been at least
+three cases where they have removed or distrusted CAs from it by their own
+decision, because they have issued dangerous certificates, (like one for `*`
+that will validate anything at all).  Certifacte owners may also revoke their
+own certificates for any reason and issue replacements.
+
+The certs in the trust bundle expire, currently 10/128 will expire within 3
+years and 50/128 over the next 10 years.  So new and replacement certificates
+are also being added at intervals.
+
+Part of using the trust bundle is building in some way to update what is trusted
+over the lifetime of the device, which may exceed 10 years.
+
+Depending on the device, it may not be any problem to keep the trust blob in the
+firmware, and update the firmware ongoing every few months.  So you could build
+it into the firmware using the C array include file (the minimal example takes
+this approach).
+
+Another device may have difficulty updating the firmware outside of emergencies,
+it could keep the trust blob in a separate area and update it separately.
+Having it as a single blob makes it easy to fetch and update.
+
+Finally constrained devices, say in ESP32 class, may not have space or desire
+to store the trust blob in the device at all, it could query a remote server on
+demand to check for any trusted CA matching a given AKID and retrieve and cache
+it in volatile ram.  This would use the webroot produced by the script, via tls
+and a fixed CA cert outside this system.
+
+## Format of the JIT trust blob
+
+The trust blob layout is currently
+
+```
+00:  54 42 4c 42     Magic "TBLB"
+04:  00 01           MSB-first trust blob layout version
+06:  XX XX           MSB-first count of certificates
+08:  XX XX XX XX     MSB-first trust blob generation unix time
+0c:  XX XX XX XX     MSB-first offset from blob start of cert length table
+10:  XX XX XX XX     MSB-first offset from blob start of SKID length table
+14:  XX XX XX XX     MSB-first offset from blob start of SKID table
+18:  XX XX XX XX     MSB-first total blob length
+
+1c:  XX .. XX        DER certs (start at +0x1c)
+  :  XX .. XX        DER cert length table (MSB-first 16-bit per cert)
+  :  XX .. XX        SKID length table (8-bit per cert)
+  :  XX .. XX        SKID table (variable per cert)
+```
+
+## Enabling JIT Trust
+
+```
+$ cmake .. -DLWS_WITH_TLS_JIT_TRUST=1
+```
+
+## Minimal example for JIT Trust
+
+`minimal-examples/http-client/minimal-http-client-jit-trust` is built if JIT
+Trust is enabled at cmake and `-DLWS_WITH_MINIMAL_EXAMPLES=1`.  This is based on
+minimal-http-client, except the loading of the system trust bundle is defeated,
+so by default it does not trust anything and cannot complete any tls connection.
+It includes the mozilla trust blob as a header file when built.
+
+It tries to do an http client connection twice, the first time fails but JIT
+Trust determines which trusted CA cert is missing, retreives it from the trust
+blob and creates the necessary temporary vhost with the correct CA cert(s)
+trusted.  On the next retry, the connection succeeds.
+
+## Processing of x509 AKID and SKIDs
+
+We study each x509 cert sent by the server in turn.  We parse out the SKID and
+AKID on each one and stash them (up to 4 deep).
+
+After the initial validation fails due to lack of any trusted CA, lws has
+collected all the AKID and SKIDs that were in certs sent by the server.  Since
+these may be sent in any order, may be malicious, and may even contain the
+(untrusted) root CA, they are sorted into a trust path using the AKID and SKID
+relationships.
+
+To cover cross-signing and cases where the root cert(s) were wrongly sent by
+a misconfigured server, all of the AKIDs in the stash are queried against the
+trusted CA store.  In cross-signing, multiple intermediates are provided with
+the same SKID, that all match the server certificate AKID parent.  Since we
+might meet certificates that trust multiple valid CAs that can validate the
+certificate, we support up to three CA certs imported.
+
+A user `lws_system_ops` handler performs the query, so it can consist of any
+kind of backing store or remote lookup. Helpers are provided to query the JIT
+trust mozilla blob, so the system helper is small in the typical case, just
+calling lws helpers.
+
+The results (up to three CA certs to account for cross-signing scenarios) are
+collected and a 1hr TTL cache entry made for the hostname and the SKIDs of the
+matched CAs, if there is no existing JIT vhost with its tls context configured
+with the needed trusted CAs, one is created.
+
+When the connection is retried, lws checks the cache for the hostname having
+a binding to an existing JIT vhost, if that exists the connection proceeds
+bound to that.  If there is a cache entry but no JIT vhost, one is created using
+the information in the cache entry.
+
+## Efficiency considerations
+
+From cold, the JIT Trust flow is 
+
+1. A sacrificial connection is made to get the server certs
+2. Query the JIT Trust database for AKIDs mentioned in the certs (this may be
+done asynchronously)
+3. Create a temporary vhost with the appropriate trusted certs enabled in it,
+   and add an entry in the cache for this hostname to the SKIDs of the CAs
+   enabled on this temporary vhost
+4. Retry, querying the cache to bind the connection to the right temporary vhost
+
+An lws_cache in heap is maintained so step 1 can be skipped while hostname->
+SKID items exist in the cache.  If the items expire or are evicted, it just
+means we have to do step 1 again.
+
+For a short time, the vhost created in step 3 is allowed to exist when idle, ie
+when no connections are actively using it.  In the case the vhost exists and
+the cache entry exists for the hostname, the connection can proceed successfully
+right away without steps 1 through 3.
+
+## APIs related to JIT Trust 
+
+Systems that support JIT trust define an `lws_system_ops` callback
+that does whatever the system needs to do for attempting to acquire
+a trusted cert with a specified SKID or issuer/serial.
+
+```
+int (*jit_trust_query)(struct lws_context *cx, const uint8_t *skid, size_t skid_len, void *got_opaque);
+```
+
+The ops handler doesn't have to find the trusted cert immediately before
+returning, it is OK starting the process and later if successful calling a
+helper `lws_tls_jit_trust_got_cert_cb()` with the `got_opaque` from the query.
+This will cache the CA cert so it's available at the next connection retry for
+preloading.
+
+An helper suitable for `ops->jit_trust_query` using trust blob lookup in .rodata
+is provided in `lws_tls_jit_trust_blob_queury_skid()`, the callback above should
+be called with its results as shown in the minimal example.
+
+## Runtime tuning for JIT Trust
+
+The context creation info struct has a couple of runtime-tunable settings
+related to JIT Trust.
+
+`.jitt_cache_max_footprint`: default 0 means no limit, otherwise the hostname->
+SKID cache is kept below this many bytes in heap, by evicting LRU entries.
+
+`.vh_idle_grace_ms`: default 0 means 5000ms, otherwise sets the length of time
+a JIT Trust vhost is allowed to exist when it has no connections using it.
+Notice that, eg, h2 connections have their own grace period when they become
+idle, to optimize reuse, this period does not start until any h2 network
+connection bound to the vhost has really closed.
+
+## Considerations around http redirects
+
+HTTP redirects are transactions that tell the client to go somewhere else to
+continue, typically a 301 response with a Location: header explaining where to
+go.
+
+JIT Trust supports redirects to hosts with the same or different trust
+requirements, each step in the redirect is treated as a new connection that will
+fail, try to create a vhost with the right trust and work on the retry.
+
+Lws rejects by default protocol downgrades (https -> http) on redirects, the
+example used a context option `LCCSCF_ACCEPT_TLS_DOWNGRADE_REDIRECTS` to
+override this.
+
+## Works out of the box on recent mbedtls and openssl
+
+No modifications are needed to either tls library.
+
+## Compatibility Testing
+
+A list of the top 100 sites each from the US and the ROW were combined to
+produce 156 unqiue domain names [1]
+
+The Mbedtls build of JIT trust minimal example was run against each of these
+doing a GET on path `/` and restricted to h1 (`--server xxx --h1`).  In some
+cases, the server at the base domain name is broken or down, as verified using
+ssllabs.com as a second opinion.  These domains only resolve properly using
+`www.` prefix.
+
+In some cases the sites check the user agent and return a 4xx, these are taken
+as success for this test, since there was no problem at the tls layer.
+
+|site|h1|h2|comment|
+|---|---|---|---|
+|adobe.com|✓|✓||
+|allegro.pl|✓|✓||
+|allrecipes.com|✓|✓||
+|amazon.co.jp|✓|✓||
+|amazon.com|✓|✓||
+|amazon.co.uk|✓|✓||
+|amazon.de|✓|✓||
+|amazon.fr|✓|✓||
+|amazon.in|✓|✓||
+|amazon.it|✓|✓||
+|aol.com|✓|✓||
+|apartments.com|✓|✓||
+|apple.com|✓|✓||
+|ar.wikipedia.org|✓|✓||
+|att.com|✓|✓||
+|bankofamerica.com|✓|✓||
+|bbc.com|✓|✓||
+|bbc.co.uk|✓|✓||
+|bestbuy.com|✕|✓|redirect-> `www.` then h1: timeout, h2: 403 forbidden... geolocated?|
+|booking.com|✓|✓||
+|britannica.com|✓|✓||
+|bulbagarden.net|✓|✓||
+|businessinsider.com|✓|✓||
+|ca.gov|✓|✓||
+|caixa.gov.br|✕|✕|TLS trust works fine.  Continuously redirects to self... sends set-cookie that we don't return yet|
+|capitalone.com|✓|✓||
+|cbssports.com|✓|✓||
+|cdc.gov|✓|✓||
+|chase.com|✓|✓||
+|chrome.google.com|✓|✓||
+|cnbc.com|✓|✓||
+|cnet.com|✓|✓||
+|cnn.com|✓|✓||
+|cookpad.com|✓|✓||
+|costco.com|✕|✓|TLS trust works fine.  But with or without `www.` server does not reply within 15s on h1, sends 403 OK on h2... Curl acts the same as we do, firefox works... geolocated?||
+|craigslist.org|✓|✓||
+|dailymotion.com|✓|✓||
+|de.wikipedia.org|✓|✓||
+|dictionary.com|✓|✓||
+|ebay.com|✓|✓||
+|ebay.co.uk|✓|✓||
+|en.wikipedia.org|✓|✓||
+|epicgames.com|✓|✓||
+|espn.com|✓|✓||
+|es.wikipedia.org|✓|✓||
+|etsy.com|✓|✓||
+|expedia.com|✓|✓||
+|facebook.com|✓|✓||
+|fandom.com|✓|✓||
+|fedex.com|✓|✓||
+|finance.yahoo.com|✓|✓||
+|www.foodnetwork.com|✓|✓|`www.` served correctly, base domain is misconfigured with expired cert, confirmed with ssllabs + curl|
+|forbes.com|✓|✓||
+|foxnews.com|✓|✓||
+|fr.wikipedia.org|✓|✓||
+|gamepedia.com|✓|✓||
+|genius.com|✓|✓||
+|glassdoor.com|✓|✓||
+|globo.com|✓|✓||
+|google.com|✓|✓||
+|healthline.com|✓|✓||
+|homedepot.com|✓|✓||
+|hulu.com|✓|✓||
+|hurriyet.com.tr|✓|✓||
+|id.wikipedia.org|✓|✓||
+|ign.com|✓|✓||
+|ikea.com|✓|✓|`www.` served correctly, base domain is misconfigured with nonresponsive server, confirmed with ssllabs|
+|ilovepdf.com|✓|✓||
+|imdb.com|✓|✓||
+|indeed.com|✓|✓||
+|indiatimes.com|✓|✓||
+|instagram.com|✓|✓||
+|investopedia.com|✓|✓||
+|irs.gov|✓|✓||
+|it.wikipedia.org|✓|✓||
+|ivi.ru|✓|✓||
+|ja.wikipedia.org|✓|✓||
+|kakaku.com|✓|✓||
+|khanacademy.org|✓|✓||
+|kinopoisk.ru|✓|✓||
+|leboncoin.fr|✓|✓||
+|linkedin.com|✓|✓||
+|live.com|✓|✓||
+|lowes.com|✓|✓||
+|macys.com|✕|✓|TLS trust works fine.  Continuously redirects to self... `www.` same, curl acts same but OK if given -b -c, so akami cookie storage issue|
+|mail.ru|✓|✓||
+|mail.yahoo.com|✓|✓||
+|mapquest.com|✓|✓||
+|mayoclinic.org|✓|✓||
+|medicalnewstoday.com|✓|✓||
+|mercadolivre.com.br|✓|✓||
+|merriam-webster.com|✓|✓||
+|microsoft.com|✓|✓||
+|msn.com|✓|✓||
+|namu.wiki|✓|✓||
+|nbcnews.com|✓|✓||
+|netflix.com|✓|✓||
+|nih.gov|✓|✓||
+|nl.wikipedia.org|✓|✓||
+|ny.gov|✓|✓||
+|nytimes.com|✓|✓||
+|ok.ru|✓|✓||
+|onet.pl|✓||
+|orange.fr|✓|✓||
+|paypal.com|✓|✓||
+|pinterest.com|✓|✓||
+|pixiv.net|✓|✓||
+|play.google.com|✓|✓||
+|pl.wikipedia.org|✓|✓||
+|www.programme-tv.net|✓|✓|OK with `www.`, without `www.` TLS trust works fine but server does not reply, same with curl|
+|pt.wikipedia.org|✓|✓||
+|quizlet.com|✓|✓||
+|quora.com|✓|✓|||
+|rakuten.co.jp|✓|✓||
+|realtor.com|✓|✓||
+|reddit.com|✓|✓||
+|reverso.net|✓|✓||
+|roblox.com|✓|✓||
+|rottentomatoes.com|✓|✓||
+|ru.wikipedia.org|✓|✓||
+|sahibinden.com|✓|✓||
+|smallpdf.com|✓|✓||
+|speedtest.net|✓|✓||
+|spotify.com|✓|✓||
+|steampowered.com|✓|✓||
+|target.com|✓|✓||
+|theguardian.com|✓|✓||
+|tripadvisor.com|✓|✓||
+|tr.wikipedia.org|✓|✓||
+|twitch.tv|✓|✓||
+|twitter.com|✓|✓||
+|uol.com.br|✓|✓||
+|ups.com|✓|✓||
+|urbandictionary.com|✓|✓||
+|usatoday.com|✓|✓||
+|usnews.com|✕|✓|TLS trust works fine. Needs `www.` else server doesn't respond in 15s, sends 403 on h2, Curl acts the same, geolocated?|
+|usps.com|✓|✓||
+|verizon.com|✓|✓||
+|vk.com|✓|✓||
+|walmart.com|✓|✓||
+|washingtonpost.com|✓|✓||
+|weather.com|✓|✓||
+|webmd.com|✓|✓||
+|whatsapp.com|✓|✓||
+|wowhead.com|✓|✓||
+|wp.pl|✓|✓||
+|www.gov.uk|✓|✓||
+|xfinity.com|✓|✓||
+|yahoo.co.jp|✓|✓||
+|yahoo.com|✓|✓||
+|yandex.ru|✓|✓||
+|yellowpages.com|✓|✓||
+|yelp.com|✓|✓||
+|youtube.com|✓|✓||
+|zh.wikipedia.org|✓|✓||
+|zillow.com|✓|✓||
+
+[1]
+```
+wget -O- https://ahrefs.com/blog/most-visited-websites/ | grep most-visited-websites-us | \
+        sed -E 's/class="column-2">/|/g' | tr '|' '\n' | \
+        sed 's/<.*//g' | grep -v Domain | grep -v Josh | sort | uniq
+```
+
--- a/cmake/lws_config.h.in
+++ b/cmake/lws_config.h.in
@ -72,10 +72,13 @@
 #cmakedefine LWS_HAVE_mbedtls_ssl_set_hs_ca_chain
 #cmakedefine LWS_HAVE_mbedtls_ssl_set_hs_own_cert
 #cmakedefine LWS_HAVE_mbedtls_ssl_set_hs_authmode
+#cmakedefine LWS_HAVE_mbedtls_ssl_set_verify
 #cmakedefine LWS_HAVE_mbedtls_x509_crt_parse_file
 #cmakedefine LWS_HAVE_MBEDTLS_NET_SOCKETS
+#cmakedefine LWS_HAVE_MBEDTLS_AUTH_KEY_ID
 #cmakedefine LWS_HAVE_NEW_UV_VERSION_H
 #cmakedefine LWS_HAVE_OPENSSL_ECDH_H
+#cmakedefine LWS_HAVE_OPENSSL_STACK
 #cmakedefine LWS_HAVE_PIPE2
 #cmakedefine LWS_HAVE_EVENTFD
 #cmakedefine LWS_HAVE_PTHREAD_H
@ -214,6 +217,7 @@
 #cmakedefine LWS_WITH_SYS_STATE
 #cmakedefine LWS_WITH_THREADPOOL
 #cmakedefine LWS_WITH_TLS
+#cmakedefine LWS_WITH_TLS_JIT_TRUST
 #cmakedefine LWS_WITH_TLS_SESSIONS
 #cmakedefine LWS_WITH_UDP
 #cmakedefine LWS_WITH_ULOOP
--- a/doc-assets/jit-trust-logo.png
+++ b/doc-assets/jit-trust-logo.png
--- a/doc-assets/jit-trust-overview.png
+++ b/doc-assets/jit-trust-overview.png
--- a/doc-assets/jit-trust-paths.png
+++ b/doc-assets/jit-trust-paths.png
--- a/doc-assets/jit-trust-single-trust.png
+++ b/doc-assets/jit-trust-single-trust.png
--- a/doc-assets/jit-trust-system-trust.png
+++ b/doc-assets/jit-trust-system-trust.png
--- a/include/libwebsockets/lws-context-vhost.h
+++ b/include/libwebsockets/lws-context-vhost.h
@ -880,6 +880,15 @@ struct lws_context_creation_info {
 	 * the custom event loop natively as if it were an "event library".
 	 */

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	size_t					jitt_cache_max_footprint;
+	/**< CONTEXT: 0 for no limit, else max bytes used by JIT Trust cache...
+	 * LRU items are evicted to keep under this limit */
+	int					vh_idle_grace_ms;
+	/**< CONTEXT: 0 for default of 5000ms, or number of ms JIT Trust vhosts
+	 * are allowed to live without active connections using them. */
+#endif
+
 	/* Add new things just above here ---^
 	 * This is part of the ABI, don't needlessly break compatibility
 	 *
--- a/include/libwebsockets/lws-misc.h
+++ b/include/libwebsockets/lws-misc.h
@ -817,6 +817,25 @@ lws_is_ssl(struct lws *wsi);
 LWS_VISIBLE LWS_EXTERN int
 lws_is_cgi(struct lws *wsi);

+/**
+ * lws_tls_jit_trust_blob_queury_skid() - walk jit trust blob for skid
+ *
+ * \param _blob: the start of the blob in memory
+ * \param blen: the length of the blob in memory
+ * \param skid: the SKID we are looking for
+ * \param skid_len: the length of the SKID we are looking for
+ * \param prpder: result pointer to receive a pointer to the matching DER
+ * \param prder_len: result pointer to receive matching DER length
+ *
+ * Helper to scan a JIT Trust blob in memory for a trusted CA cert matching
+ * a given SKID.  Returns 0 if found and *prpder and *prder_len are set, else
+ * nonzero.
+ */
+LWS_VISIBLE LWS_EXTERN int
+lws_tls_jit_trust_blob_queury_skid(const void *_blob, size_t blen,
+				   const uint8_t *skid, size_t skid_len,
+				   const uint8_t **prpder, size_t *prder_len);
+
 /**
 * lws_open() - platform-specific wrapper for open that prepares the fd
 *
--- a/include/libwebsockets/lws-system.h
+++ b/include/libwebsockets/lws-system.h
@ -154,6 +154,11 @@ typedef enum {
 typedef void (*lws_attach_cb_t)(struct lws_context *context, int tsi, void *opaque);
 struct lws_attach_item;

+LWS_EXTERN LWS_VISIBLE int
+lws_tls_jit_trust_got_cert_cb(struct lws_context *cx, void *got_opaque,
+			      const uint8_t *skid, size_t skid_len,
+			      const uint8_t *der, size_t der_len);
+
 typedef struct lws_system_ops {
 	int (*reboot)(void);
 	int (*set_clock)(lws_usec_t us);
@ -186,6 +191,15 @@ typedef struct lws_system_ops {
 	 * held in \p mdata... return 0 to leave the metric object as it is,
 	 * or nonzero to reset it. */

+	int (*jit_trust_query)(struct lws_context *cx, const uint8_t *skid,
+			       size_t skid_len, void *got_opaque);
+	/**< user defined trust store search, if we do trust a cert with SKID
+	 * matching skid / skid_len, then it should get hold of the DER for the
+	 * matching root CA and call
+	 * lws_tls_jit_trust_got_cert_cb(..., got_opaque) before cleaning up and
+	 * returning.  The DER should be destroyed if in heap before returning.
+	 */
+
 	uint32_t	wake_latency_us;
 	/**< time taken for this device to wake from suspend, in us
 	 */
--- a/lib/core-net/client/connect.c
+++ b/lib/core-net/client/connect.c
@ -122,14 +122,20 @@ lws_client_connect_via_info(const struct lws_client_connect_info *i)

 	vh = i->vhost;
 	if (!vh) {
-		vh = i->context->vhost_list;

-		if (!vh) { /* coverity */
-			lwsl_err("%s: no vhost\n", __func__);
-			goto bail;
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+		if (lws_tls_jit_trust_vhost_bind(i->context, i->address, &vh))
+#endif
+		{
+			vh = i->context->vhost_list;
+
+			if (!vh) { /* coverity */
+				lwsl_err("%s: no vhost\n", __func__);
+				goto bail;
+			}
+			if (!strcmp(vh->name, "system"))
+				vh = vh->vhost_next;
 		}
-		if (!strcmp(vh->name, "system"))
-			vh = vh->vhost_next;
 	}

 #if defined(LWS_WITH_SECURE_STREAMS)
--- a/lib/core-net/private-lib-core-net.h
+++ b/lib/core-net/private-lib-core-net.h
@ -472,6 +472,10 @@ struct lws_vhost {

 	const lws_retry_bo_t *retry_policy;

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	lws_sorted_usec_list_t		sul_unref; /* grace period after idle */
+#endif
+
 #if defined(LWS_WITH_SERVER) && defined(LWS_WITH_SECURE_STREAMS)
 	lws_ss_handle_t		*ss_handle; /* ss handle for the server obj */
 #endif
@ -545,6 +549,10 @@ struct lws_vhost {
 	uint8_t created_vhost_protocols:1;
 	uint8_t being_destroyed:1;
 	uint8_t from_ss_policy:1;
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	uint8_t 		grace_after_unref:1;
+	/* grace time / autodelete aoplies to us */
+#endif

 	unsigned char default_protocol_index;
 	unsigned char raw_protocol_index;
--- a/lib/core-net/vhost.c
+++ b/lib/core-net/vhost.c
@ -1296,6 +1296,9 @@ lws_vhost_destroy1(struct lws_vhost *vh)
 		}
 	}
 #endif
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	lws_sul_cancel(&vh->sul_unref);
+#endif

 	lws_vhost_unlock(vh); /* } vh -------------- */

@ -1480,6 +1483,9 @@ __lws_vhost_destroy2(struct lws_vhost *vh)
 #if defined(LWS_WITH_SYS_FAULT_INJECTION)
 	lws_fi_destroy(&vh->fic);
 #endif
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	lws_sul_cancel(&vh->sul_unref);
+#endif

 	__lws_lc_untag(&vh->lc);

--- a/lib/core-net/wsi.c
+++ b/lib/core-net/wsi.c
@ -58,6 +58,14 @@ lws_vhost_bind_wsi(struct lws_vhost *vh, struct lws *wsi)
 		return;
 	lws_context_lock(vh->context, __func__); /* ---------- context { */
 	wsi->a.vhost = vh;
+
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	if (!vh->count_bound_wsi && vh->grace_after_unref) {
+		lwsl_info("%s: %s: in use\n", __func__, vh->lc.gutag);
+		lws_sul_cancel(&vh->sul_unref);
+	}
+#endif
+
 	vh->count_bound_wsi++;
 	lws_context_unlock(vh->context); /* } context ---------- */
 	lwsl_debug("%s: vh %s: wsi %s/%s, count_bound_wsi %d\n", __func__,
@ -72,20 +80,29 @@ lws_vhost_bind_wsi(struct lws_vhost *vh, struct lws *wsi)
 void
 __lws_vhost_unbind_wsi(struct lws *wsi)
 {
-	if (!wsi->a.vhost)
-		return;
+        struct lws_vhost *vh = wsi->a.vhost;
+
+        if (!vh)
+                return;

 	lws_context_assert_lock_held(wsi->a.context);

-	lws_vhost_lock(wsi->a.vhost);
+	lws_vhost_lock(vh);
+
+	assert(vh->count_bound_wsi > 0);
+	vh->count_bound_wsi--;
+
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	if (!vh->count_bound_wsi && vh->grace_after_unref)
+		lws_tls_jit_trust_vh_start_grace(vh);
+#endif

-	assert(wsi->a.vhost->count_bound_wsi > 0);
-	wsi->a.vhost->count_bound_wsi--;
 	lwsl_debug("%s: vh %s: count_bound_wsi %d\n", __func__,
-		   wsi->a.vhost->name, wsi->a.vhost->count_bound_wsi);
+		   vh->name, vh->count_bound_wsi);

-	if (!wsi->a.vhost->count_bound_wsi &&
-	    wsi->a.vhost->being_destroyed) {
+	lws_vhost_unlock(vh);
+
+	if (!vh->count_bound_wsi && vh->being_destroyed)
 		/*
 		 * We have closed all wsi that were bound to this vhost
 		 * by any pt: nothing can be servicing any wsi belonging
@ -93,13 +110,8 @@ __lws_vhost_unbind_wsi(struct lws *wsi)
 		 *
 		 * Finalize the vh destruction... must drop vh lock
 		 */
-		lws_vhost_unlock(wsi->a.vhost);
-		__lws_vhost_destroy2(wsi->a.vhost);
-		wsi->a.vhost = NULL;
-		return;
-	}
+		__lws_vhost_destroy2(vh);

-	lws_vhost_unlock(wsi->a.vhost);
 	wsi->a.vhost = NULL;
 }

--- a/lib/core/context.c
+++ b/lib/core/context.c
@ -613,6 +613,18 @@ lws_create_context(const struct lws_context_creation_info *info)
 #if defined(LWS_WITH_NETWORK)
 	context->event_loop_ops = plev->ops;
 	context->us_wait_resolution = us_wait_resolution;
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	{
+		struct lws_cache_creation_info ci;
+
+		memset(&ci, 0, sizeof(ci));
+		ci.cx = context;
+		ci.ops = &lws_cache_ops_heap;
+		ci.name = "jitt";
+		ci.max_footprint = info->jitt_cache_max_footprint;
+		context->trust_cache = lws_cache_create(&ci);
+	}
+#endif
 #endif
 #if defined(LWS_WITH_EVENT_LIBS)
 	/* at the very end */
@ -631,6 +643,11 @@ lws_create_context(const struct lws_context_creation_info *info)
 #endif
 	context->system_ops = info->system_ops;
 	context->pt_serv_buf_size = (unsigned int)s1;
+	context->protocols_copy = info->protocols;
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	context->vh_idle_grace_ms = info->vh_idle_grace_ms ?
+			info->vh_idle_grace_ms : 5000;
+#endif

 #if defined(LWS_WITH_SYS_FAULT_INJECTION)
 	context->fic.name = "ctx";
@ -2072,6 +2089,11 @@ next:
 		 * clean up the context and things hanging off it
 		 */

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+		lws_cache_destroy(&context->trust_cache);
+		lws_tls_jit_trust_inflight_destroy_all(context);
+#endif
+
 #if defined(LWS_WITH_SYS_SMD)
 		_lws_smd_destroy(context);
 #endif
--- a/lib/core/private-lib-core.h
+++ b/lib/core/private-lib-core.h
@ -445,6 +445,8 @@ struct lws_context {

 	lws_lifecycle_group_t			lcg[LWSLCG_COUNT];

+	const struct lws_protocols		*protocols_copy;
+
 #if defined(LWS_WITH_NETLINK)
 	lws_sorted_usec_list_t			sul_nl_coldplug;
 	/* process can only have one netlink socket, have to do it in ctx */
@ -514,6 +516,12 @@ struct lws_context {

 #if defined(LWS_WITH_TLS)
 	struct lws_context_tls		tls;
+#if defined (LWS_WITH_TLS_JIT_TRUST)
+	lws_dll2_owner_t		jit_inflight;
+	/* ongoing sync or async jit trust lookups */
+	struct lws_cache_ttl_lru	*trust_cache;
+	/* caches host -> truncated trust SKID mappings */
+#endif
 #endif
 #if defined(LWS_WITH_DRIVERS)
 	lws_netdevs_t			netdevs;
@ -685,6 +693,9 @@ struct lws_context {
 	unsigned int max_http_header_pool;
 	int simultaneous_ssl_restriction;
 	int simultaneous_ssl;
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	int		vh_idle_grace_ms;
+#endif
 #if defined(LWS_WITH_PEER_LIMITS)
 	uint32_t pl_hash_elements;	/* protected by context->lock */
 	uint32_t count_peers;		/* protected by context->lock */
--- a/lib/tls/CMakeLists.txt
+++ b/lib/tls/CMakeLists.txt
@ -117,6 +117,10 @@ if (LWS_WITH_SSL)
 		list(APPEND SOURCES
 			tls/tls-sessions.c)
 	endif()
+	if (LWS_WITH_TLS_JIT_TRUST)
+		list(APPEND SOURCES
+			tls/tls-jit-trust.c)
+	endif()
 	
 	if (LWS_WITH_MBEDTLS)
 		list(APPEND SOURCES
@ -127,6 +131,10 @@ if (LWS_WITH_SSL)
 			list(APPEND SOURCES
 				tls/mbedtls/mbedtls-ssl.c)
 		endif()
+		if (LWS_WITH_TLS_JIT_TRUST)
+			list(APPEND SOURCES
+				tls/mbedtls/mbedtls-extensions.c)
+		endif()
 		if (LWS_WITH_TLS_SESSIONS)
 			list(APPEND SOURCES
 				tls/mbedtls/mbedtls-session.c)
--- a/lib/tls/mbedtls/mbedtls-client.c
+++ b/lib/tls/mbedtls/mbedtls-client.c
@ -25,6 +25,49 @@
 #include "private-lib-core.h"
 #include "private-lib-tls-mbedtls.h"

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+
+/*
+ * We get called for each peer certificate that was provided in turn.
+ *
+ * Our job is just to collect the AKID and SKIDs into ssl->kid_chain, and walk
+ * later at verification result time if it failed.
+ *
+ * None of these should be trusted, even if a misconfigured server sends us
+ * his root CA.
+ */
+
+static int
+lws_mbedtls_client_verify_callback(SSL *ssl, mbedtls_x509_crt *x509)
+{
+	union lws_tls_cert_info_results ci;
+
+	/* we reached the max we can hold? */
+
+	if (ssl->kid_chain.count == LWS_ARRAY_SIZE(ssl->kid_chain.akid))
+		return 0;
+
+	/* if not, stash the SKID and AKID into the next kid slot */
+
+	if (!lws_tls_mbedtls_cert_info(x509, LWS_TLS_CERT_INFO_SUBJECT_KEY_ID,
+				       &ci, 0))
+		lws_tls_kid_copy(&ci,
+				 &ssl->kid_chain.skid[ssl->kid_chain.count]);
+
+	if (!lws_tls_mbedtls_cert_info(x509, LWS_TLS_CERT_INFO_AUTHORITY_KEY_ID,
+				       &ci, 0))
+		lws_tls_kid_copy(&ci,
+				 &ssl->kid_chain.akid[ssl->kid_chain.count]);
+
+	ssl->kid_chain.count++;
+
+	// lwsl_notice("%s: %u\n", __func__, ssl->kid_chain.count);
+
+	return 0;
+}
+
+#endif
+
 int
 lws_ssl_client_bio_create(struct lws *wsi)
 {
@ -104,6 +147,10 @@ lws_ssl_client_bio_create(struct lws *wsi)
 	 * use server name indication (SNI), if supported,
 	 * when establishing connection
 	 */
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	SSL_set_verify(wsi->tls.ssl, SSL_VERIFY_PEER,
+			lws_mbedtls_client_verify_callback);
+#endif

 	SSL_set_fd(wsi->tls.ssl, (int)wsi->desc.sockfd);

@ -285,6 +332,10 @@ lws_tls_client_confirm_peer_cert(struct lws *wsi, char *ebuf, size_t ebuf_len)
 		return 0;
 	}

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	if (n == X509_V_ERR_INVALID_CA)
+	    lws_tls_jit_trust_sort_kids(wsi, &wsi->tls.ssl->kid_chain);
+#endif
 	lws_snprintf(ebuf, ebuf_len,
 		"server's cert didn't look good, %s (use_ssl 0x%x) X509_V_ERR = %d: %s\n",
 		type, (unsigned int)wsi->tls.use_ssl, n,
--- a/lib/tls/mbedtls/wrapper/include/internal/ssl_types.h
+++ b/lib/tls/mbedtls/wrapper/include/internal/ssl_types.h
@ -33,6 +33,8 @@

 #include <mbedtls/x509_crt.h>

+#include "private-jit-trust.h"
+
 typedef void SSL_CIPHER;

 typedef void X509_STORE_CTX;
@ -227,6 +229,10 @@ struct ssl_st

    int (*verify_callback) (SSL *, mbedtls_x509_crt *);

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+    lws_tls_kid_chain_t		kid_chain;
+#endif
+
    int rwstate;
    int interrupted_remaining_write;

--- a/lib/tls/openssl/openssl-client.c
+++ b/lib/tls/openssl/openssl-client.c
@ -44,16 +44,56 @@ extern int openssl_websocket_private_data_index,

 #if !defined(USE_WOLFSSL)

+#if 0
+#if defined(LWS_WITH_TLS_JIT_TRUST)
+
+/*
+ * Completion of sync or async JIT trust lookup
+ */
+
+int
+lws_tls_jit_trust_got_cert_cb(void *got_opaque, const uint8_t *der,
+			      size_t der_len)
+{
+	X509 *x = d2i_X509(NULL, &der, (long)der_len);
+	/** !!! this is not safe for async atm */
+	struct lws *wsi = (struct lws *)got_opaque;
+	X509_STORE *xs;
+	int ret = 0;
+
+	if (!x) {
+		lwsl_err("%s: failed\n", __func__);
+		return 1;
+	}
+
+	xs = SSL_CTX_get_cert_store(SSL_get_SSL_CTX(wsi->tls.ssl));
+	if (xs) {
+		if (X509_STORE_add_cert(xs, x) != 1) {
+			lwsl_warn("%s: unable to set trusted CA\n", __func__);
+			ret = 1;
+		} else
+			lwsl_notice("%s: added trusted CA to CTX for next time\n",
+					__func__);
+	} else
+		lwsl_warn("%s: couldn't get cert store\n", __func__);
+
+	X509_free(x);
+
+	return ret;
+}
+#endif
+#endif
+
 static int
 OpenSSL_client_verify_callback(int preverify_ok, X509_STORE_CTX *x509_ctx)
 {
 	SSL *ssl;
-	int n;
+	int n, err = 0;
 	struct lws *wsi;

 	/* keep old behaviour accepting self-signed server certs */
 	if (!preverify_ok) {
-		int err = X509_STORE_CTX_get_error(x509_ctx);
+		err = X509_STORE_CTX_get_error(x509_ctx);

 		if (err != X509_V_OK) {
 			ssl = X509_STORE_CTX_get_ex_data(x509_ctx,
@ -106,6 +146,43 @@ OpenSSL_client_verify_callback(int preverify_ok, X509_STORE_CTX *x509_ctx)
 		return 0;
 	}

+#if defined(LWS_WITH_TLS_JIT_TRUST)
+	if (err == X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY) {
+		union lws_tls_cert_info_results ci;
+		STACK_OF(X509) *x509_stack;
+
+		x509_stack = X509_STORE_CTX_get1_chain(x509_ctx);
+		if (x509_stack) {
+
+			for (n = 0; n < OPENSSL_sk_num((const OPENSSL_STACK *)x509_stack) &&
+				    wsi->tls.kid_chain.count !=
+				     LWS_ARRAY_SIZE(wsi->tls.kid_chain.akid); n++) {
+				X509 *x509 = OPENSSL_sk_value((const OPENSSL_STACK *)x509_stack, n);
+
+				if (!lws_tls_openssl_cert_info(x509,
+					    LWS_TLS_CERT_INFO_SUBJECT_KEY_ID,
+					    &ci, 0))
+					lws_tls_kid_copy(&ci,
+						&wsi->tls.kid_chain.skid[
+						     wsi->tls.kid_chain.count]);
+
+				if (!lws_tls_openssl_cert_info(x509,
+					     LWS_TLS_CERT_INFO_AUTHORITY_KEY_ID,
+					     &ci, 0))
+					lws_tls_kid_copy(&ci,
+						 &wsi->tls.kid_chain.akid[
+						     wsi->tls.kid_chain.count]);
+
+				wsi->tls.kid_chain.count++;
+			}
+
+			sk_X509_pop_free(x509_stack, X509_free);
+		}
+
+		lws_tls_jit_trust_sort_kids(wsi, &wsi->tls.kid_chain);
+	}
+#endif
+
 	n = lws_get_context_protocol(wsi->a.context, 0).callback(wsi,
 			LWS_CALLBACK_OPENSSL_PERFORM_SERVER_CERT_VERIFICATION,
 			x509_ctx, ssl, (unsigned int)preverify_ok);
@ -151,7 +228,6 @@ OpenSSL_client_verify_callback(int preverify_ok, X509_STORE_CTX *x509_ctx)
 }
 #endif

-
 int
 lws_ssl_client_bio_create(struct lws *wsi)
 {
--- a/lib/tls/private-jit-trust.h
+++ b/lib/tls/private-jit-trust.h
@ -0,0 +1,149 @@
+ /*
+ * libwebsockets - small server side websockets and web server implementation
+ *
+ * Copyright (C) 2010 - 2021 Andy Green <andy@warmcat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ *  This is included from private-lib-core.h if LWS_WITH_TLS
+ *
+ * First-party trusted certs are handled outside of JIT Trust, eg, in SS policy.
+ * JIT Trust is used to validate arbitrary connections on demand, without
+ * needing a complete set of CAs in memory.
+ *
+ * Instantiated CA X509s are bound to dedicated SSL_CTX in their own dynamic
+ * vhosts for client connections to use, these are lazily culled when they have
+ * no remaining active connections using them.
+ *
+ *   - check jit trust cache to see if hostname has vhost already
+ *      - if so, use it
+ *      - if not, check jit trust cache to see if we know the trusted kids list,
+ *   - attempt connection
+ *   - remote or local trust blob / store
+ */
+
+#if !defined(__LWS_TLS_PRIVATE_JIT_TRUST_H__)
+#define __LWS_TLS_PRIVATE_JIT_TRUST_H__
+
+/*
+ * Refer to ./READMEs/README.jit-trust.md for blob layout specification
+ */
+
+#define LWS_JIT_TRUST_MAGIC_BE		0x54424c42
+
+enum {
+	LJT_OFS_32_COUNT_CERTS		= 6,
+	LJT_OFS_32_DERLEN		= 0x0c,
+	LJT_OFS_32_SKIDLEN		= 0x10,
+	LJT_OFS_32_SKID			= 0x14,
+	LJT_OFS_END			= 0x18,
+
+	LJT_OFS_DER			= 0x1c,
+};
+
+typedef struct {
+	uint8_t				kid[20];
+	uint8_t				kid_len;
+} lws_tls_kid_t;
+
+typedef struct {
+	lws_tls_kid_t			akid[4];
+	lws_tls_kid_t			skid[4];
+	uint8_t				count;
+} lws_tls_kid_chain_t;
+
+/*
+ * This is used to manage ongoing jit trust lookups for a specific host.  It
+ * collects results and any trusted DER certs until all of them have arrived,
+ * then caches the hostname -> trusted SKIDs mapping, and creates a vhost +
+ * SSL_CTX trusting the certs named after the trusted SKIDs.
+ *
+ * The cert copies and this inflight object are then freed.
+ *
+ * JIT Trust lookups may be async, there may be multiple lookups fired at one
+ * time, and these mappings are not actually related to a wsi lifetime, so these
+ * separate inflight tracking objects are needed.
+ *
+ * These objects only live until all the AKID lookups for the host that created
+ * them complete.
+ */
+
+typedef struct {
+	lws_dll2_t			list;
+
+	lws_tls_kid_t			kid[2];	/* SKID of the der if any */
+	uint8_t				*der[2]; /* temp allocated */
+
+	int				ders;
+
+	uint32_t			tag; /* xor'd from start of SKIDs that
+					      * that contributed certs, so we
+					      * can name the vhost in a way that
+					      * can be regenerated no matter
+					      * the order of SKID results
+					      */
+
+	short				der_len[2];
+
+	char				refcount; /* expected results left */
+
+	/* hostname overcommitted */
+} lws_tls_jit_inflight_t;
+
+/*
+ * These are the items in the jit trust cache, the cache tag is the hostname
+ * and it resolves to one of these if present.  It describes 1 - 3 SKIDs
+ * of trusted CAs needed to validate that host, and a 32-bit tag that is
+ * the first 4 bytes of each valid SKID xor'd together, so you can find any
+ * existing vhost that already has the required trust (independent of the
+ * order they are checked in due to commutative xor).
+ */
+
+typedef struct {
+	lws_tls_kid_t			skids[3];
+	int				count_skids;
+	uint32_t			xor_tag;
+} lws_tls_jit_cache_item_t;
+
+union lws_tls_cert_info_results;
+
+void
+lws_tls_kid_copy(union lws_tls_cert_info_results *ci, lws_tls_kid_t *kid);
+
+int
+lws_tls_kid_cmp(const lws_tls_kid_t *a, const lws_tls_kid_t *b);
+
+int
+lws_tls_jit_trust_sort_kids(struct lws *wsi, lws_tls_kid_chain_t *ch);
+
+void
+lws_tls_jit_trust_inflight_destroy(lws_tls_jit_inflight_t *inf);
+
+void
+lws_tls_jit_trust_inflight_destroy_all(struct lws_context *cx);
+
+int
+lws_tls_jit_trust_vhost_bind(struct lws_context *cx, const char *address,
+			     struct lws_vhost **pvh);
+
+void
+lws_tls_jit_trust_vh_start_grace(struct lws_vhost *vh);
+
+#endif
+
--- a/lib/tls/private-lib-tls.h
+++ b/lib/tls/private-lib-tls.h
@ -30,6 +30,8 @@

 #if defined(LWS_WITH_TLS)

+#include "private-jit-trust.h"
+
 #if defined(USE_WOLFSSL)
 #if defined(USE_OLD_CYASSL)
  #if defined(_WIN32)
--- a/lib/tls/private-network.h
+++ b/lib/tls/private-network.h
@ -79,6 +79,10 @@ struct lws_lws_tls {
 	lws_tls_bio		*client_bio;
 #if defined(LWS_TLS_SYNTHESIZE_CB)
 	lws_sorted_usec_list_t	sul_cb_synth;
+#endif
+#if !defined(LWS_WITH_MBEDTLS) && defined(LWS_WITH_TLS_JIT_TRUST)
+	/* mbedtls has this in the wrapper, since no wsi ptr at validation */
+	lws_tls_kid_chain_t	kid_chain;
 #endif
 	struct lws_dll2		dll_pending_tls;
 	char			err_helper[32];
--- a/lib/tls/tls-jit-trust.c
+++ b/lib/tls/tls-jit-trust.c
@ -0,0 +1,689 @@
+/*
+ * libwebsockets - small server side websockets and web server implementation
+ *
+ * Copyright (C) 2010 - 2021 Andy Green <andy@warmcat.com>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to
+ * deal in the Software without restriction, including without limitation the
+ * rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+ * sell copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ */
+
+#include "private-lib-core.h"
+
+void
+lws_tls_kid_copy(union lws_tls_cert_info_results *ci, lws_tls_kid_t *kid)
+{
+
+	/*
+	 * KIDs all seem to be 20 bytes / SHA1 or less.  If we get one that
+	 * is bigger, treat only the first 20 bytes as significant.
+	 */
+
+	if ((size_t)ci->ns.len > sizeof(kid->kid))
+		kid->kid_len = sizeof(kid->kid);
+	else
+		kid->kid_len = (uint8_t)ci->ns.len;
+
+	memcpy(kid->kid, ci->ns.name, kid->kid_len);
+}
+
+void
+lws_tls_kid_copy_kid(lws_tls_kid_t *kid, const lws_tls_kid_t *src)
+{
+	int klen = sizeof(kid->kid);
+
+	if (src->kid_len < klen)
+		klen = src->kid_len;
+
+	kid->kid_len = (uint8_t)klen;
+
+	memcpy(kid->kid, src->kid, (size_t)klen);
+}
+
+int
+lws_tls_kid_cmp(const lws_tls_kid_t *a, const lws_tls_kid_t *b)
+{
+	if (a->kid_len != b->kid_len)
+		return 1;
+
+	return memcmp(a->kid, b->kid, a->kid_len);
+}
+
+/*
+ * We have the SKID and AKID for every peer cert captured, but they may be
+ * in any order, and eg, falsely have sent the root CA, or an attacker may
+ * send unresolveable self-referencing loops of KIDs.
+ *
+ * Let's sort them into the SKID -> AKID hierarchy, so the last entry is the
+ * server cert and the first entry is the highest parent that the server sent.
+ * Normally the top one will be an intermediate, and its AKID is the ID of the
+ * root CA cert we would need to trust to validate the chain.
+ *
+ * It's not unknown the server is misconfigured to also send the root CA, if so
+ * the top slot's AKID is empty and we should look for its SKID in the trust
+ * blob.
+ *
+ * If we return 0, we succeeded and the AKID of ch[0] is the SKID we want to see
+ * try to import from the trust blob.
+ *
+ * If we return nonzero, we can't identify what we want and should abandon the
+ * connection.
+ */
+
+int
+lws_tls_jit_trust_sort_kids(struct lws *wsi, lws_tls_kid_chain_t *ch)
+{
+	size_t hl;
+	lws_tls_jit_inflight_t *inf;
+	int n, m, sanity = 10;
+	const char *host = wsi->cli_hostname_copy;
+	char more = 1;
+
+	lwsl_info("%s\n", __func__);
+
+	if (!host) {
+		if (wsi->stash && wsi->stash->cis[CIS_HOST])
+			host = wsi->stash->cis[CIS_HOST];
+#if defined(LWS_ROLE_H1) || defined(LWS_ROLE_H2)
+		else
+			host = lws_hdr_simple_ptr(wsi,
+					      _WSI_TOKEN_CLIENT_PEER_ADDRESS);
+	}
+#endif
+	if (!host)
+		return 1;
+
+	hl = strlen(host);
+
+	/* something to work with? */
+
+	if (!ch->count)
+		return 1;
+
+	/* do we need to sort? */
+
+	if (ch->count > 1) {
+
+		/* okie... */
+
+		while (more) {
+
+			if (!sanity--)
+				/* let's not get fooled into spinning */
+				return 1;
+
+			more = 0;
+			for (n = 0; n < ch->count - 1; n++) {
+
+				if (!lws_tls_kid_cmp(&ch->skid[n],
+						     &ch->akid[n + 1]))
+					/* next belongs with this one */
+					continue;
+
+				/*
+				 * next doesn't belong with this one, let's
+				 * try to figure out where this one does belong
+				 * then
+				 */
+
+				for (m = 0; m < ch->count; m++) {
+					if (n == m)
+						continue;
+					if (!lws_tls_kid_cmp(&ch->skid[n],
+							     &ch->akid[m])) {
+						lws_tls_kid_t t;
+
+						/*
+						 * m references us, so we
+						 * need to go one step above m,
+						 * swap m and n
+						 */
+
+						more = 1;
+						t = ch->akid[m];
+						ch->akid[m] = ch->akid[n];
+						ch->akid[n] = t;
+						t = ch->skid[m];
+						ch->skid[m] = ch->skid[n];
+						ch->skid[n] = t;
+
+						break;
+					}
+				}
+
+				if (more)
+					break;
+			}
+		}
+
+		/* then we should be sorted */
+	}
+
+	for (n = 0; n < ch->count; n++) {
+		lwsl_info("%s: AKID[%d]\n", __func__, n);
+		lwsl_hexdump_info(ch->akid[n].kid, ch->akid[n].kid_len);
+		lwsl_info("%s: SKID[%d]\n", __func__, n);
+		lwsl_hexdump_info(ch->skid[n].kid, ch->skid[n].kid_len);
+	}
+
+	/* to go further, user must provide a lookup helper */
+
+	if (!wsi->a.context->system_ops ||
+	    !wsi->a.context->system_ops->jit_trust_query)
+		return 1;
+
+	/*
+	 * If there's already a pending lookup for this host, let's bail and
+	 * just wait for that to complete (since it will be done async if we
+	 * can see it)
+	 */
+
+	lws_start_foreach_dll(struct lws_dll2 *, d,
+			      wsi->a.context->jit_inflight.head) {
+		inf = lws_container_of(d, lws_tls_jit_inflight_t, list);
+
+		if (!strcmp((const char *)&inf[1], host))
+			/* already being handled */
+			return 1;
+
+	} lws_end_foreach_dll(d);
+
+	/*
+	 * No... let's make an inflight entry for this host, then
+	 */
+
+	inf = lws_zalloc(sizeof(*inf) + hl + 1, __func__);
+	if (!inf)
+		return 1;
+
+	memcpy(&inf[1], host, hl + 1);
+	inf->refcount = (char)ch->count;
+	lws_dll2_add_tail(&inf->list, &wsi->a.context->jit_inflight);
+
+	/*
+	 * ...kid_chain[0] AKID should indicate the right CA SKID that we want.
+	 *
+	 * Because of cross-signing, we check all of them and accept we may get
+	 * multiple (the inflight accepts up to 2) CAs needed.
+	 */
+
+	for (n = 0; n < ch->count; n++)
+		wsi->a.context->system_ops->jit_trust_query(wsi->a.context,
+			ch->akid[n].kid, (size_t)ch->akid[n].kid_len,
+			(void *)inf);
+
+	return 0;
+}
+
+static void
+tag_to_vh_name(char *result, size_t max, uint32_t tag)
+{
+	lws_snprintf(result, max, "jitt-%08X", tag);
+}
+
+int
+lws_tls_jit_trust_vhost_bind(struct lws_context *cx, const char *address,
+			     struct lws_vhost **pvh)
+{
+	lws_tls_jit_cache_item_t *ci, jci;
+	lws_tls_jit_inflight_t *inf;
+	char vhtag[32];
+	size_t size;
+	int n;
+
+	if (lws_cache_item_get(cx->trust_cache, address, (const void **)&ci,
+									&size))
+		/*
+		 * There's no cached info, we have to start from scratch on
+		 * this one
+		 */
+		return 1;
+
+	/* gotten cache item may be evicted by jit_trust_query */
+	jci = *ci;
+
+	/*
+	 * We have some trust cache information for this host already, it tells
+	 * us the trusted CA SKIDs we found before, and the xor tag used to name
+	 * the vhost configured for these trust CAs in its SSL_CTX.
+	 *
+	 * Let's check first if the correct prepared vhost already exists, if
+	 * so, we can just bind to that and go.
+	 */
+
+	tag_to_vh_name(vhtag, sizeof(vhtag), jci.xor_tag);
+
+	*pvh = lws_get_vhost_by_name(cx, vhtag);
+	if (*pvh) {
+		lwsl_info("%s: %s -> existing %s\n", __func__, address, vhtag);
+		/* hit, let's just use that then */
+		return 0;
+	}
+
+	/*
+	 * ... so, we know the SKIDs of the missing CAs, but we don't have the
+	 * DERs for them, and so no configured vhost trusting them yet.  We have
+	 * had the DERs at some point, but we can't afford to cache them, so
+	 * we will have to get them again.
+	 *
+	 * Let's make an inflight for this, it will create the vhost when it
+	 * completes.  If syncrhronous, then it will complete before we leave
+	 * here, otherwise it will have a life of its own until all the
+	 * queries use the cb to succeed or fail.
+	 */
+
+	size = strlen(address);
+	inf = lws_zalloc(sizeof(*inf) + size + 1, __func__);
+	if (!inf)
+		return 1;
+
+	memcpy(&inf[1], address, size + 1);
+	inf->refcount = (char)jci.count_skids;
+	lws_dll2_add_tail(&inf->list, &cx->jit_inflight);
+
+	/*
+	 * ...kid_chain[0] AKID should indicate the right CA SKID that we want.
+	 *
+	 * Because of cross-signing, we check all of them and accept we may get
+	 * multiple (we can handle 3) CAs needed.
+	 */
+
+	for (n = 0; n < jci.count_skids; n++)
+		cx->system_ops->jit_trust_query(cx, jci.skids[n].kid,
+						(size_t)jci.skids[n].kid_len,
+						(void *)inf);
+
+	/* ... in case synchronous and it already finished the queries */
+
+	*pvh = lws_get_vhost_by_name(cx, vhtag);
+	if (*pvh) {
+		/* hit, let's just use that then */
+		lwsl_info("%s: bind to created vhost %s\n", __func__, vhtag);
+		return 0;
+	} else
+		lwsl_err("%s: unable to bind to %s\n", __func__, vhtag);
+
+	/* right now, nothing to offer */
+
+	return 1;
+}
+
+void
+lws_tls_jit_trust_inflight_destroy(lws_tls_jit_inflight_t *inf)
+{
+	int n;
+
+	for (n = 0; n < inf->ders; n++)
+		lws_free_set_NULL(inf->der[n]);
+	lws_dll2_remove(&inf->list);
+
+	lws_free(inf);
+}
+
+static int
+inflight_destroy(struct lws_dll2 *d, void *user)
+{
+	lws_tls_jit_inflight_t *inf;
+
+	inf = lws_container_of(d, lws_tls_jit_inflight_t, list);
+
+	lws_tls_jit_trust_inflight_destroy(inf);
+
+	return 0;
+}
+
+void
+lws_tls_jit_trust_inflight_destroy_all(struct lws_context *cx)
+{
+	lws_dll2_foreach_safe(&cx->jit_inflight, cx, inflight_destroy);
+}
+
+static void
+unref_vh_grace_cb(lws_sorted_usec_list_t *sul)
+{
+	struct lws_vhost *vh = lws_container_of(sul, struct lws_vhost,
+						sul_unref);
+
+	lwsl_info("%s: %s\n", __func__, vh->lc.gutag);
+
+	lws_vhost_destroy(vh);
+}
+
+void
+lws_tls_jit_trust_vh_start_grace(struct lws_vhost *vh)
+{
+	lwsl_info("%s: %s: unused, grace %dms\n", __func__, vh->lc.gutag,
+			vh->context->vh_idle_grace_ms);
+	lws_sul_schedule(vh->context, 0, &vh->sul_unref, unref_vh_grace_cb,
+			 (lws_usec_t)vh->context->vh_idle_grace_ms *
+								LWS_US_PER_MS);
+}
+
+#if defined(_DEBUG)
+static void
+lws_tls_jit_trust_cert_info(const uint8_t *der, size_t der_len)
+{
+	struct lws_x509_cert *x;
+	union lws_tls_cert_info_results *u;
+	char p = 0, buf[192 + sizeof(*u)];
+
+	if (lws_x509_create(&x))
+		return;
+
+	if (!lws_x509_parse_from_pem(x, der, der_len)) {
+
+		u = (union lws_tls_cert_info_results *)buf;
+
+		if (!lws_x509_info(x, LWS_TLS_CERT_INFO_ISSUER_NAME, u, 192)) {
+			lwsl_info("ISS: %s\n", u->ns.name);
+			p = 1;
+		}
+		if (!lws_x509_info(x, LWS_TLS_CERT_INFO_COMMON_NAME, u, 192)) {
+			lwsl_info("CN: %s\n", u->ns.name);
+			p = 1;
+		}
+
+		if (!p) {
+			lwsl_err("%s: unable to get any info\n", __func__);
+			lwsl_hexdump_err(der, der_len);
+		}
+	} else
+		lwsl_err("%s: unable to load DER\n", __func__);
+
+	lws_x509_destroy(&x);
+}
+#endif
+
+/*
+ * This processes the JIT Trust lookup results independent of the tls backend.
+ */
+
+int
+lws_tls_jit_trust_got_cert_cb(struct lws_context *cx, void *got_opaque,
+			      const uint8_t *skid, size_t skid_len,
+			      const uint8_t *der, size_t der_len)
+{
+	lws_tls_jit_inflight_t *inf = (lws_tls_jit_inflight_t *)got_opaque;
+	struct lws_context_creation_info info;
+	lws_tls_jit_cache_item_t jci;
+	struct lws_vhost *v;
+	char vhtag[20];
+	char hit = 0;
+	int n;
+
+	/*
+	 * Before anything else, check the inf is still valid.  In the low
+	 * probability but possible case it was reallocated to be a different
+	 * inflight, that may cause different CA certs to apply to a connection,
+	 * but since mbedtls will then validate the server cert using the wrong
+	 * trusted CA, it will just cause temporary conn fail.
+	 */
+
+	lws_start_foreach_dll(struct lws_dll2 *, e, cx->jit_inflight.head) {
+		lws_tls_jit_inflight_t *i = lws_container_of(e,
+						lws_tls_jit_inflight_t, list);
+		if (i == inf) {
+			hit = 1;
+			break;
+		}
+
+	} lws_end_foreach_dll(e);
+
+	if (!hit)
+		/* inf has already gone */
+		return 1;
+
+	inf->refcount--;
+
+	if (skid_len >= 4)
+		inf->tag ^= *((uint32_t *)skid);
+
+	if (der && inf->ders < (int)LWS_ARRAY_SIZE(inf->der) && inf->refcount) {
+		/*
+		 * We have a trusted CA, but more results coming... stash it
+		 * in heap.
+		 */
+
+		inf->kid[inf->ders].kid_len = (uint8_t)((skid_len >
+				     (uint8_t)sizeof(inf->kid[inf->ders].kid)) ?
+				     sizeof(inf->kid[inf->ders].kid) : skid_len);
+		memcpy(inf->kid[inf->ders].kid, skid,
+		       inf->kid[inf->ders].kid_len);
+
+		inf->der[inf->ders] = lws_malloc(der_len, __func__);
+		if (!inf->der[inf->ders])
+			return 1;
+		memcpy(inf->der[inf->ders], der, der_len);
+		inf->der_len[inf->ders] = (short)der_len;
+		inf->ders++;
+
+		return 0;
+	}
+
+	/*
+	 * We accept up to three valid CA, and then end the inflight early.
+	 * Any further pending results are dropped, since we got all we could
+	 * use.  Up to two valid CA would be held in the inflight and the other
+	 * provided in the params.
+	 *
+	 * If we did not already fill up the inflight, keep waiting for any
+	 * others expected
+	 */
+
+	if (inf->refcount && inf->ders < (int)LWS_ARRAY_SIZE(inf->der))
+		return 0;
+
+	if (!der && !inf->ders) {
+		lwsl_warn("%s: no trusted CA certs matching\n", __func__);
+
+		goto destroy_inf;
+	}
+
+	tag_to_vh_name(vhtag, sizeof(vhtag), inf->tag);
+
+	/*
+	 * We have got at least one CA, it's all the CAs we're going to get,
+	 * or that we can handle.  So we have to process and drop the inf.
+	 *
+	 * First let's make a cache entry with a shortish ttl, mapping the
+	 * hostname we were trying to connect to, to the SKIDs that actually
+	 * had trust results.  This may come in handy later when we want to
+	 * connect to the same host again, but any vhost from before has been
+	 * removed... we can just ask for the specific CAs to regenerate the
+	 * vhost, without having to first fail the connection attempt to get the
+	 * server cert.
+	 *
+	 * The cache entry can be evicted at any time, so it is selfcontained.
+	 * If it's also lost, we start over with the initial failing connection
+	 * to figure out what we need to make it work.
+	 */
+
+	memset(&jci, 0, sizeof(jci));
+
+	jci.xor_tag = inf->tag;
+
+	/* copy the SKIDs from the inflight and params into the cache item */
+
+	for (n = 0; n < (int)LWS_ARRAY_SIZE(inf->der); n++)
+		if (inf->kid[n].kid_len)
+			lws_tls_kid_copy_kid(&jci.skids[jci.count_skids++],
+						&inf->kid[n]);
+
+	if (skid_len) {
+		if (skid_len > sizeof(inf->kid[0].kid))
+			skid_len = sizeof(inf->kid[0].kid);
+		jci.skids[jci.count_skids].kid_len = (uint8_t)skid_len;
+		memcpy(jci.skids[jci.count_skids++].kid, skid, skid_len);
+	}
+
+	lwsl_info("%s: adding cache mapping %s -> %s\n", __func__,
+			(const char *)&inf[1], vhtag);
+
+	if (lws_cache_write_through(cx->trust_cache, (const char *)&inf[1],
+				    (const uint8_t *)&jci, sizeof(jci),
+				    lws_now_usecs() + (3600ll *LWS_US_PER_SEC),
+				    NULL))
+		lwsl_warn("%s: add to cache failed\n", __func__);
+
+	/* is there already a vhost for this commutative-xor SKID trust? */
+
+	if (lws_get_vhost_by_name(cx, vhtag)) {
+		lwsl_info("%s: tag vhost %s already exists, skipping\n",
+				__func__, vhtag);
+		goto destroy_inf;
+	}
+
+	/*
+	 * We only end up here when we attempted a connection to this hostname.
+	 *
+	 * We have the identified CA trust DER(s) to hand, let's create the
+	 * necessary vhost + prepared SSL_CTX for it to use on the retry, it
+	 * will be used straight away if the retry comes before the idle vhost
+	 * timeout.
+	 *
+	 * We also use this path in the case we have the cache entry but no
+	 * matching vhost already existing, to create one.
+	 */
+
+	memset(&info, 0, sizeof(info));
+	info.vhost_name = vhtag;
+	info.port = CONTEXT_PORT_NO_LISTEN;
+	info.options = cx->options;
+
+	/*
+	 * We have to create the vhost with the first valid trusted DER...
+	 * if we have a params one, use that so the rest are all from inflight
+	 */
+
+	if (der) {
+		info.client_ssl_ca_mem = der;
+		info.client_ssl_ca_mem_len = (unsigned int)der_len;
+		n = 0;
+	} else {
+		info.client_ssl_ca_mem = inf->der[0];
+		info.client_ssl_ca_mem_len = (unsigned int)inf->der_len[0];
+		n = 1;
+	}
+
+#if defined(_DEBUG)
+	lws_tls_jit_trust_cert_info(info.client_ssl_ca_mem,
+				    info.client_ssl_ca_mem_len);
+#endif
+
+	info.protocols = cx->protocols_copy;
+
+	v = lws_create_vhost(cx, &info);
+	if (!v)
+		lwsl_err("%s: failed to create vh %s\n", __func__, vhtag);
+
+	v->grace_after_unref = 1;
+	lws_tls_jit_trust_vh_start_grace(v);
+
+	/*
+	 * Do we need to add more trusted certs from inflight?
+	 */
+
+	while (n < inf->ders) {
+
+#if defined(_DEBUG)
+		lws_tls_jit_trust_cert_info(inf->der[n],
+					    (size_t)inf->der_len[n]);
+#endif
+
+		if (lws_tls_client_vhost_extra_cert_mem(v, inf->der[n],
+						(size_t)inf->der_len[n]))
+			lwsl_err("%s: add extra cert failed\n", __func__);
+		n++;
+	}
+
+	lwsl_info("%s: created jitt %s -> vh %s\n", __func__,
+				(const char *)&inf[1], vhtag);
+
+destroy_inf:
+	lws_tls_jit_trust_inflight_destroy(inf);
+
+	return 0;
+}
+
+/*
+ * Refer to ./READMEs/README.jit-trust.md for blob layout specification
+ */
+
+int
+lws_tls_jit_trust_blob_queury_skid(const void *_blob, size_t blen,
+				   const uint8_t *skid, size_t skid_len,
+				   const uint8_t **prpder, size_t *prder_len)
+{
+	const uint8_t *pskidlen, *pskids, *pder, *blob = (uint8_t *)_blob;
+	const uint16_t *pderlen;
+	int certs;
+
+	/* sanity check blob length and magic */
+
+	if (blen < 32768 ||
+	   lws_ser_ru32be(blob) != LWS_JIT_TRUST_MAGIC_BE ||
+	   lws_ser_ru32be(blob + LJT_OFS_END) != blen) {
+		lwsl_err("%s: blob not sane\n", __func__);
+
+		return -1;
+	}
+
+	if (!skid_len)
+		return 1;
+
+	/* point into the various sub-tables */
+
+	certs		= (int)lws_ser_ru16be(blob + LJT_OFS_32_COUNT_CERTS);
+
+	pderlen		= (uint16_t *)(blob + lws_ser_ru32be(blob +
+							LJT_OFS_32_DERLEN));
+	pskidlen	= blob + lws_ser_ru32be(blob + LJT_OFS_32_SKIDLEN);
+	pskids		= blob + lws_ser_ru32be(blob + LJT_OFS_32_SKID);
+	pder		= blob + LJT_OFS_DER;
+
+	/* check each cert SKID in turn, return the DER if found */
+
+	while (certs--) {
+
+		/* paranoia / sanity */
+
+		assert(pskids < blob + blen);
+		assert(pder < blob + blen);
+		assert(pskidlen < blob + blen);
+		assert((uint8_t *)pderlen < blob + blen);
+
+		/* we will accept to match on truncated SKIDs */
+
+		if (*pskidlen >= skid_len &&
+		    !memcmp(skid, pskids, skid_len)) {
+			/*
+			 * We found a trusted CA cert of the right SKID
+			 */
+		        *prpder = pder;
+		        *prder_len = lws_ser_ru16be((uint8_t *)pderlen);
+
+		        return 0;
+		}
+
+		pder += lws_ser_ru16be((uint8_t *)pderlen);
+		pskids += *pskidlen;
+		pderlen++;
+		pskidlen++;
+	}
+
+	return 1;
+}
--- a/lib/tls/tls-sessions.c
+++ b/lib/tls/tls-sessions.c
@ -46,7 +46,7 @@ lws_tls_session_tag_from_wsi(struct lws *wsi, char *buf, size_t len)
 		return 1;

 	if (!wsi->stash) {
-		lwsl_warn("%s: wsi has no stash\n", __func__);
+		lwsl_info("%s: wsi has no stash\n", __func__);
 		return 1;
 	}

--- a/minimal-examples/http-client/minimal-http-client-jit-trust/CMakeLists.txt
+++ b/minimal-examples/http-client/minimal-http-client-jit-trust/CMakeLists.txt
@ -0,0 +1,157 @@
+project(lws-minimal-http-client-jit-trust C)
+cmake_minimum_required(VERSION 2.8.12)
+find_package(libwebsockets CONFIG REQUIRED)
+list(APPEND CMAKE_MODULE_PATH ${LWS_CMAKE_DIR})
+include(CheckCSourceCompiles)
+include(LwsCheckRequirements)
+
+set(SAMP lws-minimal-http-client-jit-trust)
+set(SRCS minimal-http-client.c)
+
+set(has_fault_injection 1)
+set(has_h2 1)
+set(has_plugins 1)
+set(has_ss_policy_parse 1)
+set(has_no_system_vhost 1)
+set(has_async_dns 1)
+
+set(requirements 1)
+
+require_lws_config(LWS_ROLE_H1 1 requirements)
+require_lws_config(LWS_WITH_CLIENT 1 requirements)
+require_lws_config(LWS_WITH_SYS_STATE 1 requirements)
+require_lws_config(LWS_WITH_TLS_JIT_TRUST 1 requirements)
+
+require_lws_config(LWS_ROLE_H2 1 has_h2)
+require_lws_config(LWS_WITH_SYS_FAULT_INJECTION 1 has_fault_injection)
+require_lws_config(LWS_WITH_EVLIB_PLUGINS 1 has_plugins)
+require_lws_config(LWS_WITH_EVENT_LIBS 1 has_plugins)
+
+require_lws_config(LWS_WITH_SECURE_STREAMS 1 has_ss_policy_parse)
+require_lws_config(LWS_WITH_SECURE_STREAMS_STATIC_POLICY_ONLY 0 has_ss_policy_parse)
+
+require_lws_config(LWS_WITH_SYS_ASYNC_DNS 0 has_no_system_vhost)
+require_lws_config(LWS_WITH_SYS_NTPCLIENT 0 has_no_system_vhost)
+require_lws_config(LWS_WITH_SYS_DHCP_CLIENT 0 has_no_system_vhost)
+
+require_lws_config(LWS_WITH_SYS_ASYNC_DNS 1 has_async_dns)
+
+if (requirements)
+	add_executable(${SAMP} ${SRCS})
+
+	find_program(VALGRIND "valgrind")
+	
+	sai_resource(warmcat_conns 1 40 http_client_warmcat)
+
+	if (LWS_CTEST_INTERNET_AVAILABLE)
+		set(mytests http-client-warmcat-h1)
+		if (has_h2)
+			add_test(NAME http-client-warmcat COMMAND lws-minimal-http-client )
+			list(APPEND mytests http-client-warmcat)
+		endif()
+
+
+		add_test(NAME http-client-warmcat-h1 COMMAND lws-minimal-http-client  --h1)
+				     
+		if (has_fault_injection)
+		
+			# creation related faults
+		
+			list(APPEND mytests http-client-fi-ctx1)
+			add_test(NAME http-client-fi-ctx1 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail1")
+			
+			# if (has_plugins)
+			# !!! need to actually select an available evlib plugin to trigger this
+			#	list(APPEND mytests http-client-fi-pi)
+			#	add_test(NAME http-client-fi-pi COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_plugin_init")
+			# endif()
+
+			list(APPEND mytests http-client-fi-ctx2)
+			add_test(NAME http-client-fi-ctx2 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_evlib_sel")
+			
+			list(APPEND mytests http-client-fi-ctx3)
+			add_test(NAME http-client-fi-ctx3 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_oom_ctx")
+			
+			list(APPEND mytests http-client-fi-ctx4)
+			add_test(NAME http-client-fi-ctx4 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_privdrop")
+
+			list(APPEND mytests http-client-fi-ctx5)
+			add_test(NAME http-client-fi-ctx5 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_maxfds")
+			
+			list(APPEND mytests http-client-fi-ctx6)
+			add_test(NAME http-client-fi-ctx6 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_oom_fds")
+			
+			list(APPEND mytests http-client-fi-ctx7)
+			add_test(NAME http-client-fi-ctx7 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_plat_init")
+			
+			list(APPEND mytests http-client-fi-ctx8)
+			add_test(NAME http-client-fi-ctx8 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_evlib_init")
+			
+			list(APPEND mytests http-client-fi-ctx9)
+			add_test(NAME http-client-fi-ctx9 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_evlib_pt")
+			
+			if (NOT has_no_system_vhost)
+			
+				list(APPEND mytests http-client-fi-ctx10)
+				add_test(NAME http-client-fi-ctx10 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_sys_vh")
+			
+				list(APPEND mytests http-client-fi-ctx11)
+				add_test(NAME http-client-fi-ctx11 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_sys_vh_init")
+				
+			endif()
+			
+			list(APPEND mytests http-client-fi-ctx12)
+			add_test(NAME http-client-fi-ctx12 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "ctx_createfail_def_vh")
+			
+
+			list(APPEND mytests http-client-fi-vh1)
+			add_test(NAME http-client-fi-vh1 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "vh/vh_create_oom")
+			
+			list(APPEND mytests http-client-fi-vh2)
+			add_test(NAME http-client-fi-vh2 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "vh/vh_create_pcols_oom")
+			
+			list(APPEND mytests http-client-fi-vh3)
+			add_test(NAME http-client-fi-vh3 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "vh/vh_create_ssl_srv")
+
+			list(APPEND mytests http-client-fi-vh4)
+			add_test(NAME http-client-fi-vh4 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "vh/vh_create_ssl_cli")
+			
+			list(APPEND mytests http-client-fi-vh5)
+			add_test(NAME http-client-fi-vh5 COMMAND lws-minimal-http-client --expected-exit 5 --fault-injection "vh/vh_create_srv_init")
+
+
+			list(APPEND mytests http-client-fi-dnsfail)
+			add_test(NAME http-client-fi-dnsfail COMMAND lws-minimal-http-client --expected-exit 3 --fault-injection "wsi=user/dnsfail")
+			
+			if (has_async_dns)
+				list(APPEND mytests http-client-fi-connfail)
+				add_test(NAME http-client-fi-connfail COMMAND lws-minimal-http-client --expected-exit 3 --fault-injection "wsi=user/connfail")
+			else()
+				list(APPEND mytests http-client-fi-connfail)
+				add_test(NAME http-client-fi-connfail COMMAND lws-minimal-http-client --expected-exit 2 --fault-injection "wsi=user/connfail")
+			endif()
+			
+			list(APPEND mytests http-client-fi-user-est-fail)
+			add_test(NAME http-client-fi-user-est-fail COMMAND lws-minimal-http-client --expected-exit 3 --fault-injection "wsi/user_reject_at_est")	
+		
+			
+		endif()
+		
+		set_tests_properties(${mytests} PROPERTIES
+		     WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-client/minimal-http-client
+		     TIMEOUT 20)
+				     
+    		if (DEFINED ENV{SAI_OVN})
+			set_tests_properties(${mytests} PROPERTIES
+					     FIXTURES_REQUIRED "res_http_client_warmcat")
+		endif()		
+	
+	endif()
+
+	if (websockets_shared)
+		target_link_libraries(${SAMP} websockets_shared ${LIBWEBSOCKETS_DEP_LIBS})
+		add_dependencies(${SAMP} websockets_shared)
+	else()
+		target_link_libraries(${SAMP} websockets ${LIBWEBSOCKETS_DEP_LIBS})
+	endif()
+endif()
--- a/minimal-examples/http-client/minimal-http-client-jit-trust/README.md
+++ b/minimal-examples/http-client/minimal-http-client-jit-trust/README.md
@ -0,0 +1,96 @@
+# lws minimal http client JIT Trust
+
+This example turns off any existing trusted CAs and then tries to connect to a server, by default, warmcat.com.
+
+It validates the remote certificates using trusted CAs from a JIT Trust blob compiled into the code.
+
+## build
+
+```
+ $ cmake . && make
+```
+
+## usage
+
+Commandline option|Meaning
+---|---
+-d <loglevel>|Debug verbosity in decimal, eg, -d15
+-l| Connect to https://localhost:7681 and accept selfsigned cert
+--h1|Specify http/1.1 only using ALPN, rejects h2 even if server supports it
+--server <name>|set server name to connect to
+-k|Apply tls option LCCSCF_ALLOW_INSECURE
+-j|Apply tls option LCCSCF_ALLOW_SELFSIGNED
+-m|Apply tls option LCCSCF_SKIP_SERVER_CERT_HOSTNAME_CHECK
+-e|Apply tls option LCCSCF_ALLOW_EXPIRED
+-v|Connection validity use 3s / 10s instead of default 5m / 5m10s
+--nossl| disable ssl connection
+--user <username>| Set Basic Auth username
+--password <password> | Set Basic Auth password
+
+```
+ $ ./bin/lws-minimal-http-client-jit-trust --h1 --server ebay.com --path /
+==1302866== 
+[2021/06/17 14:33:54:7500] U: LWS minimal http client JIT Trust [-d<verbosity>] [-l] [--h1]
+[2021/06/17 14:33:54:7956] N: LWS: 4.2.99-v4.2.0-70-g80e7e39bae, loglevel 1031
+[2021/06/17 14:33:54:7960] N: NET CLI SRV H1 H2 WS MbedTLS ConMon IPv6-absent
+[2021/06/17 14:33:54:8165] N:  ++ [wsi|0|pipe] (1)
+[2021/06/17 14:33:54:8227] N:  ++ [vh|0|netlink] (1)
+[2021/06/17 14:33:54:8319] N:  ++ [vh|1|default||-1] (2)
+[2021/06/17 14:33:55:0107] N:  ++ [wsicli|0|GET/h1/ebay.com] (1)
+[2021/06/17 14:33:56:0291] N:  ++ [vh|2|jitt-7F69A044||-1] (3)
+[2021/06/17 14:33:56:0355] E: CLIENT_CONNECTION_ERROR: server's cert didn't look good, invalidca (use_ssl 0x20000061) X509_V_ERR = 24: CA is not trusted
+
+[2021/06/17 14:33:56:0376] N:  ++ [wsicli|1|GET/h1/ebay.com] (2)
+[2021/06/17 14:33:56:0746] N:  -- [wsicli|0|GET/h1/ebay.com] (1) 1.061s
+[2021/06/17 14:33:56:7555] N: lws_client_reset: REDIRECT www.ebay.com:443, path='/', ssl = 1, alpn='http/1.1'
+[2021/06/17 14:33:57:0205] N:  ++ [vh|3|jitt-DFF2B5B4||-1] (4)
+[2021/06/17 14:33:57:0208] E: CLIENT_CONNECTION_ERROR: server's cert didn't look good, invalidca (use_ssl 0x1) X509_V_ERR = 24: CA is not trusted
+
+[2021/06/17 14:33:57:0210] N:  ++ [wsicli|2|GET/h1/ebay.com] (2)
+[2021/06/17 14:33:57:0288] N:  -- [wsicli|1|GET/h1/ebay.com] (1) 991.119ms
+[2021/06/17 14:33:57:7528] N: lws_client_reset: REDIRECT www.ebay.com:443, path='/', ssl = 1, alpn='http/1.1'
+[2021/06/17 14:33:58:1564] U: Connected to 195.95.193.127, http response: 200
+[2021/06/17 14:33:58:1637] U: RECEIVE_CLIENT_HTTP_READ: read 209
+[2021/06/17 14:33:58:1796] U: RECEIVE_CLIENT_HTTP_READ: read 197
+[2021/06/17 14:33:58:1822] U: RECEIVE_CLIENT_HTTP_READ: read 1014
+[2021/06/17 14:33:58:1847] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:1851] U: RECEIVE_CLIENT_HTTP_READ: read 1022
+[2021/06/17 14:33:58:2748] U: RECEIVE_CLIENT_HTTP_READ: read 242
+[2021/06/17 14:33:58:2782] U: RECEIVE_CLIENT_HTTP_READ: read 1014
+[2021/06/17 14:33:58:2784] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:2785] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+...
+[2021/06/17 14:33:58:4661] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4662] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4663] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4664] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4665] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4666] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4667] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4668] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4669] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4670] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4671] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4672] U: RECEIVE_CLIENT_HTTP_READ: read 1024
+[2021/06/17 14:33:58:4673] U: RECEIVE_CLIENT_HTTP_READ: read 286
+[2021/06/17 14:33:58:4690] U: LWS_CALLBACK_COMPLETED_CLIENT_HTTP
+[2021/06/17 14:33:58:4712] E: main: destroying context, interrupted = 1
+[2021/06/17 14:33:58:4774] N:  -- [wsi|0|pipe] (0) 3.661s
+[2021/06/17 14:33:58:4780] N: callback_http: LWS_CALLBACK_CLOSED_CLIENT_HTTP
+[2021/06/17 14:33:58:4829] N:  -- [vh|3|jitt-DFF2B5B4||-1] (3) 1.462s
+[2021/06/17 14:33:58:4833] N:  -- [wsicli|2|GET/h1/ebay.com] (0) 1.462s
+[2021/06/17 14:33:58:4834] N:  -- [vh|0|netlink] (2) 3.660s
+[2021/06/17 14:33:58:4858] N:  -- [vh|1|default||-1] (1) 3.654s
+[2021/06/17 14:33:58:4860] N:  -- [vh|2|jitt-7F69A044||-1] (0) 2.456s
+[2021/06/17 14:33:58:4974] U: Completed: OK (seen expected 0)
+```
+
+You can also test the client Basic Auth support against the http-server/minimal-http-server-basicauth
+example.  In one console window run the server and in the other
+
+```
+$ lws-minimal-http-client -l --nossl --path /secret/index.html --user user --password password
+```
+
+The Basic Auth credentials for the test server are literally username "user" and password "password".
+
--- a/minimal-examples/http-client/minimal-http-client-jit-trust/minimal-http-client.c
+++ b/minimal-examples/http-client/minimal-http-client-jit-trust/minimal-http-client.c
@ -0,0 +1,468 @@
+/*
+ * lws-minimal-http-client-jit-trust
+ *
+ * Written in 2010-2021 by Andy Green <andy@warmcat.com>
+ *
+ * This file is made available under the Creative Commons CC0 1.0
+ * Universal Public Domain Dedication.
+ *
+ * This demonstrates the a minimal http client using lws.
+ *
+ * It visits https://warmcat.com/ and receives the html page there.  You
+ * can dump the page data by changing the #if 0 below.
+ */
+
+#include <libwebsockets.h>
+#include <string.h>
+#include <signal.h>
+
+static int interrupted, bad = 1, status, conmon;
+#if defined(LWS_WITH_HTTP2)
+static int long_poll;
+#endif
+static struct lws *client_wsi;
+static const char *ba_user, *ba_password;
+static int budget = 6;
+
+/*
+ * For this example, we import the C-formatted array version of the trust blob
+ * directly.  This is produced by running scripts/mozilla-trust-gen.sh and can
+ * be found in ./_trust after that.
+ */
+
+static uint8_t jit_trust_blob[] = {
+#include "./trust_blob.h"
+};
+
+static const lws_retry_bo_t retry = {
+	.secs_since_valid_ping = 3,
+	.secs_since_valid_hangup = 10,
+};
+
+#if defined(LWS_WITH_CONMON)
+void
+dump_conmon_data(struct lws *wsi)
+{
+	const struct addrinfo *ai;
+	struct lws_conmon cm;
+	char ads[48];
+
+	lws_conmon_wsi_take(wsi, &cm);
+
+	lws_sa46_write_numeric_address(&cm.peer46, ads, sizeof(ads));
+	lwsl_notice("%s: peer %s, dns: %uus, sockconn: %uus, "
+		    "tls: %uus, txn_resp: %uus\n",
+		    __func__, ads,
+		    (unsigned int)cm.ciu_dns,
+		    (unsigned int)cm.ciu_sockconn,
+		    (unsigned int)cm.ciu_tls,
+		    (unsigned int)cm.ciu_txn_resp);
+
+	ai = cm.dns_results_copy;
+	while (ai) {
+		lws_sa46_write_numeric_address((lws_sockaddr46 *)ai->ai_addr,
+						ads, sizeof(ads));
+		lwsl_notice("%s: DNS %s\n", __func__, ads);
+		ai = ai->ai_next;
+	}
+
+	/*
+	 * This destroys the DNS list in the lws_conmon that we took
+	 * responsibility for when we used lws_conmon_wsi_take()
+	 */
+
+	lws_conmon_release(&cm);
+}
+#endif
+
+struct args {
+	int argc;
+	const char **argv;
+};
+
+static const struct lws_protocols protocols[];
+
+static int
+try_connect(struct lws_context *cx)
+{
+	struct lws_client_connect_info i;
+	struct args *a = lws_context_user(cx);
+	const char *p;
+
+	memset(&i, 0, sizeof i); /* otherwise uninitialized garbage */
+	i.context = cx;
+	if (!lws_cmdline_option(a->argc, a->argv, "-n")) {
+		i.ssl_connection = LCCSCF_USE_SSL;
+#if defined(LWS_WITH_HTTP2)
+		/* requires h2 */
+		if (lws_cmdline_option(a->argc, a->argv, "--long-poll")) {
+			lwsl_user("%s: long poll mode\n", __func__);
+			long_poll = 1;
+		}
+#endif
+	}
+
+	if (lws_cmdline_option(a->argc, a->argv, "-l")) {
+		i.port = 7681;
+		i.address = "localhost";
+		i.ssl_connection |= LCCSCF_ALLOW_SELFSIGNED;
+	} else {
+		i.port = 443;
+		i.address = "warmcat.com";
+	}
+
+	if (lws_cmdline_option(a->argc, a->argv, "--nossl"))
+		i.ssl_connection = 0;
+
+	i.ssl_connection |= LCCSCF_H2_QUIRK_OVERFLOWS_TXCR |
+			    LCCSCF_H2_QUIRK_NGHTTP2_END_STREAM |
+			    LCCSCF_ACCEPT_TLS_DOWNGRADE_REDIRECTS;
+
+	i.alpn = "h2,http/1.1";
+	if (lws_cmdline_option(a->argc, a->argv, "--h1"))
+		i.alpn = "http/1.1";
+
+	if (lws_cmdline_option(a->argc, a->argv, "--h2-prior-knowledge"))
+		i.ssl_connection |= LCCSCF_H2_PRIOR_KNOWLEDGE;
+
+	if ((p = lws_cmdline_option(a->argc, a->argv, "-p")))
+		i.port = atoi(p);
+
+	if ((p = lws_cmdline_option(a->argc, a->argv, "--user")))
+		ba_user = p;
+	if ((p = lws_cmdline_option(a->argc, a->argv, "--password")))
+		ba_password = p;
+
+	if (lws_cmdline_option(a->argc, a->argv, "-j"))
+		i.ssl_connection |= LCCSCF_ALLOW_SELFSIGNED;
+
+	if (lws_cmdline_option(a->argc, a->argv, "-k"))
+		i.ssl_connection |= LCCSCF_ALLOW_INSECURE;
+
+	if (lws_cmdline_option(a->argc, a->argv, "-m"))
+		i.ssl_connection |= LCCSCF_SKIP_SERVER_CERT_HOSTNAME_CHECK;
+
+	if (lws_cmdline_option(a->argc, a->argv, "-e"))
+		i.ssl_connection |= LCCSCF_ALLOW_EXPIRED;
+
+	if ((p = lws_cmdline_option(a->argc, a->argv, "-f"))) {
+		i.ssl_connection |= LCCSCF_H2_MANUAL_RXFLOW;
+		i.manual_initial_tx_credit = atoi(p);
+		lwsl_notice("%s: manual peer tx credit %d\n", __func__,
+				i.manual_initial_tx_credit);
+	}
+
+#if defined(LWS_WITH_CONMON)
+	if (lws_cmdline_option(a->argc, a->argv, "--conmon")) {
+		i.ssl_connection |= LCCSCF_CONMON;
+		conmon = 1;
+	}
+#endif
+
+	/* the default validity check is 5m / 5m10s... -v = 3s / 10s */
+
+	if (lws_cmdline_option(a->argc, a->argv, "-v"))
+		i.retry_and_idle_policy = &retry;
+
+	if ((p = lws_cmdline_option(a->argc, a->argv, "--server")))
+		i.address = p;
+
+	if ((p = lws_cmdline_option(a->argc, a->argv, "--path")))
+		i.path = p;
+	else
+		i.path = "/";
+
+	i.host = i.address;
+	i.origin = i.address;
+	i.method = "GET";
+
+	i.protocol = protocols[0].name;
+	i.pwsi = &client_wsi;
+	i.fi_wsi_name = "user";
+
+	if (!lws_client_connect_via_info(&i)) {
+		lwsl_err("Client creation failed\n");
+		interrupted = 1;
+		bad = 2; /* could not even start client connection */
+		lws_cancel_service(cx);
+
+		return 1;
+	}
+
+	return 0;
+}
+
+static const char *ua = "Mozilla/5.0 (X11; Linux x86_64) "
+			"AppleWebKit/537.36 (KHTML, like Gecko) "
+			"Chrome/51.0.2704.103 Safari/537.36",
+		  *acc = "*/*";
+
+static int
+callback_http(struct lws *wsi, enum lws_callback_reasons reason,
+	      void *user, void *in, size_t len)
+{
+	switch (reason) {
+
+	/* because we are protocols[0] ... */
+	case LWS_CALLBACK_CLIENT_CONNECTION_ERROR:
+		lwsl_err("CLIENT_CONNECTION_ERROR: %s\n",
+			 in ? (char *)in : "(null)");
+
+		if (budget--) {
+			try_connect(lws_get_context(wsi));
+			break;
+		}
+
+		interrupted = 1;
+		bad = 3; /* connection failed before we could make connection */
+		lws_cancel_service(lws_get_context(wsi));
+
+#if defined(LWS_WITH_CONMON)
+	if (conmon)
+		dump_conmon_data(wsi);
+#endif
+		break;
+
+	case LWS_CALLBACK_ESTABLISHED_CLIENT_HTTP:
+		{
+			char buf[128];
+
+			lws_get_peer_simple(wsi, buf, sizeof(buf));
+			status = (int)lws_http_client_http_response(wsi);
+
+			lwsl_user("Connected to %s, http response: %d\n",
+					buf, status);
+		}
+#if defined(LWS_WITH_HTTP2)
+		if (long_poll) {
+			lwsl_user("%s: Client entering long poll mode\n", __func__);
+			lws_h2_client_stream_long_poll_rxonly(wsi);
+		}
+#endif
+
+		if (lws_fi_user_wsi_fi(wsi, "user_reject_at_est"))
+			return -1;
+
+		break;
+
+
+	case LWS_CALLBACK_CLIENT_APPEND_HANDSHAKE_HEADER:
+	{
+		unsigned char **p = (unsigned char **)in, *end = (*p) + len;
+
+		if (lws_add_http_header_by_token(wsi, WSI_TOKEN_HTTP_USER_AGENT,
+				(unsigned char *)ua, (int)strlen(ua), p, end))
+			return -1;
+
+		if (lws_add_http_header_by_token(wsi, WSI_TOKEN_HTTP_ACCEPT,
+				(unsigned char *)acc, (int)strlen(acc), p, end))
+			return -1;
+
+#if defined(LWS_WITH_HTTP_BASIC_AUTH)
+		{
+		char b[128];
+
+	/* you only need this if you need to do Basic Auth */
+
+		if (!ba_user || !ba_password)
+			break;
+
+		if (lws_http_basic_auth_gen(ba_user, ba_password, b, sizeof(b)))
+			break;
+		if (lws_add_http_header_by_token(wsi, WSI_TOKEN_HTTP_AUTHORIZATION,
+				(unsigned char *)b, (int)strlen(b), p, end))
+			return -1;
+		}
+#endif
+
+		break;
+	}
+
+	/* chunks of chunked content, with header removed */
+	case LWS_CALLBACK_RECEIVE_CLIENT_HTTP_READ:
+		lwsl_user("RECEIVE_CLIENT_HTTP_READ: read %d\n", (int)len);
+#if defined(LWS_WITH_HTTP2)
+		if (long_poll) {
+			char dotstar[128];
+			lws_strnncpy(dotstar, (const char *)in, len,
+				     sizeof(dotstar));
+			lwsl_notice("long poll rx: %d '%s'\n", (int)len,
+					dotstar);
+		}
+#endif
+#if 0
+		lwsl_hexdump_notice(in, len);
+#endif
+
+		return 0; /* don't passthru */
+
+	/* uninterpreted http content */
+	case LWS_CALLBACK_RECEIVE_CLIENT_HTTP:
+		{
+			char buffer[1024 + LWS_PRE];
+			char *px = buffer + LWS_PRE;
+			int lenx = sizeof(buffer) - LWS_PRE;
+
+			if (lws_fi_user_wsi_fi(wsi, "user_reject_at_rx"))
+				return -1;
+
+			if (lws_http_client_read(wsi, &px, &lenx) < 0)
+				return -1;
+		}
+		return 0; /* don't passthru */
+
+	case LWS_CALLBACK_COMPLETED_CLIENT_HTTP:
+		lwsl_user("LWS_CALLBACK_COMPLETED_CLIENT_HTTP\n");
+		interrupted = 1;
+		bad = 0; // we accept 403 or whatever for this test status != 200;
+		lws_cancel_service(lws_get_context(wsi)); /* abort poll wait */
+		break;
+
+	case LWS_CALLBACK_CLOSED_CLIENT_HTTP:
+		lwsl_notice("%s: LWS_CALLBACK_CLOSED_CLIENT_HTTP\n", __func__);
+		interrupted = 1;
+		bad = 0; // status != 200;
+		lws_cancel_service(lws_get_context(wsi)); /* abort poll wait */
+#if defined(LWS_WITH_CONMON)
+		if (conmon)
+			dump_conmon_data(wsi);
+#endif
+		break;
+
+	default:
+		break;
+	}
+
+	return lws_callback_http_dummy(wsi, reason, user, in, len);
+}
+
+static const struct lws_protocols protocols[] = {
+	{
+		"http",
+		callback_http,
+		0,
+		0,
+	},
+	{ NULL, NULL, 0, 0 }
+};
+
+static void
+sigint_handler(int sig)
+{
+	interrupted = 1;
+}
+
+static int
+system_notify_cb(lws_state_manager_t *mgr, lws_state_notify_link_t *link,
+		   int current, int target)
+{
+	struct lws_context *cx = mgr->parent;
+
+	if (current != LWS_SYSTATE_OPERATIONAL ||
+	    target != LWS_SYSTATE_OPERATIONAL)
+		return 0;
+
+	lwsl_info("%s: operational\n", __func__);
+
+	try_connect(cx);
+
+	return 0;
+}
+
+static int
+jit_trust_query(struct lws_context *cx, const uint8_t *skid,
+		size_t skid_len, void *got_opaque)
+{
+	const uint8_t *der = NULL;
+	size_t der_len = 0;
+
+	lwsl_info("%s\n", __func__);
+	lwsl_hexdump_info(skid, skid_len);
+
+	/*
+	 * For this example, we look up SKIDs using a trust table that's
+	 * compiled in, synchronously.  Lws provides the necessary helper.
+	 *
+	 * DER will remain NULL if no match.
+	 */
+
+	lws_tls_jit_trust_blob_queury_skid(jit_trust_blob,
+					   sizeof(jit_trust_blob), skid,
+					   skid_len, &der, &der_len);
+
+	if (der)
+		lwsl_info("%s: found len %d\n", __func__, (int)der_len);
+	else
+		lwsl_info("%s: not trusted\n", __func__);
+
+	/* Once we have a result, pass it to the completion helper */
+
+	return lws_tls_jit_trust_got_cert_cb(cx, got_opaque, skid, skid_len,
+					     der, der_len);
+}
+
+static lws_system_ops_t system_ops = {
+	.jit_trust_query		= jit_trust_query
+};
+
+int main(int argc, const char **argv)
+{
+	lws_state_notify_link_t notifier = { {0}, system_notify_cb, "app" };
+	lws_state_notify_link_t *na[] = { &notifier, NULL };
+	struct lws_context_creation_info info;
+	struct lws_context *context;
+	int n = 0, expected = 0;
+	struct args args;
+	const char *p;
+
+	args.argc = argc;
+	args.argv = argv;
+
+	signal(SIGINT, sigint_handler);
+
+	memset(&info, 0, sizeof info); /* otherwise uninitialized garbage */
+	lws_cmdline_option_handle_builtin(argc, argv, &info);
+
+	lwsl_user("LWS minimal http client JIT Trust [-d<verbosity>] [-l] [--h1]\n");
+
+	info.options = LWS_SERVER_OPTION_DO_SSL_GLOBAL_INIT |
+		       /* we start off not trusting anything */
+		       LWS_SERVER_OPTION_DISABLE_OS_CA_CERTS |
+		       LWS_SERVER_OPTION_H2_JUST_FIX_WINDOW_UPDATE_OVERFLOW;
+	info.port = CONTEXT_PORT_NO_LISTEN; /* we do not run any server */
+	info.protocols = protocols;
+	info.user = &args;
+	info.register_notifier_list = na;
+	info.connect_timeout_secs = 30;
+	info.system_ops = &system_ops;
+	info.fd_limit_per_thread = 1 + 6 + 1;
+	info.max_http_header_data = 8192;
+
+	context = lws_create_context(&info);
+	if (!context) {
+		lwsl_err("lws init failed\n");
+		bad = 5;
+		goto bail;
+	}
+
+	while (n >= 0 && !interrupted)
+		n = lws_service(context, 0);
+
+	lwsl_err("%s: destroying context, interrupted = %d\n", __func__,
+			interrupted);
+
+	lws_context_destroy(context);
+
+bail:
+	if ((p = lws_cmdline_option(argc, argv, "--expected-exit")))
+		expected = atoi(p);
+
+	if (bad == expected) {
+		lwsl_user("Completed: OK (seen expected %d)\n", expected);
+		return 0;
+	}
+
+	lwsl_err("Completed: failed: exit %d, expected %d\n", bad, expected);
+
+	return 1;
+}
--- a/minimal-examples/http-client/minimal-http-client-jit-trust/trust_blob.h
+++ b/minimal-examples/http-client/minimal-http-client-jit-trust/trust_blob.h
--- a/minimal-examples/http-client/minimal-http-client-jit-trust/warmcat.com.cer
+++ b/minimal-examples/http-client/minimal-http-client-jit-trust/warmcat.com.cer
@ -0,0 +1,21 @@
+ -----BEGIN CERTIFICATE-----
+MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
+MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
+DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
+PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
+Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
+AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
+rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
+OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
+xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
+7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
+aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
+HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
+SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
+ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
+AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
+R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
+JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
+Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
+-----END CERTIFICATE-----
+
--- a/minimal-examples/http-client/minimal-http-client/minimal-http-client.c
+++ b/minimal-examples/http-client/minimal-http-client/minimal-http-client.c
@ -256,6 +256,7 @@ system_notify_cb(lws_state_manager_t *mgr, lws_state_notify_link_t *link,
 		i.ssl_connection = 0;

 	i.ssl_connection |= LCCSCF_H2_QUIRK_OVERFLOWS_TXCR |
+			    LCCSCF_ACCEPT_TLS_DOWNGRADE_REDIRECTS |
 			    LCCSCF_H2_QUIRK_NGHTTP2_END_STREAM;

 	i.alpn = "h2";
--- a/scripts/ahrefs-topsites.sh
+++ b/scripts/ahrefs-topsites.sh
@ -0,0 +1,6 @@
+#!/bin/bash
+
+wget -O- https://ahrefs.com/blog/most-visited-websites/ | grep most-visited-websites-us | \
+	sed -E 's/class="column-2">/|/g' | tr '|' '\n' | \
+	sed 's/<.*//g' | grep -v Domain | grep -v Josh | sort | uniq
+
--- a/scripts/mozilla-trust-gen.sh
+++ b/scripts/mozilla-trust-gen.sh
@ -0,0 +1,194 @@
+#!/bin/bash
+
+# This script fetches the current list of trusted CAs blessed by Mozilla
+# for web tls validation, and processes it into two outputs
+#
+# - ./trust/webroot/* consisting of ./_trust/webroot/der  a static, serveable set
+#     of trusted DER certs, with symlinks in ./_trust/webroot/by-skid and
+#     ./_trust/webroot/by-iss allowing serving the DER matching a given
+#     SubjectKeyIdentifier or Issuer + serial combination (suitably encoded)
+#
+# - ./_trust/blob-XXXX.bin  a single blob containing indexes and DER CA certs
+#
+# - ./_trust/trust_blob.h   a C uint8_t array formatted copy of blob-XXXX.bin
+
+# The trust blob layout is currently
+#
+# 54 42 4c 42     Magic "TBLB"
+# 00 01           MSB-first trust blob layout version
+# XX XX           MSB-first count of certificates
+# XX XX XX XX     MSB-first trust blob generation unix time
+# XX XX XX XX     MSB-first offset of cert length table (MSB-first 16-bit length-per-cert)
+# XX XX XX XX     MSB-first offset of SKID length table (8-bit length-per-cert)
+# XX XX XX XX     MSB-first offset of SKID table
+# XX XX XX XX     MSB-first total blob length
+#
+# XX .. XX        DER certs (start at +0x1c)
+# XX .. XX        DER cert length table (MSB-first 16-bit per cert)
+# XX .. XX        SKID length table (8-bit per cert)
+# XX .. XX        SKID table (variable per cert)
+#
+
+echo "Mozilla trust bundle for TLS validation processing  Andy Green <andy@warmcat.com>"
+echo
+
+rm -rf _trust
+mkdir _trust
+
+wget -O _trust/trusted.txt "https://ccadb-public.secure.force.com/mozilla/IncludedRootsPEMTxt?TrustBitsInclude=Websites"
+#cp ~/Downloads/IncludedRootsPEM.txt _trust/trusted.txt
+
+if [ $? -ne 0 ]; then
+	echo "Failed to get current website trust bundle"
+	exit 1
+fi
+
+mkdir -p _trust/webroot/by-skid _trust/webroot/by-iss _trust/webroot/der
+
+echo 0 > _trust/ofs
+echo 0 > _trust/count
+echo 0 > _trust/skidtab
+
+GT=`date +%s`
+BN=_trust/blob-$GT.bin
+
+cat _trust/trusted.txt | while read _line ; do
+	line=`echo -n $_line | sed 's/\r$//g'`
+	if [ "$line" == "-----BEGIN CERTIFICATE-----" ] ; then
+		echo $line > _trust/single
+	else
+		echo $line >> _trust/single
+
+		if [ "$line" == "-----END CERTIFICATE-----" ] ; then
+			openssl x509 -in _trust/single -text -noout > _trust/c1
+			if [ $? -ne 0 ] ; then
+				echo "FAILED"
+				exit 1
+			fi
+
+			ISS=`cat _trust/c1 | grep Issuer: | sed "s/.*://g" | sed "s/^\ *//g"`
+			SER=`cat _trust/c1 | grep "Serial Number:" | sed "s/.*://g" | sed "s/^\ *//g" | sed "s/\ .*//g"`
+			if [ -z "$SER" ] ; then
+				SER=`cat _trust/c1 | sed -e "1,/.*Serial Number:/ d" | head -n 1 | sed "s/^\ *//g" | sed "s/\ .*//g"`
+			fi
+			SKID=`cat _trust/c1 | sed -e '1,/.*X509v3 Subject Key Identifier:/ d' | sed -n '/Signature.*/q;p' | \
+				grep ':' | grep -v ': ' | grep -v ':$' | grep -v U | grep -v k | grep -v T | grep -v "i" | \
+				grep -v "S" | grep -v "V" | sed "s/^\ *//g"`
+			SKID_NO_COLONS=`echo -n $SKID | sed "s/://g"`
+
+			na=`cat _trust/c1 | grep "Not\ After\ :" | sed "s/.*\ :\ //g"`
+			ct=`date +%s`
+			ts=`date --date="$na" +%s`
+			life_days=`echo -n "$(( ( $ts - $ct ) / 86400 ))"`
+
+			echo "$life_days $safe" >> _trust/life
+			if [ $life_days -lt 1095 ] ; then
+				echo "$life_days $safe" >> _trust/life_lt_3y
+			fi
+
+			echo "issuer=\"$ISS\", serial=\"${SER^^}\", skid=\"${SKID_NO_COLONS^^}\", life_days=\"${life_days}\""
+
+			issname=`echo -n "$ISS"_"$SER" | tr -cd '[a-zA-Z0-9]_'`
+			skidname=`echo -n "$SKID_NO_COLONS" | tr -cd '[a-zA-Z0-9]_'`
+			safe=$issname"_"$skidname
+
+			cat _trust/single | grep -v -- '---' | base64 -d > _trust/webroot/der/$safe
+			cd _trust/webroot/by-skid
+			ln -sf ../der/$safe $SKID_NO_COLONS
+			cd ../../..
+			cd _trust/webroot/by-iss
+			ln -sf ../der/$safe $issname
+			cd ../../..
+
+			DERSIZ=`cat _trust/single | grep -v -- '---' | base64 -d | wc -c | cut -d' ' -f1`
+
+			cat _trust/single | grep -v -- '---' | base64 -d | hexdump -C | tr -s ' ' | sed 's/\ $//g' | \
+				cut -d' ' -f 2-17 | cut -d'|' -f1 | grep -v 000 | sed "s/\ //g" | sed ':a;N;$!ba;s/\n//g' | xxd -r -p >> _trust/_ders
+
+			printf "%04x" $DERSIZ | xxd -r -p  >> _trust/_derlens
+
+echo $SKID
+
+			if [ ! -z "$SKID" ] ; then
+				echo -n "$SKID_NO_COLONS" | xxd -r -p >> _trust/_skid
+			fi
+			SKIDSIZ=`echo -n $SKID_NO_COLONS | xxd -r -p | wc -c | cut -d' ' -f1`
+			printf "%02x" $SKIDSIZ | xxd -r -p  >> _trust/_skidlens
+
+			OFS=`cat _trust/ofs`
+			echo -n $(( $OFS + $DERSIZ )) > _trust/ofs
+			COUNT=`cat _trust/count`
+			echo -n $(( $COUNT +1 )) > _trust/count
+			ST=`cat _trust/skidtab`
+			echo -n $(( $ST + ( `echo -n $skidname | wc -c | cut -d' ' -f1` / 2 ) )) > _trust/skidtab
+
+			rm -f _trust/single
+
+		fi
+	fi
+
+done
+
+	COUNT=`cat _trust/count`
+	OFS=`cat _trust/ofs`
+	ST=`cat _trust/skidtab`
+
+	# everything in the layout framing is MSB-first
+
+	# magic
+	echo -n "TBLB" > $BN
+	# blob layout version
+	echo -n 0001 | xxd -r -p >> $BN
+	# number of certs in the blob
+	printf "%04x" $COUNT | xxd -r -p >> $BN
+	# unix time blob was created
+	printf "%08x" $GT | xxd -r -p >> $BN
+
+	POS=28
+	POS=$(( $POS + `cat _trust/_ders | wc -c | cut -d' ' -f1` ))
+
+	# blob offset of start of cert length table
+	printf "%08x" $POS | xxd -r -p >> $BN
+
+	POS=$(( $POS + `cat _trust/_derlens | wc -c | cut -d' ' -f1` ))
+
+	# blob offset of start of SKID length table
+	printf "%08x" $POS | xxd -r -p >> $BN
+
+	POS=$(( $POS + `cat _trust/_skidlens | wc -c | cut -d' ' -f1` ))
+
+	# blob offset of start of SKID table
+	printf "%08x" $POS | xxd -r -p >> $BN
+
+	POS=$(( $POS + `cat _trust/_skid | wc -c | cut -d' ' -f1` ))
+
+	# blob total length
+	printf "%08x" $POS | xxd -r -p >> $BN
+
+
+	# the DER table, start at +0x1c
+	cat _trust/_ders >> $BN
+	# the DER length table
+	cat _trust/_derlens >> $BN
+	# the SKID length table
+	cat _trust/_skidlens >> $BN
+	# the SKID table
+	cat _trust/_skid >> $BN
+
+# produce a C-friendly version of the blob
+
+	cat $BN | hexdump -v -C | tr -s ' ' | sed 's/\ $//g' | \
+		cut -d' ' -f 2-17 | cut -d'|' -f1 | grep -v 000 | sed "s/\ /,\ 0x/g" | sed "s/^/0x/g" | \
+		sed 's/\, 0x$//g' | sed 's/$/,/g' >> _trust/trust_blob.h
+
+
+	echo
+	echo "$COUNT CA certs, $POS byte blob"
+	echo
+	echo "CAs expiring in less than 3 years (days left):"
+	sort -h _trust/life_lt_3y
+
+	rm -f _trust/count _trust/_idx _trust/_idx_skid _trust/ofs _trust/_skid _trust/skidtab _trust/life _trust/life_lt_3y _trust/c1 _trust/single _trust/_derlens _trust/_ders _trust/_skid _trust/_skidlens
+
+exit 0
+