This is a squashed commit from https://github.com/andrew-canaday/libwebsockets,
dev/http_keepalive branch (strategies changed a few times, so the commit
history is clutteread). This branch is submitted for clarity, but the other
can be used as a reference or alternative.
* added **enum http_version** to track HTTP/1.0 vs HTTP/1.1 requests
* added **enum http_connection_type** to track keep-alive vs close
* replaced content_length_seen and body_index with **content_remain**
* removed **post_buffer** (see handshake.c modifications)
* removed post_buffer free
* switch state to WSI_TOKEN_SKIPPING after URI is complete to store version
* delete *spill* label (unused)
* add vars to track HTTP version and connection type
* HTTP version defaults to 1.0
* connection type defaults to 'close' for 1.0, keep-alive for 1.1
* additional checks in **cleanup:** label:
* if HTTP version string is present and valid, set enum val appropriately
* override connection default with the "Connection:" header, if present
* set state to WSI_STATE_HTTP_BODY if content_length > 0
* return 0 on HTTP requests, unless LWS_CALLBACK_HTTP indicates otherwise
* add vars to track remaining content_length and body chunk size
* re-arrange switch case order to facilitate creation of jump-table
* added new labels:
* **read_ok**: normal location reach on break from switch; just return 0
* **http_complete**: check for keep-alive + init state, mode, hdr table
* **http_new**: jump location for keep-alive when http_complete sees len>0
* after libwebsocket_parse, jump to one of those labels based on state
* POST body handling:
* don't bother iterating over input byte-by-byte or using memcpy
* just pass the relevant portion of the context->service_buffer to callback
- added options to minilex.c
- regenerated lextable.h
- added WSI_TOKEN_OPTIONS_URI to libwebsockets.h
- tweaked parsers.c to accommodate OPTIONS token
- tweaked server.c to set uri_ptr and uri_len for HTTP callback on OPTIONS as well
Until now minilex has done fine with providing a simple and fast header
decode state machine. But for HTTP2.0, new headers must be added and it
is already on the limit of table branching in 1 byte (already using +0xf8
of a max limit of 0xff).
This changes the minilex format to improve storage size without loss of
decode efficiency. It reduces the curent lws header table from 546 -> 403
bytes and upgrades the ability to increase table size by allowing jumps
to increase from the old limit of +255 states to +65535 states, which should
be enough for anything we ever want to do.
The max number of terminals is also increased from 128 to 2048.
Signed-off-by: Andy Green <andy.green@linaro.org>
Svetlin wrote on github
According to RFC2616, all header field names in both HTTP requests and HTTP responses are case-insensitive. But libwebsockets uses a case-sensitive compare.
Reproduce:
Run libwebsockets against a server that sends all of its HTTP header field names in lower-case (for example: https://github.com/extend/cowboy). libwebsockets reports an error. The expected behavior is no errors reported and a successful handshake procedure.
This changes the parser reference table in minilex to all lower case.
The code to walk the parser tables then just forces a tolower on the incoming chars.
This (and minilex tables) only applies to header names.
Reported-by: svetlin-mladnov <?@github>
Signed-off-by: Andy Green <andy.green@linaro.org>
If the URI coming from the client contains '?' then
- the URI part is terminated with a '\0'
- the remainder of the URI goes in a new header WSI_TOKEN_HTTP_URI_ARGS
- the remainder of the URI is not subject to path sanitization measures (it
still has %xx processing done on it)
In the test server, http requests now also dump header information to stderr.
The attack.sh script is simplified and can now parse the test server header dumps.
Signed-off-by: Andy Green <andy.green@linaro.org>
This translates %xx in the GET uri and removes /.. and /... type sequences along with
translating // or /// etc to /.
Since the result is hopefully secure, it also changes the test server to actually use
the uri path pasted on a resource directory without whitelisting.
Signed-off-by: Andy Green <andy.green@linaro.org>
Clean up minilex
Move the header output to stdout
Introduce lexfile.h as the header output
Use lexfile.h in both minilex itself and lws
Add the following header support
"Accept:",
"If-Modified-Since:",
"Accept-Encoding:",
"Accept-Language:",
"Pragma:",
"Cache-Control:",
"Authorization:",
"Cookie:",
"Content-Type:",
"Date:",
"Range:",
"Referer:"
Signed-off-by: Andy Green <andy.green@linaro.org>
close() from javascript api in Chrome and Firefox doesn't do the
right thing. It's because the payload is zero-length (with a frame
key...) This fixes it.
Reported-by: 巫书轶
Signed-off-by: Andy Green <andy.green@linaro.org>
under load, writing packet sizes to the socket that are normally fine
can do partial writes, eg asking to write 4096 may only take 2800 of
it and return 2800 from the actual send.
Until now lws assumed that if it was safe to send, it could take any
size buffer, that's not the case under load.
This patch changes lws_write to return the amount actually taken...
that and the meaning of it becomes tricky when dealing with
compressed links, the amount taken and the amount sent differ. Also
there is no way to recover at the moment from a protocol-encoded
frame only being partially accepted... however for http file send
content it can and does recover now.
Small frames don't have to take any care about it but large atomic
sends (> 2K) have been seen to fail under load.
Signed-off-by: Andy Green <andy.green@linaro.org>
The header name buffer and its max length handling has actually
been unused since the minilex parser was introduced. We hold
parsing state in the lex-type parts and don't need to store or
worry about max length, since the parser will let us know as
soon as it can't be a match for the valid header names.
This strips it out reducing the per-connection allocation for
x86_64 with default configure from 224 to 160.
Signed-off-by: Andy Green <andy.green@linaro.org>
Drop the connection during parsing for a few more cases that can't be legit.
Take care about trying to free rxflow_buffer only if we reached a connmode
where it exists
Change behaviour on setting unknown HTTP method to kill connection
Signed-off-by: Andy Green <andy.green@linaro.org>
If the SSL connection failed before the headers came, we were not
dealing with deallocating the header malloc. This takes care of it.
Using CyaSSL, we are then valgrind-clean for ssl client and server.
With OpenSSL, there is 88 bytes lost at init that never changes or
gets recovered. AFAIK there's nothing to do about that.
OpenSSL also blows these during operation
==1059== Conditional jump or move depends on uninitialised value(s)
==1059== at 0x4A0B131: bcmp (mc_replace_strmem.c:935)
==1059== by 0x3014CDDBA8: ??? (in /usr/lib64/libcrypto.so.1.0.1c)
==1059== by 0x3015430852: tls1_enc (in /usr/lib64/libssl.so.1.0.1c)
==1059== by 0x3015428CEC: ssl3_read_bytes (in /usr/lib64/libssl.so.1.0.1c)
==1059== by 0x30154264C5: ??? (in /usr/lib64/libssl.so.1.0.1c)
==1059== by 0x4C3C596: lws_server_socket_service (server.c:153)
==1059== by 0x4C32C1E: libwebsocket_service_fd (libwebsockets.c:927)
==1059== by 0x4C33270: libwebsocket_service (libwebsockets.c:1225)
==1059== by 0x401C84: main (in /usr/bin/libwebsockets-test-server)
However googling around
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/60021http://www.openssl.org/support/faq.html#PROG13
(also the next FAQ down)
it seems OpenSSL have a relaxed attitude to this and it's expected.
It's interesting CyaSSL works fine but doesn't have that problem...
Signed-off-by: Andy Green <andy.green@linaro.org>
This brings the library sources into compliance with checkpatch
style except for three or four exceptions like WIN32 related stuff
and one long string constant I don't want to break into multiple
sprintf calls.
There should be no functional or compilability change from all
this (hopefully).
Signed-off-by: Andy Green <andy.green@linaro.org>
This removes all the direct wsi members specific to clients,
most of them are moved to being fake headers in the next 3-layer
header scheme, c_port moves to being a member of the u.hdr
unionized struct.
It gets rid of a lot of fiddly mallocs and frees(), despite it
adds a small internal API to create the fake headers, actually
the patch deletes more than it adds...
Signed-off-by: Andy Green <andy.green@linaro.org>
This big patch replaces the malloc / realloc per header
approach used until now with a single three-level struct
that gets malloc'd during the header union phase and freed
in one go when we transition to a different union phase.
It's more expensive in that we malloc a bit more than 4Kbytes,
but it's a lot cheaper in terms of malloc, frees, heap fragmentation,
no reallocs, nothing to configure. It also moves from arrays of
pointers (8 bytes on x86_64) to unsigned short offsets into the
data array, (2 bytes on all platforms).
The 3-level thing is all in one struct
- array indexed by the header enum, pointing to first "fragment" index
(ie, header type to fragment lookup, or 0 for none)
- array of fragments indexes, enough for 2 x the number of known headers
(fragment array... note that fragments can point to a "next"
fragment if the same header is spread across multiple entries)
- linear char array where the known header payload gets written
(fragments point into null-terminated strings stored in here,
only the known header content is stored)
http headers can legally be split over multiple headers of the same
name which should be concatenated. This scheme does not linearly
conatenate them but uses a linked list in the fragment structs to
link them. There are apis to get the total length and copy out a
linear, concatenated version to a buffer.
Signed-off-by: Andy Green <andy.green@linaro.org>
A new protocol member is defined that controls the size of rx
buffer allocation per connection. For compatibility 0 size
allocates 4096, but you should adapt your protocol definition
array in the user code to declare an appropriate value.
See the changelog for more detail.
The advantage is the rx frame buffer size is now tailored to
what is expected from the protocol, rather than being fixed
to a default of 4096. If your protocol only sends frames of
a dozen bytes this allows you to only allocate an rx frame
buffer of the same size.
For example the per-connection allocation (excluding headers)
for the test server fell from ~4500 to < 750 bytes with this.
Signed-off-by: Andy Green <andy.green@linaro.org>
This reduces the size of struct libwebscocket from 4840 to 4552
on x86_64
There are also big benefits on malloc pool fragmentation and
allocation, the header allocations only exist between the first
peer communication and websocket connection establishment for
both server and client.
Signed-off-by: Andy Green <andy.green@linaro.org>
Large chunks of struct libwebsocket members actually have a mutually
exclusive lifecycle, eg, once the http headers are finished they sit
there unused until the instance is destroyed.
This makes a big improvement in memory efficiency by making four
categories of member: always needed, needed for header processing,
needed for http processing, and needed for ws processing. The last
three are mutually exclusive and bound into a union inside the wsi.
Care needs taking now at "union transitions", although we zeroed down
the struct at init, the other union siblings have been writing the
same memory by the time later member siblings start to use it. So
it must be cleared down appropriately when we cross from one
mutually-exclusive use to another.
Signed-off-by: Andy Green <andy.green@linaro.org>
Since v13 was defined as the released ietf version the older versions
are deprecated. This patch strips out everything to do with the older
versions and gets rid of the option to send stuff unmasked.
The in-tree md5 implementation is then also deleted as nothing needs
it any more, 1280 loc are shed in all
Signed-off-by: Andy Green <andy.green@linaro.org>
