stv0g/libwebsockets
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

listen socket more frequent service

Browse source 
From an idea by Edwin van den Oetelaar <oetelaar.automatisering@gmail.com>

When testing libwebsockets with ab, Edwin found an unexpected bump in
the distribution of latencies, some connections were held back almost
the whole test duration.

http://ml.libwebsockets.org/pipermail/libwebsockets/2013-January/000006.html

Studying the problem revealed that when there are mass pending connections
amongst many active connections, we do not service the listen socket often
enough to clear the backlog, some seem to get stale violating FIFO ordering.

This patch introduces listen socket service "piggybacking", where every n
normal socket service actions we also check the listen socket and deal with
pending connections there.

Normally, it checks the listen socket gratuitously every 10 normal socket
services.  However, if it finds something waiting, it forces a check on the
next normal socket service too by keeping stats on how often something was
waiting.  If the probability of something waiting each time becomes high,
it will allow up to two waiting connections to be serviced for each normal
socket service.

In that way it has low burden in the normal case, but rapidly adapts by
detecting mass connection loads as found in ab.

Signed-off-by: Andy Green <andy.green@linaro.org>
This commit is contained in:
Andy Green 2013-01-16 07:59:47 +08:00
parent fa17ca7225
commit 65b0e91061
2 changed files with 49 additions and 1 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

41
lib/libwebsockets.c
View file

@ -1534,6 +1534,41 @@ libwebsocket_service_fd(struct libwebsocket_context *context,
/* no, here to service a socket descriptor */
/*
* deal with listen service piggybacking
* every listen_service_modulo services of other fds, we
* sneak one in to service the listen socket if there's anything waiting
*
* To handle connection storms, as found in ab, if we previously saw a
* pending connection here, it causes us to check again next time.
*/
if (context->listen_service_fd && pollfd->fd != context->listen_service_fd) {
context->listen_service_count++;
if (context->listen_service_extraseen ||
context->listen_service_count == context->listen_service_modulo) {
context->listen_service_count = 0;
m = 1;
if (context->listen_service_extraseen > 5)
m = 2;
while (m--) {
/* even with extpoll, we prepared this internal fds for listen */
n = poll(&context->fds[0], 1, 0);
if (n > 0) { /* there's a connection waiting for us */
libwebsocket_service_fd(context, &context->fds[0]);
context->listen_service_extraseen++;
} else {
if (context->listen_service_extraseen)
context->listen_service_extraseen--;
break;
}
}
}
}
/* okay, what we came here to do... */
wsi = wsi_from_fd(context, pollfd->fd);
if (wsi == NULL) {
@ -3040,10 +3075,14 @@ libwebsocket_create_context(int port, const char *interf,
wsi->mode = LWS_CONNMODE_SERVER_LISTENER;
insert_wsi(context, wsi);
context->listen_service_modulo = LWS_LISTEN_SERVICE_MODULO;
context->listen_service_count = 0;
context->listen_service_fd = sockfd;
listen(sockfd, LWS_SOMAXCONN);
lwsl_info(" Listening on port %d\n", port);
/* list in the internal poll array */
/* list in the internal poll array - we're always first */
context->fds[context->fds_count].fd = sockfd;
context->fds[context->fds_count++].events = POLLIN;

9
lib/private-libwebsockets.h
View file

@ -173,6 +173,11 @@ extern void _lws_log(int filter, const char *format, ...);
#define LWS_MAX_ZLIB_CONN_BUFFER (64 * 1024)
#endif
/*
* if not in a connection storm, check for incoming
* connections this many normal connection services
*/
#define LWS_LISTEN_SERVICE_MODULO 10
enum lws_websocket_opcodes_04 {
LWS_WS_OPCODE_04__CONTINUATION = 0,
@ -294,6 +299,10 @@ struct libwebsocket_context {
unsigned long last_timeout_check_s;
int fd_random;
int listen_service_modulo;
int listen_service_count;
int listen_service_fd;
int listen_service_extraseen;
#ifdef LWS_OPENSSL_SUPPORT
int use_ssl;