mirror of
https://github.com/warmcat/libwebsockets.git
synced 2025-03-30 00:00:16 +01:00
docs: more details on ctest buddies
This commit is contained in:
Andy Green
parent
d1763df9ed
commit
fecf496ef9
1 changed files with 285 additions and 7 deletions
|
|
@ -23,6 +23,16 @@ replacing `bionic` with `xenial` as needed, and save (:wq). Then
|
||||||
# apt install cmake
|
# apt install cmake
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Tests live in CMakeLists.txt
|
||||||
|
|
||||||
|
The rules for tests are described in ctest / cmake language inside the minimal
|
||||||
|
examples and api tests that are enabled by current build options, so you need
|
||||||
|
to build with `-DLWS_WITH_MINIMAL_EXAMPLES=1` to build the examples along with
|
||||||
|
the library.
|
||||||
|
|
||||||
|
The tests are typically running the examples or api tests and regarding the
|
||||||
|
process exiting with exit code 0 as success, anything else as failure.
|
||||||
|
|
||||||
## Generating the tests
|
## Generating the tests
|
||||||
|
|
||||||
The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`. You can optionally set
|
The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`. You can optionally set
|
||||||
|
|
@ -31,20 +41,191 @@ internet connectivity.
|
||||||
|
|
||||||
## Preparing to run the tests
|
## Preparing to run the tests
|
||||||
|
|
||||||
The tests have to spawn by script some "test buddies", for example the client
|
The tests have to be able to run without root and without disturbing any other
|
||||||
tests have to run a test server from the built lws image. For that reason you
|
install of lws in the build machine.
|
||||||
have to do a side-install into `./destdir` using `make install DESTDIR=../destdir`
|
|
||||||
from the build directory before all the tests will work properly.
|
For that reason you have to do an unprivileged side-install into `../destdir`,
|
||||||
|
using `make install DESTDIR=../destdir` from the build directory and perform the
|
||||||
|
tests on the pieces in there.
|
||||||
|
|
||||||
## Running the tests
|
## Running the tests
|
||||||
|
|
||||||
CMake puts the test action into a build-host type specific form, for unix type
|
We must take care to run the pieces (.so etc) we just built, without having
|
||||||
platforms you just run `make test` or `CTEST_OUTPUT_ON_FAILURE=1 make test` to
|
root access, and not any of the same pieces from some other lws version that may
|
||||||
see what happened to any broken tests.
|
have been installed on the build machine. That includes, eg, plugins that
|
||||||
|
we just built, to ensure precedence of those in the search path we can set our
|
||||||
|
DESTDIR unprivileged install path in `LD_LIBRARY_PATH`.
|
||||||
|
|
||||||
|
Then we can run ctest on the unprivileged install. The whole step looks
|
||||||
|
something like this:
|
||||||
|
|
||||||
|
```
|
||||||
|
build $ make -j12 && \
|
||||||
|
rm -rf ../destdir && \
|
||||||
|
make -j12 DESTDIR=../destdir install && \\
|
||||||
|
LD_LIBRARY_PATH=../destdir/usr/local/share/libwebsockets-test-server/plugins ctest -j2 --output-on-failure
|
||||||
|
```
|
||||||
|
|
||||||
On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build
|
On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build
|
||||||
type.
|
type.
|
||||||
|
|
||||||
|
Good results look something like this (which tests can run depend on your
|
||||||
|
build options)
|
||||||
|
|
||||||
|
```
|
||||||
|
Test project /projects/libwebsockets/build
|
||||||
|
Start 71: st_wcs_srv
|
||||||
|
Start 43: st_hcp_srv
|
||||||
|
1/73 Test #71: st_wcs_srv .................................. Passed 5.01 sec
|
||||||
|
Start 19: st_hcmp_srv
|
||||||
|
2/73 Test #43: st_hcp_srv .................................. Passed 5.01 sec
|
||||||
|
Start 17: st_hcm_srv
|
||||||
|
3/73 Test #19: st_hcmp_srv ................................. Passed 5.01 sec
|
||||||
|
Start 55: st_ssproxyctx
|
||||||
|
4/73 Test #17: st_hcm_srv .................................. Passed 5.01 sec
|
||||||
|
Start 52: st_ssproxy
|
||||||
|
5/73 Test #55: st_ssproxyctx ............................... Passed 1.02 sec
|
||||||
|
Start 67: st_sstfproxy
|
||||||
|
6/73 Test #52: st_ssproxy .................................. Passed 1.02 sec
|
||||||
|
Start 60: st_ssprxsmd_sspc
|
||||||
|
7/73 Test #67: st_sstfproxy ................................ Passed 1.01 sec
|
||||||
|
Start 63: st_mulssprxsmd_sspc
|
||||||
|
8/73 Test #60: st_ssprxsmd_sspc ............................ Passed 1.01 sec
|
||||||
|
Start 69: sspc-minimaltf
|
||||||
|
9/73 Test #63: st_mulssprxsmd_sspc ......................... Passed 1.02 sec
|
||||||
|
Start 73: ws-client-spam
|
||||||
|
10/73 Test #73: ws-client-spam .............................. Passed 12.21 sec
|
||||||
|
Start 57: sspc-minimaltx
|
||||||
|
11/73 Test #57: sspc-minimaltx .............................. Passed 5.90 sec
|
||||||
|
Start 65: mulsspcsmd_sspc
|
||||||
|
12/73 Test #65: mulsspcsmd_sspc ............................. Passed 3.58 sec
|
||||||
|
Start 62: sspcsmd_sspc
|
||||||
|
13/73 Test #62: sspcsmd_sspc ................................ Passed 1.73 sec
|
||||||
|
Start 22: http-client-multi-h1
|
||||||
|
14/73 Test #22: http-client-multi-h1 ........................ Passed 5.04 sec
|
||||||
|
Start 25: http-client-multi-stag
|
||||||
|
15/73 Test #25: http-client-multi-stag ...................... Passed 4.53 sec
|
||||||
|
Start 26: http-client-multi-stag-h1
|
||||||
|
16/73 Test #26: http-client-multi-stag-h1 ................... Passed 4.40 sec
|
||||||
|
Start 21: http-client-multi
|
||||||
|
17/73 Test #21: http-client-multi ........................... Passed 4.37 sec
|
||||||
|
Start 36: http-client-multi-post-h1
|
||||||
|
18/73 Test #36: http-client-multi-post-h1 ................... Passed 2.73 sec
|
||||||
|
Start 54: sspc-minimal
|
||||||
|
19/73 Test #54: sspc-minimal ................................ Passed 0.93 sec
|
||||||
|
Start 39: http-client-multi-post-stag
|
||||||
|
20/73 Test #39: http-client-multi-post-stag ................. Passed 2.29 sec
|
||||||
|
Start 40: http-client-multi-post-stag-h1
|
||||||
|
21/73 Test #69: sspc-minimaltf .............................. Passed 49.83 sec
|
||||||
|
Start 35: http-client-multi-post
|
||||||
|
22/73 Test #40: http-client-multi-post-stag-h1 .............. Passed 4.30 sec
|
||||||
|
Start 33: http-client-multi-restrict-nopipe-fail
|
||||||
|
23/73 Test #35: http-client-multi-post ...................... Passed 3.23 sec
|
||||||
|
Start 28: http-client-multi-stag-h1-pipe
|
||||||
|
24/73 Test #33: http-client-multi-restrict-nopipe-fail ...... Passed 2.86 sec
|
||||||
|
Start 32: http-client-multi-restrict-stag-h1-pipe
|
||||||
|
25/73 Test #28: http-client-multi-stag-h1-pipe .............. Passed 2.86 sec
|
||||||
|
Start 27: http-client-multi-stag-pipe
|
||||||
|
26/73 Test #32: http-client-multi-restrict-stag-h1-pipe ..... Passed 1.51 sec
|
||||||
|
Start 31: http-client-multi-restrict-stag-pipe
|
||||||
|
27/73 Test #27: http-client-multi-stag-pipe ................. Passed 1.52 sec
|
||||||
|
Start 34: http-client-multi-restrict-h1-nopipe-fail
|
||||||
|
28/73 Test #34: http-client-multi-restrict-h1-nopipe-fail ... Passed 2.78 sec
|
||||||
|
Start 46: http-client-post-m
|
||||||
|
29/73 Test #31: http-client-multi-restrict-stag-pipe ........ Passed 2.80 sec
|
||||||
|
Start 42: http-client-multi-post-stag-h1-pipe
|
||||||
|
30/73 Test #42: http-client-multi-post-stag-h1-pipe ......... Passed 1.51 sec
|
||||||
|
Start 41: http-client-multi-post-stag-pipe
|
||||||
|
31/73 Test #46: http-client-post-m .......................... Passed 1.59 sec
|
||||||
|
Start 48: http-client-post-m-h1
|
||||||
|
32/73 Test #48: http-client-post-m-h1 ....................... Passed 1.10 sec
|
||||||
|
Start 23: http-client-multi-pipe
|
||||||
|
33/73 Test #41: http-client-multi-post-stag-pipe ............ Passed 1.51 sec
|
||||||
|
Start 29: http-client-multi-restrict-pipe
|
||||||
|
34/73 Test #23: http-client-multi-pipe ...................... Passed 1.09 sec
|
||||||
|
Start 24: http-client-multi-h1-pipe
|
||||||
|
35/73 Test #29: http-client-multi-restrict-pipe ............. Passed 0.74 sec
|
||||||
|
Start 30: http-client-multi-restrict-h1-pipe
|
||||||
|
36/73 Test #24: http-client-multi-h1-pipe ................... Passed 1.14 sec
|
||||||
|
Start 45: http-client-post
|
||||||
|
37/73 Test #30: http-client-multi-restrict-h1-pipe .......... Passed 1.14 sec
|
||||||
|
Start 38: http-client-multi-post-h1-pipe
|
||||||
|
38/73 Test #45: http-client-post ............................ Passed 0.30 sec
|
||||||
|
Start 37: http-client-multi-post-pipe
|
||||||
|
39/73 Test #38: http-client-multi-post-h1-pipe .............. Passed 0.49 sec
|
||||||
|
Start 47: http-client-post-h1
|
||||||
|
40/73 Test #37: http-client-multi-post-pipe ................. Passed 0.31 sec
|
||||||
|
Start 50: hs_evlib_foreign_event
|
||||||
|
41/73 Test #47: http-client-post-h1 ......................... Passed 0.29 sec
|
||||||
|
Start 66: ss-tf
|
||||||
|
42/73 Test #50: hs_evlib_foreign_event ...................... Passed 22.02 sec
|
||||||
|
Start 49: hs_evlib_foreign_uv
|
||||||
|
43/73 Test #49: hs_evlib_foreign_uv ......................... Passed 21.03 sec
|
||||||
|
Start 51: ss-warmcat
|
||||||
|
44/73 Test #51: ss-warmcat .................................. Passed 2.69 sec
|
||||||
|
Start 59: ss-smd
|
||||||
|
45/73 Test #59: ss-smd ...................................... Passed 1.78 sec
|
||||||
|
Start 10: api-test-secure-streams
|
||||||
|
46/73 Test #10: api-test-secure-streams ..................... Passed 1.34 sec
|
||||||
|
Start 11: http-client-warmcat
|
||||||
|
47/73 Test #11: http-client-warmcat ......................... Passed 0.27 sec
|
||||||
|
Start 58: sspost-warmcat
|
||||||
|
48/73 Test #58: sspost-warmcat .............................. Passed 0.84 sec
|
||||||
|
Start 12: http-client-warmcat-h1
|
||||||
|
49/73 Test #12: http-client-warmcat-h1 ...................... Passed 0.25 sec
|
||||||
|
Start 2: api-test-jose
|
||||||
|
50/73 Test #2: api-test-jose ............................... Passed 0.27 sec
|
||||||
|
Start 70: ws-client-rx-warmcat
|
||||||
|
51/73 Test #70: ws-client-rx-warmcat ........................ Passed 0.27 sec
|
||||||
|
Start 56: ki_ssproxyctx
|
||||||
|
52/73 Test #56: ki_ssproxyctx ............................... Passed 0.12 sec
|
||||||
|
Start 68: ki_ssproxy
|
||||||
|
53/73 Test #68: ki_ssproxy .................................. Passed 0.11 sec
|
||||||
|
Start 64: ki_mulssprxsmd_sspc
|
||||||
|
54/73 Test #64: ki_mulssprxsmd_sspc ......................... Passed 0.10 sec
|
||||||
|
Start 61: ki_ssprxsmd_sspc
|
||||||
|
55/73 Test #61: ki_ssprxsmd_sspc ............................ Passed 0.11 sec
|
||||||
|
Start 13: http-client-h2-rxflow-warmcat
|
||||||
|
56/73 Test #13: http-client-h2-rxflow-warmcat ............... Passed 0.28 sec
|
||||||
|
Start 14: http-client-h2-rxflow-warmcat-h1
|
||||||
|
57/73 Test #14: http-client-h2-rxflow-warmcat-h1 ............ Passed 0.34 sec
|
||||||
|
Start 16: http-client-hugeurl-warmcat-h1
|
||||||
|
58/73 Test #16: http-client-hugeurl-warmcat-h1 .............. Passed 0.16 sec
|
||||||
|
Start 15: http-client-hugeurl-warmcat
|
||||||
|
59/73 Test #15: http-client-hugeurl-warmcat ................. Passed 0.16 sec
|
||||||
|
Start 72: ki_wcs_srv
|
||||||
|
60/73 Test #72: ki_wcs_srv .................................. Passed 0.12 sec
|
||||||
|
Start 44: ki_hcp_srv
|
||||||
|
61/73 Test #44: ki_hcp_srv .................................. Passed 0.11 sec
|
||||||
|
Start 20: ki_hcmp_srv
|
||||||
|
62/73 Test #20: ki_hcmp_srv ................................. Passed 0.11 sec
|
||||||
|
Start 18: ki_hcm_srv
|
||||||
|
63/73 Test #18: ki_hcm_srv .................................. Passed 0.11 sec
|
||||||
|
Start 7: api-test-lws_struct_sqlite
|
||||||
|
64/73 Test #7: api-test-lws_struct_sqlite .................. Passed 0.03 sec
|
||||||
|
Start 1: api-test-gencrypto
|
||||||
|
65/73 Test #1: api-test-gencrypto .......................... Passed 0.02 sec
|
||||||
|
Start 6: api-test-lws_struct-json
|
||||||
|
66/73 Test #6: api-test-lws_struct-json .................... Passed 0.01 sec
|
||||||
|
Start 4: api-test-lws_dsh
|
||||||
|
67/73 Test #4: api-test-lws_dsh ............................ Passed 0.01 sec
|
||||||
|
Start 8: api-test-lws_tokenize
|
||||||
|
68/73 Test #8: api-test-lws_tokenize ....................... Passed 0.01 sec
|
||||||
|
Start 9: api-test-lwsac
|
||||||
|
69/73 Test #9: api-test-lwsac .............................. Passed 0.00 sec
|
||||||
|
Start 3: api-test-lejp
|
||||||
|
70/73 Test #3: api-test-lejp ............................... Passed 0.00 sec
|
||||||
|
Start 53: ki_ssproxy
|
||||||
|
71/73 Test #53: ki_ssproxy .................................. Passed 0.11 sec
|
||||||
|
72/73 Test #66: ss-tf ....................................... Passed 55.51 sec
|
||||||
|
Start 5: api-test-lws_smd
|
||||||
|
73/73 Test #5: api-test-lws_smd ............................ Passed 4.22 sec
|
||||||
|
|
||||||
|
100% tests passed, 0 tests failed out of 73
|
||||||
|
|
||||||
|
Total Test time (real) = 137.76 sec
|
||||||
|
```
|
||||||
|
|
||||||
## Considerations for creating tests
|
## Considerations for creating tests
|
||||||
|
|
||||||
### Timeout
|
### Timeout
|
||||||
|
|
@ -64,4 +245,101 @@ warmcat.com or libwebsockets.org additionally.
|
||||||
For that reason it's good practice to set the `WORKING_DIRECTORY` property to
|
For that reason it's good practice to set the `WORKING_DIRECTORY` property to
|
||||||
the home dir of the example app in all cases.
|
the home dir of the example app in all cases.
|
||||||
|
|
||||||
|
### Spawning Buddies
|
||||||
|
|
||||||
|
Many networking tests need to either spawn a client or a server in order to
|
||||||
|
have a "buddy" to talk to during the test for the opposing side. This is a
|
||||||
|
bit awkward in cmake since it does not directly support spawning daemons as
|
||||||
|
test dependencies.
|
||||||
|
|
||||||
|
Lws provides helper scripts for unix type targets in `./scripts/ctest-background.sh`
|
||||||
|
and `./scripts/ctest-background-kill.sh`, which spawn background processes,
|
||||||
|
save the pid in a decorated /tmp file and can later take the process down. This
|
||||||
|
also has arrangements to dump the log of any background process that exited
|
||||||
|
early.
|
||||||
|
|
||||||
|
To arrange the buddy to run aligned with the test, you first explain to cmake
|
||||||
|
how to start and stop the buddy using phony tests to make a "fixture" in cmake
|
||||||
|
terms.
|
||||||
|
|
||||||
|
In this example, taken from minimal-http-client-multi, we arrange for
|
||||||
|
minimal-http-server-tls to be available for our actual test. The starting and
|
||||||
|
stopping definition, for "st_hcm_srv" and "ki_hcm_srv":
|
||||||
|
|
||||||
|
```
|
||||||
|
add_test(NAME st_hcm_srv COMMAND
|
||||||
|
${CMAKE_SOURCE_DIR}/scripts/ctest-background.sh
|
||||||
|
hcm_srv $<TARGET_FILE:lws-minimal-http-server-tls>
|
||||||
|
--port ${PORT_HCM_SRV} )
|
||||||
|
add_test(NAME ki_hcm_srv COMMAND
|
||||||
|
${CMAKE_SOURCE_DIR}/scripts/ctest-background-kill.sh
|
||||||
|
hcm_srv $<TARGET_FILE_NAME:lws-minimal-http-server-tls>
|
||||||
|
--port ${PORT_HCM_SRV})
|
||||||
|
```
|
||||||
|
|
||||||
|
... and binding those together so cmake knows they start and stop a specific
|
||||||
|
named fixture "hcm_srv", itself with an 800s timeout
|
||||||
|
|
||||||
|
```
|
||||||
|
set_tests_properties(st_hcm_srv PROPERTIES
|
||||||
|
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-server/minimal-http-server-tls
|
||||||
|
FIXTURES_SETUP hcm_srv
|
||||||
|
TIMEOUT 800)
|
||||||
|
set_tests_properties(ki_hcm_srv PROPERTIES
|
||||||
|
FIXTURES_CLEANUP hcm_srv)
|
||||||
|
```
|
||||||
|
|
||||||
|
... and finally, adding the "hcm_srv" fixture as a requirement on the actual
|
||||||
|
test (http-client-multi) we are testing
|
||||||
|
|
||||||
|
```
|
||||||
|
set_tests_properties(http-client-multi
|
||||||
|
PROPERTIES
|
||||||
|
FIXTURES_REQUIRED "hcm_srv"
|
||||||
|
WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-client/minimal-http-client-multi
|
||||||
|
TIMEOUT 50)
|
||||||
|
```
|
||||||
|
|
||||||
|
Once all that explaining is done, ctest itself will take care about starting
|
||||||
|
and killing hcm_srv before and after http-client-multi test.
|
||||||
|
|
||||||
|
### Buddy sockets and test concurrency
|
||||||
|
|
||||||
|
For tests with local buddies using tcp sockets inside the same VM or systemd-
|
||||||
|
nspawn networking context, you cannot just use a well-known port like 7681.
|
||||||
|
|
||||||
|
ctest itself is usually executed concurrently, and Sai is typically building
|
||||||
|
multiple different instances concurrently as well (typically 3), so it may be
|
||||||
|
running different ctests inside the same VM simultaneously.
|
||||||
|
|
||||||
|
Different tests can have their own convention for port ranges, to solve the
|
||||||
|
problem about Sai running different tests concurrently inside one ctest.
|
||||||
|
|
||||||
|
For the case there are multiple ctests running, we can use the env var
|
||||||
|
`$ENV{SAI_INSTANCE_IDX}`, which is an ordinal like 0 or 1, to further ensure
|
||||||
|
that port selections won't conflict. If not using Sai, you can just set this
|
||||||
|
in the evironment yourself to reflect your build instance index.
|
||||||
|
|
||||||
|
```
|
||||||
|
#
|
||||||
|
# instantiate the server per sai builder instance, they are running in the same
|
||||||
|
# machine context in parallel so they can tread on each other otherwise
|
||||||
|
#
|
||||||
|
set(PORT_HCM_SRV "7670")
|
||||||
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "0")
|
||||||
|
set(PORT_HCM_SRV 7671)
|
||||||
|
endif()
|
||||||
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "1")
|
||||||
|
set(PORT_HCM_SRV 7672)
|
||||||
|
endif()
|
||||||
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "2")
|
||||||
|
set(PORT_HCM_SRV 7673)
|
||||||
|
endif()
|
||||||
|
if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "3")
|
||||||
|
set(PORT_HCM_SRV 7674)
|
||||||
|
endif()
|
||||||
|
```
|
||||||
|
|
||||||
|
This is complicated enough that the best approach is copy an existing simple
|
||||||
|
case like the CMakeLists.txt for minimal-http-client and change the names and
|
||||||
|
ports to be unique.
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue