From fecf496ef953df0f3a05a43aed27f27d71238213 Mon Sep 17 00:00:00 2001 From: Andy Green Date: Tue, 2 Mar 2021 07:44:54 +0000 Subject: [PATCH] docs: more details on ctest buddies --- READMEs/README.ctest.md | 292 +++++++++++++++++++++++++++++++++++++++- 1 file changed, 285 insertions(+), 7 deletions(-) diff --git a/READMEs/README.ctest.md b/READMEs/README.ctest.md index 48db124ac..0ed8a8400 100644 --- a/READMEs/README.ctest.md +++ b/READMEs/README.ctest.md @@ -23,6 +23,16 @@ replacing `bionic` with `xenial` as needed, and save (:wq). Then # apt install cmake ``` +## Tests live in CMakeLists.txt + +The rules for tests are described in ctest / cmake language inside the minimal +examples and api tests that are enabled by current build options, so you need +to build with `-DLWS_WITH_MINIMAL_EXAMPLES=1` to build the examples along with +the library. + +The tests are typically running the examples or api tests and regarding the +process exiting with exit code 0 as success, anything else as failure. + ## Generating the tests The main tests just need `-DLWS_WITH_MINIMAL_EXAMPLES=1`. You can optionally set @@ -31,20 +41,191 @@ internet connectivity. ## Preparing to run the tests -The tests have to spawn by script some "test buddies", for example the client -tests have to run a test server from the built lws image. For that reason you -have to do a side-install into `./destdir` using `make install DESTDIR=../destdir` -from the build directory before all the tests will work properly. +The tests have to be able to run without root and without disturbing any other +install of lws in the build machine. + +For that reason you have to do an unprivileged side-install into `../destdir`, +using `make install DESTDIR=../destdir` from the build directory and perform the +tests on the pieces in there. ## Running the tests -CMake puts the test action into a build-host type specific form, for unix type -platforms you just run `make test` or `CTEST_OUTPUT_ON_FAILURE=1 make test` to -see what happened to any broken tests. +We must take care to run the pieces (.so etc) we just built, without having +root access, and not any of the same pieces from some other lws version that may +have been installed on the build machine. That includes, eg, plugins that +we just built, to ensure precedence of those in the search path we can set our +DESTDIR unprivileged install path in `LD_LIBRARY_PATH`. + +Then we can run ctest on the unprivileged install. The whole step looks +something like this: + +``` +build $ make -j12 && \ + rm -rf ../destdir && \ + make -j12 DESTDIR=../destdir install && \\ + LD_LIBRARY_PATH=../destdir/usr/local/share/libwebsockets-test-server/plugins ctest -j2 --output-on-failure +``` On windows, it looks like `ctest . -C DEBUG` or RELEASE if that was the build type. +Good results look something like this (which tests can run depend on your +build options) + +``` +Test project /projects/libwebsockets/build + Start 71: st_wcs_srv + Start 43: st_hcp_srv + 1/73 Test #71: st_wcs_srv .................................. Passed 5.01 sec + Start 19: st_hcmp_srv + 2/73 Test #43: st_hcp_srv .................................. Passed 5.01 sec + Start 17: st_hcm_srv + 3/73 Test #19: st_hcmp_srv ................................. Passed 5.01 sec + Start 55: st_ssproxyctx + 4/73 Test #17: st_hcm_srv .................................. Passed 5.01 sec + Start 52: st_ssproxy + 5/73 Test #55: st_ssproxyctx ............................... Passed 1.02 sec + Start 67: st_sstfproxy + 6/73 Test #52: st_ssproxy .................................. Passed 1.02 sec + Start 60: st_ssprxsmd_sspc + 7/73 Test #67: st_sstfproxy ................................ Passed 1.01 sec + Start 63: st_mulssprxsmd_sspc + 8/73 Test #60: st_ssprxsmd_sspc ............................ Passed 1.01 sec + Start 69: sspc-minimaltf + 9/73 Test #63: st_mulssprxsmd_sspc ......................... Passed 1.02 sec + Start 73: ws-client-spam +10/73 Test #73: ws-client-spam .............................. Passed 12.21 sec + Start 57: sspc-minimaltx +11/73 Test #57: sspc-minimaltx .............................. Passed 5.90 sec + Start 65: mulsspcsmd_sspc +12/73 Test #65: mulsspcsmd_sspc ............................. Passed 3.58 sec + Start 62: sspcsmd_sspc +13/73 Test #62: sspcsmd_sspc ................................ Passed 1.73 sec + Start 22: http-client-multi-h1 +14/73 Test #22: http-client-multi-h1 ........................ Passed 5.04 sec + Start 25: http-client-multi-stag +15/73 Test #25: http-client-multi-stag ...................... Passed 4.53 sec + Start 26: http-client-multi-stag-h1 +16/73 Test #26: http-client-multi-stag-h1 ................... Passed 4.40 sec + Start 21: http-client-multi +17/73 Test #21: http-client-multi ........................... Passed 4.37 sec + Start 36: http-client-multi-post-h1 +18/73 Test #36: http-client-multi-post-h1 ................... Passed 2.73 sec + Start 54: sspc-minimal +19/73 Test #54: sspc-minimal ................................ Passed 0.93 sec + Start 39: http-client-multi-post-stag +20/73 Test #39: http-client-multi-post-stag ................. Passed 2.29 sec + Start 40: http-client-multi-post-stag-h1 +21/73 Test #69: sspc-minimaltf .............................. Passed 49.83 sec + Start 35: http-client-multi-post +22/73 Test #40: http-client-multi-post-stag-h1 .............. Passed 4.30 sec + Start 33: http-client-multi-restrict-nopipe-fail +23/73 Test #35: http-client-multi-post ...................... Passed 3.23 sec + Start 28: http-client-multi-stag-h1-pipe +24/73 Test #33: http-client-multi-restrict-nopipe-fail ...... Passed 2.86 sec + Start 32: http-client-multi-restrict-stag-h1-pipe +25/73 Test #28: http-client-multi-stag-h1-pipe .............. Passed 2.86 sec + Start 27: http-client-multi-stag-pipe +26/73 Test #32: http-client-multi-restrict-stag-h1-pipe ..... Passed 1.51 sec + Start 31: http-client-multi-restrict-stag-pipe +27/73 Test #27: http-client-multi-stag-pipe ................. Passed 1.52 sec + Start 34: http-client-multi-restrict-h1-nopipe-fail +28/73 Test #34: http-client-multi-restrict-h1-nopipe-fail ... Passed 2.78 sec + Start 46: http-client-post-m +29/73 Test #31: http-client-multi-restrict-stag-pipe ........ Passed 2.80 sec + Start 42: http-client-multi-post-stag-h1-pipe +30/73 Test #42: http-client-multi-post-stag-h1-pipe ......... Passed 1.51 sec + Start 41: http-client-multi-post-stag-pipe +31/73 Test #46: http-client-post-m .......................... Passed 1.59 sec + Start 48: http-client-post-m-h1 +32/73 Test #48: http-client-post-m-h1 ....................... Passed 1.10 sec + Start 23: http-client-multi-pipe +33/73 Test #41: http-client-multi-post-stag-pipe ............ Passed 1.51 sec + Start 29: http-client-multi-restrict-pipe +34/73 Test #23: http-client-multi-pipe ...................... Passed 1.09 sec + Start 24: http-client-multi-h1-pipe +35/73 Test #29: http-client-multi-restrict-pipe ............. Passed 0.74 sec + Start 30: http-client-multi-restrict-h1-pipe +36/73 Test #24: http-client-multi-h1-pipe ................... Passed 1.14 sec + Start 45: http-client-post +37/73 Test #30: http-client-multi-restrict-h1-pipe .......... Passed 1.14 sec + Start 38: http-client-multi-post-h1-pipe +38/73 Test #45: http-client-post ............................ Passed 0.30 sec + Start 37: http-client-multi-post-pipe +39/73 Test #38: http-client-multi-post-h1-pipe .............. Passed 0.49 sec + Start 47: http-client-post-h1 +40/73 Test #37: http-client-multi-post-pipe ................. Passed 0.31 sec + Start 50: hs_evlib_foreign_event +41/73 Test #47: http-client-post-h1 ......................... Passed 0.29 sec + Start 66: ss-tf +42/73 Test #50: hs_evlib_foreign_event ...................... Passed 22.02 sec + Start 49: hs_evlib_foreign_uv +43/73 Test #49: hs_evlib_foreign_uv ......................... Passed 21.03 sec + Start 51: ss-warmcat +44/73 Test #51: ss-warmcat .................................. Passed 2.69 sec + Start 59: ss-smd +45/73 Test #59: ss-smd ...................................... Passed 1.78 sec + Start 10: api-test-secure-streams +46/73 Test #10: api-test-secure-streams ..................... Passed 1.34 sec + Start 11: http-client-warmcat +47/73 Test #11: http-client-warmcat ......................... Passed 0.27 sec + Start 58: sspost-warmcat +48/73 Test #58: sspost-warmcat .............................. Passed 0.84 sec + Start 12: http-client-warmcat-h1 +49/73 Test #12: http-client-warmcat-h1 ...................... Passed 0.25 sec + Start 2: api-test-jose +50/73 Test #2: api-test-jose ............................... Passed 0.27 sec + Start 70: ws-client-rx-warmcat +51/73 Test #70: ws-client-rx-warmcat ........................ Passed 0.27 sec + Start 56: ki_ssproxyctx +52/73 Test #56: ki_ssproxyctx ............................... Passed 0.12 sec + Start 68: ki_ssproxy +53/73 Test #68: ki_ssproxy .................................. Passed 0.11 sec + Start 64: ki_mulssprxsmd_sspc +54/73 Test #64: ki_mulssprxsmd_sspc ......................... Passed 0.10 sec + Start 61: ki_ssprxsmd_sspc +55/73 Test #61: ki_ssprxsmd_sspc ............................ Passed 0.11 sec + Start 13: http-client-h2-rxflow-warmcat +56/73 Test #13: http-client-h2-rxflow-warmcat ............... Passed 0.28 sec + Start 14: http-client-h2-rxflow-warmcat-h1 +57/73 Test #14: http-client-h2-rxflow-warmcat-h1 ............ Passed 0.34 sec + Start 16: http-client-hugeurl-warmcat-h1 +58/73 Test #16: http-client-hugeurl-warmcat-h1 .............. Passed 0.16 sec + Start 15: http-client-hugeurl-warmcat +59/73 Test #15: http-client-hugeurl-warmcat ................. Passed 0.16 sec + Start 72: ki_wcs_srv +60/73 Test #72: ki_wcs_srv .................................. Passed 0.12 sec + Start 44: ki_hcp_srv +61/73 Test #44: ki_hcp_srv .................................. Passed 0.11 sec + Start 20: ki_hcmp_srv +62/73 Test #20: ki_hcmp_srv ................................. Passed 0.11 sec + Start 18: ki_hcm_srv +63/73 Test #18: ki_hcm_srv .................................. Passed 0.11 sec + Start 7: api-test-lws_struct_sqlite +64/73 Test #7: api-test-lws_struct_sqlite .................. Passed 0.03 sec + Start 1: api-test-gencrypto +65/73 Test #1: api-test-gencrypto .......................... Passed 0.02 sec + Start 6: api-test-lws_struct-json +66/73 Test #6: api-test-lws_struct-json .................... Passed 0.01 sec + Start 4: api-test-lws_dsh +67/73 Test #4: api-test-lws_dsh ............................ Passed 0.01 sec + Start 8: api-test-lws_tokenize +68/73 Test #8: api-test-lws_tokenize ....................... Passed 0.01 sec + Start 9: api-test-lwsac +69/73 Test #9: api-test-lwsac .............................. Passed 0.00 sec + Start 3: api-test-lejp +70/73 Test #3: api-test-lejp ............................... Passed 0.00 sec + Start 53: ki_ssproxy +71/73 Test #53: ki_ssproxy .................................. Passed 0.11 sec +72/73 Test #66: ss-tf ....................................... Passed 55.51 sec + Start 5: api-test-lws_smd +73/73 Test #5: api-test-lws_smd ............................ Passed 4.22 sec + +100% tests passed, 0 tests failed out of 73 + +Total Test time (real) = 137.76 sec +``` + ## Considerations for creating tests ### Timeout @@ -64,4 +245,101 @@ warmcat.com or libwebsockets.org additionally. For that reason it's good practice to set the `WORKING_DIRECTORY` property to the home dir of the example app in all cases. +### Spawning Buddies +Many networking tests need to either spawn a client or a server in order to +have a "buddy" to talk to during the test for the opposing side. This is a +bit awkward in cmake since it does not directly support spawning daemons as +test dependencies. + +Lws provides helper scripts for unix type targets in `./scripts/ctest-background.sh` +and `./scripts/ctest-background-kill.sh`, which spawn background processes, +save the pid in a decorated /tmp file and can later take the process down. This +also has arrangements to dump the log of any background process that exited +early. + +To arrange the buddy to run aligned with the test, you first explain to cmake +how to start and stop the buddy using phony tests to make a "fixture" in cmake +terms. + +In this example, taken from minimal-http-client-multi, we arrange for +minimal-http-server-tls to be available for our actual test. The starting and +stopping definition, for "st_hcm_srv" and "ki_hcm_srv": + +``` + add_test(NAME st_hcm_srv COMMAND + ${CMAKE_SOURCE_DIR}/scripts/ctest-background.sh + hcm_srv $ + --port ${PORT_HCM_SRV} ) + add_test(NAME ki_hcm_srv COMMAND + ${CMAKE_SOURCE_DIR}/scripts/ctest-background-kill.sh + hcm_srv $ + --port ${PORT_HCM_SRV}) +``` + +... and binding those together so cmake knows they start and stop a specific +named fixture "hcm_srv", itself with an 800s timeout + +``` + set_tests_properties(st_hcm_srv PROPERTIES + WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-server/minimal-http-server-tls + FIXTURES_SETUP hcm_srv + TIMEOUT 800) + set_tests_properties(ki_hcm_srv PROPERTIES + FIXTURES_CLEANUP hcm_srv) +``` + +... and finally, adding the "hcm_srv" fixture as a requirement on the actual +test (http-client-multi) we are testing + +``` + set_tests_properties(http-client-multi + PROPERTIES + FIXTURES_REQUIRED "hcm_srv" + WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/minimal-examples/http-client/minimal-http-client-multi + TIMEOUT 50) +``` + +Once all that explaining is done, ctest itself will take care about starting +and killing hcm_srv before and after http-client-multi test. + +### Buddy sockets and test concurrency + +For tests with local buddies using tcp sockets inside the same VM or systemd- +nspawn networking context, you cannot just use a well-known port like 7681. + +ctest itself is usually executed concurrently, and Sai is typically building +multiple different instances concurrently as well (typically 3), so it may be +running different ctests inside the same VM simultaneously. + +Different tests can have their own convention for port ranges, to solve the +problem about Sai running different tests concurrently inside one ctest. + +For the case there are multiple ctests running, we can use the env var +`$ENV{SAI_INSTANCE_IDX}`, which is an ordinal like 0 or 1, to further ensure +that port selections won't conflict. If not using Sai, you can just set this +in the evironment yourself to reflect your build instance index. + +``` + # + # instantiate the server per sai builder instance, they are running in the same + # machine context in parallel so they can tread on each other otherwise + # + set(PORT_HCM_SRV "7670") + if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "0") + set(PORT_HCM_SRV 7671) + endif() + if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "1") + set(PORT_HCM_SRV 7672) + endif() + if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "2") + set(PORT_HCM_SRV 7673) + endif() + if ("$ENV{SAI_INSTANCE_IDX}" STREQUAL "3") + set(PORT_HCM_SRV 7674) + endif() +``` + +This is complicated enough that the best approach is copy an existing simple +case like the CMakeLists.txt for minimal-http-client and change the names and +ports to be unique.