diff --git a/documentation/Clients.md b/documentation/Clients.md index 8ef49397c..bb96a3434 100644 --- a/documentation/Clients.md +++ b/documentation/Clients.md @@ -2,6 +2,8 @@ @subpage gtfpga @subpage opal +@subpage file +@subpage socket Every server needs clients which act as sinks / sources for simulation data. In case of S2SS these clients are called _nodes_. diff --git a/documentation/Concept.md b/documentation/Concept.md index c8e183f32..432f1f2bb 100644 --- a/documentation/Concept.md +++ b/documentation/Concept.md @@ -21,53 +21,26 @@ Possible types of nodes are: ## Paths -A path is a **unidirectional** connection between incoming and outgoing nodes. +A path is a **uni-directional** connection between incoming and outgoing nodes. -A path simply forwards messages from the incoming node to the outgoing node. -By default the server does not alter message contents. -It only checks if the message headers are valid (sequence number, cryptographic signature..) -However every path supports optional callback function which allow user-defined operations on the message contents. +It forwards messages from a single incoming node to multiple outgoing nodes. +Therefore it represents a 1-to-n relation between nodes. For bidirectional communication a corresponding path in the reverse direction must be added. - -The server is designed to allow multiple outgoing and incoming nodes per path as an extension. -@todo This has **not** been implemented at the moment but will come in future versions. + +By default, message contents are not altered. +The server only performs checks for valid message headers (sequence number, cryptographic signature..). +However every path supports optional hook/callback functions which allow user-defined operations on the message contents. @diafile path_simple.dia @see path for implementation details. -## Message - -A message contains a variable number of values. -Usually a a simulator sends one message per timestep. - -Simulation data is encapsulated in UDP packages in sent over IP networks like the internet. -We designed a lightweight message format (or protocol) to facilitate a fast transmission. - -@diafile msg_format.dia - -### Protocol details - -There are several types of messages: - - * **Data**: contains simulation data. - * **Control**: start/stops a new simulation case. - * **Config**: used to pass settings between nodes. - * **Sync**: used to syncrhonize the clocks between nodes. - -For now, only the first type (data) is implemented. -Therefore the complete protocol is **stateless**. -Later we might want to support more complex simulation scenarios which require some kind of controlling. - -Except for the simulation data, all values are sent in **network byte order** (big endian)! -The endianess of the simulation data is indicated by a single bit in the message header. -This allows us to reduce the amount of conversions during one transfer. - -@see msg for implementation details. - ## Interface +Interfaces are used to represent physical network ports on the server. +They are only used by the [Socket](socket) type of nodes. + @todo Add documentation @see interface for implementation details. diff --git a/documentation/Configuration.md b/documentation/Configuration.md index e7399a360..3e5e73c3b 100644 --- a/documentation/Configuration.md +++ b/documentation/Configuration.md @@ -1,18 +1,22 @@ # Configuration -The S2SS server configuration is completly done in a single file. +The S2SS configuration is completly contained in a single file. Take a look at the example configuration: `server/etc/example.conf`. -The configuration file mainly consists of three sections: +The configuration file consists of three sections: -### Global +## Global The global section consists of some global configuration parameters: +#### `debug` + `debug` expects an integer number (0-10) which determines the verbosity of debug messages during the execution of the server. Use this with care! Producing a lot of IO might decrease the performance of the server. Omitting this setting or setting it to zero will disable debug messages completely. +#### `stats` + `stats` specifies the rate in which statistics about the actives paths will be printed to the screen. Setting this value to 5, will print 5 lines per second. A line of includes information such as: @@ -21,38 +25,44 @@ A line of includes information such as: - Messages sent - Messaged dropped +#### `affinity` + The `affinity` setting allows to restrict the exeuction of the daemon to certain CPU cores. This technique, also called 'pinning', improves the determinism of the server by isolating the daemon processes on exclusive cores. +#### `priority` + The `priority` setting allows to adjust the scheduling priority of the deamon processes. By default, the daemon uses a real-time optimized FIFO scheduling algorithm. -### Nodes +## Nodes -The node section is a directory of nodes (clients) which are connected to the S2SS instance. +The node section is a **directory** of nodes (clients) which are connected to the S2SS instance. The directory is indexed by the name of the node: nodes = { "sintef_node" = { - type = "udp" + type = "socket" .... } } There are multiple diffrent type of nodes. But all types have the following settings in common: +#### `type` + `type` sets the type of the node. This should be one of: - - `udp` - - `ip` - - `gtfpga` - - `opal` - - `file` + - `socket` which refers to a [Socket](socket) node. + - `gtfpga` which refers to a [GTFPGA](gtfpga) node. + - `opal` which refers to a [OPAL Asynchronous Process](opal) node. + - `file` which refers to a [File](file) node. -For a detailed description of the node-specific settings, please +The remaining settings per node a depending on `type`. +Take a look a the specific pages for details. -### Paths +## Paths -The path section consists of a list of paths. +The path section consists of a **list** of paths. Every path is allowed to have the following settings: The `in` and `out` settings expect the name of the source and destination node. diff --git a/documentation/Setup.md b/documentation/Setup.md index ae620ff7b..eb7ad7bf2 100644 --- a/documentation/Setup.md +++ b/documentation/Setup.md @@ -1,8 +1,4 @@ -# Server - -@subpage configuration -@subpage usage -@subpage netem +# Setup ## Compilation @@ -10,12 +6,16 @@ Install libraries including developement headers for: - - _libconfig_ for parsing the config file + - [libconfig](http://www.hyperrealm.com/libconfig/) for parsing the config file + - [libnl3](http://www.infradead.org/~tgr/libnl/) for the network emulation support + +Use the following command to install the dependencies under Debian-based distributions: - $ sudo apt-get install iproute2 libconfig-dev libsodium-dev -or: + $ sudo apt-get install iproute2 libconfig-dev linbl-3-dev - $ sudo yum install iproute2 libconfig-devel libsodium-devel +or the following line for Fedora / CentOS / Redhat systems: + + $ sudo yum install iproute2 libconfig-devel libnl3-devel **Important:** Please note that the server requires the [iproute2](http://www.linuxfoundation.org/collaborate/workgroups/networking/iproute2) diff --git a/documentation/Timing.md b/documentation/Timing.md index 337cd101a..f59cb244f 100644 --- a/documentation/Timing.md +++ b/documentation/Timing.md @@ -1,2 +1,3 @@ +# Timing @diafile timing.dia diff --git a/documentation/Tools.md b/documentation/Tools.md new file mode 100644 index 000000000..a59dcc538 --- /dev/null +++ b/documentation/Tools.md @@ -0,0 +1,29 @@ +# Tools {#tools} + +S2SS comes with a couple of tools to test and debug: + + 1. `send` can be used to send samples from STDIN to a node + 2. `receive` can be used to read samples from a node and write it to STDOUT + 3. `random` writes random sample data to STDOUT + +## Examples + + 1. Start server: + + $ ./server etc/example.conf + + 2. Receive/dump data to file + + $ ./receive etc/example.conf node_name > dump.csv + + 3. Replay recorded data: + + $ ./send etc/example.conf -r node_name < dump.csv + + 4. Send random generated values: + + $ ./random 4 100 | sudo ./send etc/example.conf destination_node + + 5. Test ping/pong latency: + + $ ./test latency etc/example.conf test_node \ No newline at end of file diff --git a/documentation/server/Configuration.md b/documentation/Tuning.md similarity index 77% rename from documentation/server/Configuration.md rename to documentation/Tuning.md index 2cc2f1584..3330da5cd 100644 --- a/documentation/server/Configuration.md +++ b/documentation/Tuning.md @@ -1,24 +1,6 @@ -# Configuration {#configuration} +# Tuning {#tuning} -## File - -The server gets exclusively configured by its configuration file. - -The following sections describes the available options. -Take a look in the 'Examples' section for some examples. - -### Global - -- *name*: -- *affinity* - -### Nodes - -### Paths - -## Tuning - -### Operating System and Kernel +## Operating System and Kernel For minimum latency several kernel and driver settings can be optimized. A [RT-preemp patched Linux](https://rt.wiki.kernel.org/index.php/Main_Page) kernel is recommended. @@ -39,7 +21,7 @@ Precompiled kernels for Fedora can be found here: http://ccrma.stanford.edu/plan # https://www.kernel.org/doc/Documentation/networking/e1000e.txt option e1000e InterruptThrottleRate= -### Hardware +## Hardware This are some proposals for the selection of appropriate server hardware: diff --git a/documentation/server/Usage.md b/documentation/Usage.md similarity index 73% rename from documentation/server/Usage.md rename to documentation/Usage.md index f27353e23..8cf8abaf8 100644 --- a/documentation/server/Usage.md +++ b/documentation/Usage.md @@ -1,4 +1,4 @@ -# Usage Instructions {#usage} +# Usage {#usage} The S2SS server (`server`) expects the path to a configuration file as a single argument. @@ -35,11 +35,11 @@ The server requires root privileges for: 3. Go to S2SS directory - $ cd /home/acs-admin/msv/s2ss/server/ + $ cd /home/acs-admin/msv/s2ss/server/ 4. Edit configuration file - $ nano etc/opal-test.conf + $ nano etc/opal-test.conf - Take a look at the examples and documentation for a detailed description - Move with: cursor keys @@ -47,7 +47,7 @@ The server requires root privileges for: 5. Start server - $ sudo ./server etc/opal-test.conf + $ sudo ./server etc/opal-test.conf - `sudo` starts the server with super user priviledges @@ -55,26 +55,4 @@ The server requires root privileges for: 7. Logout - $ exit - -## Examples - - 1. Start server: - - $ ./server etc/example.conf - - 2. Receive/dump data to file - - $ ./receive *:10200 > dump.csv - - 3. Replay recorded data: - - $ ./send 4 192.168.1.12:10200 < dump.csv - - 4. Send random generated values: - - $ ./random 1 4 100 | ./send 4 192.168.1.12:10200 - - 5. Test ping/pong latency: - - $ ./test latency 192.168.1.12:10200 + $ exit diff --git a/documentation/clients/File.md b/documentation/clients/File.md index abc646266..5e0e80e23 100644 --- a/documentation/clients/File.md +++ b/documentation/clients/File.md @@ -1,4 +1,4 @@ -# File +# File {#file} The `file` node-type can be used to log or replay sample values to disk. @@ -6,13 +6,43 @@ The `file` node-type can be used to log or replay sample values to disk. Every `file` node supports the following settings: -`in` specifies the path to a file which contains data for replaying. +#### `in` -`out` specifies the path to a file where samples will be dumped. +Specifies the path to a file which contains data for replaying. +See below for a description of the file format. -`mode` +#### `out` -`rate` +Specifies the path to a file where samples will be written to. +This setting allows to add special paceholders for time and date values. +See [strftime(3)](http://man7.org/linux/man-pages/man3/strftime.3.html) for a list of supported placeholder. + +**Example**: + + out = "logs/measurements_%Y-%m-%d_%H-%M-%S.log" + +will create a file called: *path_of_working_directory*/logs/measurements_2015-08-09_22-20-50.log + +#### `file_mode` + +Specifies the mode which should be used to open the output file. +See [open(2)](http://man7.org/linux/man-pages/man2/open.2.html) for an explanation of allowed values. +The default value is `w+` which will start writing at the beginning of the file and create it in case it does not exist yet. + +#### `epoch_mode` + +This setting allows to select the behaviour of the following `epoch` setting. +It can be used to adjust the point in time when the first value should be read. + +The behaviour of `epoch` is depending on the value of `epoch_mode`. + + - `epoch_mode = now`: The first value is read at *now* + `epoch` seconds. + - `epoch_mode = relative`: The first value is read at *start* + `epoch` seconds. + - `epoch_mode = absolute`: The first value is read at `epoch` seconds after 1970-01-01 00:00:00. + +#### `rate` + +By default `rate` has the value `0`. If the value is non-zero, ### Example @@ -26,9 +56,18 @@ Every line in a file correspondents to a message / sample of simulation data. The columns of a line are seperated by whitespaces (tabs or spaces). The columns are defined as follows: - 1. Floating point timestamp in seconds since 1970-01-01 00:00:00 (aka Unix Epoch ). - 2. Up to `MSG_VALUES` floating point values per sample. The values are seperated by whitespaces as well. + 1. Seconds in floating point format since 1970-01-01 00:00:00 (aka Unix epoch timestamp: `date +%s`). + 2. Sequence number + 3. Up to `MSG_VALUES` floating point values per sample. The values are seperated by whitespaces as well. ### Example -@todo Add example dump +This example shows a dump with three values per sample: + + 1438959964.162102394 6 3.489760 -1.882725 0.860070 + 1438959964.261677582 7 2.375948 -2.204084 0.907518 + 1438959964.361622787 8 3.620115 -1.359236 -0.622333 + 1438959964.461907066 9 5.844254 -0.966527 -0.628751 + 1438959964.561499526 10 6.317059 -1.716363 0.351925 + 1438959964.661578339 11 6.471288 -0.159862 0.123948 + 1438959964.761956859 12 7.365932 -1.488268 -0.780568 diff --git a/documentation/clients/GTFPGA.md b/documentation/clients/GTFPGA.md index 6235fa811..015942eca 100644 --- a/documentation/clients/GTFPGA.md +++ b/documentation/clients/GTFPGA.md @@ -10,16 +10,13 @@ The PowerPC core is used to forward values between RTDS and the S2SS server. ## Configuration -`slot` +#### `slot` -`id` +#### `id` -`rate` +#### `rate` -## Source Code +### Example -The source code for the PowerPC is located at: +@todo Add excerpt from example.conf - /clients/gtfpga/s2ss/ - -@todo Add documentation for GTFPGA card. diff --git a/documentation/clients/Socket.md b/documentation/clients/Socket.md index a76c19b43..491242593 100644 --- a/documentation/clients/Socket.md +++ b/documentation/clients/Socket.md @@ -1,4 +1,4 @@ -# Socket +# Socket {#socket} The socket node-type is the most comprehensive and complex one. It allows to send and receive simulation data over the network. @@ -15,13 +15,13 @@ The implementation supports multiple protocols / OSI layers: ## Configuration -`local` +#### `local` -`remote` +#### `remote` -`netem` +#### `netem` -`layer` +#### `layer` ### Example @@ -29,10 +29,68 @@ The implementation supports multiple protocols / OSI layers: ## Packet Format -@todo add DIA figure here +The on-wire format of the network packets is not subject to a standardization process. +It's a very simple packet-based format which includes: + + - 32bit floating-point or integer values + - 32bit timestamp (integral seconds) + - 32bit timestamp (integral nanoseconds) + - 16bit sequence number + - 4bit version identified + - and several flags... + +## Message + +A message contains a variable number of values. +Usually a a simulator sends one message per timestep. + +Simulation data is encapsulated in UDP packages in sent over IP networks like the internet. +We designed a lightweight message format (or protocol) to facilitate a fast transmission. + +@diafile msg_format.dia + +For now, only the first message type (`data`) is used. +Therefore the complete protocol is **stateless**. +Later we might want to support more complex simulation scenarios which require some kind of controlling. + +Except for the simulation data, all values are sent in **network byte order** (big endian)! +The endianess of the simulation data is indicated by a single bit in the message header. +This allows us to reduce the amount of conversions during one transfer. + +@see msg for implementation details. ### Example @todo add screenshot of wireshark dump -## Network Emulation +## Network Emulation {#netem} + +S2SS supports the emulation of wide-area network characterisics. + +This emulation can be configured on a per-node basis for **outgoing** / **egress** data only. +Incoming data is not processed by the network emulation! + +This network emulation is handled by Linux' [netem queuing discipline](http://www.linuxfoundation.org/collaborate/workgroups/networking/netem) which is part of the traffic control subsystem. +Take a look at the following manual page for supported metrics: [tc-netem(8)](http://man7.org/linux/man-pages/man8/tc-netem.8.html). + +S2SS only takes care of setup and initalizing the netem queuing discipline inside the kernel. +For this the iproute2 software package (`ip` & `tc` commands) must be installed. +The configuration is done via the config file. +Look at `etc/example.conf` for a section called `netem` or `tc-netem(8)` for more details. + +### Custom delay distribution + +Netem supports loading custom delay distributions. + +1. Load and compile the netem tools from: + https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/netem +2. Create a custom distribution by following the steps described here: + https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/README.distribution +3. Put the generated distrubtion with the suffix `.dist` in the `tc` lib directory: `/usr/lib/tc/`. +4. Load the distribution specifying the basename in the server config. + +### Further information + + - https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/README.iproute2+tc + - https://github.com/stv0g/netem + diff --git a/documentation/server/Netem.md b/documentation/server/Netem.md deleted file mode 100644 index 758e785e3..000000000 --- a/documentation/server/Netem.md +++ /dev/null @@ -1,29 +0,0 @@ -# Network Emulation {#netem} - -S2SS supports the emulation of wide-area network characterisics. - -For this an existing network emulator called 'netem' is used. -Netem is part of the Linux kernel and therefore shipped with most distributions by default. -Netem is implemented as a queuing discipline. -This discipline define the order and point in time when packets will be send over the network. - -S2SS only takes care of setup and initalizing the netem queuing discipline inside the kernel. -For this the iproute2 software package (`ip` & `tc` commands) must be installed. -The configuration is done via the config file. -Look at `etc/example.conf` for a section called `netem` or `tc-netem(8)` for more details. - -## Custom delay distributions - -Netem supports loading custom delay distributions. - -1. Load and compile the netem tools from: - https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/netem -2. Create a custom distribution by following the steps described here: - https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/README.distribution -3. Put the generated distrubtion with the suffix `.dist` in the `tc` lib directory: `/usr/lib/tc/`. -4. Load the distribution specifying the basename in the server config. - -## Links - - - http://www.linuxfoundation.org/collaborate/workgroups/networking/netem - - https://git.kernel.org/cgit/linux/kernel/git/shemminger/iproute2.git/tree/README.iproute2+tc