1
0
Fork 0
mirror of https://git.rwth-aachen.de/acs/public/villas/node/ synced 2025-03-09 00:00:00 +01:00

openapi: convert last node configurations to OpenAPI

This commit is contained in:
Steffen Vogel 2022-04-05 12:41:19 +02:00
parent d0da86ae3f
commit 83107cbbe0
9 changed files with 583 additions and 14 deletions

View file

@ -4,7 +4,192 @@
allOf:
- type: object
properties:
format:
$ref: ../format_spec.yaml
rdma_port_space:
type: string
enum:
- RC
- UC
- UD
default: RC
description: |
This specifies the type of connection the node will set up.
* `RC` provides reliable, connection-oriented, message based communication between the nodes. Packets are delivered in order. In this mode, one Queue Pair is connected to one other Queue Pair.
* `UC` provides unreliable, connection-oriented, message based communication between the nodes. This service type is not officially supported by the RDMA communication manager and is implemented for scientific purposes in VILLASnode. [The InfiniBand node-type source code provides information on how to enable this service type.](https://git.rwth-aachen.de/acs/public/villas/node/blob/master/lib/nodes/infiniband.c#L429)
* `UD` provides unreliable, connection-less, datagram communication between nodes. Both ordering and delivery are not guaranteed in this mode.
`RC`, `UC`, and `UD` are mapped to the Queue Pair types as `RDMA_PS_TCP`/`IBV_QPT_RC`, `RDMA_PS_IPOIB`/`IBV_QPT_UC`, and `RDMA_PS_UDP`/`IBV_QPT_UD`, respectively.
If two nodes should be connected, both should be set to the same `rdma_port_space`.
More information on these two modes can be found on the manual page for [`rdma_create_id()`](https://linux.die.net/man/3/rdma_create_id).
in:
type: object
properties:
address:
type: string
description: |
Connections between `infiniband` nodes are established over IP over IB (IPoIP).
To use this node, you have to make sure that the linux driver `ib_ipoib` is loaded.
If it is not loaded, load it with `modprobe ib_ipoib`.
If it is loaded, you have to make sure that the Host Channel Adapters (HCAs) have an IP address.
You can configure the IP address of the Infiniband HCA with the `ifconfig` utility, exactly like you would configure normal Ethernet adapters.
As soon as an IP is set for the local HCA, this entry can be used to point to the adapter and to define the port which will be used for connection related communication.
**Example**:
```
in = {
address="10.0.0.1:1337"
}
```
binds the node to the local device which is bound to `10.0.0.1`. It will use port `1337` for communication related to the connection.
max_wrs:
type: integer
default: 128
description: |
Before a packet can be received with Infiniband, the application has to describe how this will be handled (e.g., to what address the data will be written).
This happens in a so called Work Request (WR).
`in.max_wrs` sets the maximum number of receive Work Requests which can be posted to the receive queue of the Queue Pair.
For higher throughput, it is recommended to increase this value since it will serve as a buffer.
cq_size:
type: integer
default: 128
description: |
This value defines the number of Work Completions the Completion Queue can hold.
If a packet is received, the Queue Pair will write a Work Completion to the Completion Queue.
The node polls this queue to process received packets. If the Completion Queue gets full, which is often caused by `cq_size` being to small, and thus the receive queue is not able to post Work Completions, the node will abort.
If a connection is disconnected, all outstanding Work Requests—even is they are not used—are flushed to the Completion Queue.
Here applies the same as mentioned above: if the Completion Queue has fewer space left than outstanding Work Requests are available, this will result in an error.
It is therefor recommended to set the value of `cq_size` to at least
```
in.cq_size >= in.max_wrs - in.buffer_subtraction
```
buffer_subtraction:
type: integer
default: 16
description: |
As mentioned in the `in.max_wrs` settings, Work Requests have to be present in the receive queue, for it to be able to process received data.
To take full advantage of the zero-copy capabilities of Infiniband this node-type directly posts addresses from the VILLASnode to the receive queue instead of copying all data over after receiving it.
This technique relies on the exchange of addresses. This means that if an array of `in.vectorize` addresses is handed over to the node-type, max `release` <= `in.vectorize` addresses that point to received data can be returned.
Furthermore, if `release` addresses should be returned, `release` addresses from the original array must be posted to the receive queue.
To ensure that we can always post at least `in.vectorize` new samples to the receive queue, `in.buffer_subtraction` must always be bigger than `in.vectorize`.
A second factor is performance: if `in.buffer_subtraction` is too small it might take long before the node starts to process data since it has to fill almost the complete queue first.
If `in.buffer_subtraction` is too big, the receive buffer might be too small.
Thus, the maximum number of Work Requests to be present in the receive queue is defined as follows:
```c
max_wrs_posted = in.max_wrs - in.buffer_subtraction
```
out:
type: object
properties:
address:
type: string
description: |
This value defines the IPoIB address of the remote node and is used to establish a connection to the remote host—in case of `RDMA_PS_TCP`—or to get the address handle of the remote host—in case of `RDMA_PS_UDP`.
This is similar to `in.address`.
`out.address` has no default value and if it is not defined the node will be set to listening mode and all `out` configuration will be ignored.
**Example**:
```
out = {
address = "10.0.0.1:1337"
}
```
timeout:
type: integer
default: 1000
description: |
This defines the time in milliseconds [`rdma_resolve_addr()`](https://linux.die.net/man/3/rdma_resolve_addr) waits for the resolution of the destination address to complete.
max_wrs:
type: integer
default: 128
description: |
This is similar to `in.max_wrs` but for the send side of the Queue Pair.
In contrast to the receive queue, there is no minimum amount of Work Requests in this queue and it can be filled up completely to `out.max_wrs`.
cq_size:
type: integer
default: 128
description: |
This is similar to `in.cq_size`.
An important side note for the receive completion queue was that it should be able to hold all Work Requests if the receive queue is flushed.
Since no "preparatory" Work Requests are posted to the send queue and and thus all work requests are send out as soon as possible, there is no need for `out.cq_size` to be as big as `out.max_wrs`.
send_inline:
type: boolean
default: true
description: |
It is possible that the CPU copies the data to be sent directly to the HCA.
Then, the HCA can take the data from it's internal memory as soon as it is ready to send it.
This has the advantage that the buffer can be returned immediately to the VILLASnode and that it increases performance.
If this flag is set, the [`infiniband`](../nodes/infiniband.md) node-type checks if a sample is small enough to be sent inline, and if this is the case sends it inline.
max_inline_data:
type: integer
default: 0
description: |
This value represents the maximum number of bytes to be send inline.
The maximum number of this value depends on the HCA.
The settings defaults to zero. However, many HCAs will automatically adjust it to 60.
*Important note*: The greater this value gets, the smaller `out.max_wrs` can be. If `out.max_inline_data` is too big for the number specified in `out.max_wrs`, the node will return an error that the Queue Pair could not be created.
Since this is different for various HCAs, it is not possible for us to give more specified errors.
**Example**:
```
out = {
send_inline = 1,
max_inline_data = 60
}
```
Every sample which is smaller than 60 bytes will be send inline. All other samples will be sent normally.
use_fallback:
type: boolean
default: true
description: |
If an out section with a valid remote entry is present in the configuration file, the node will first bind to the local host channel adapter and subsequentially try to connect to the remote host.
If the latter fails (e.g., because the remote host was not reachable or rejected the connection), there are two possible outcomes: the node can throw an error and abort or it can show a warning and continue in listening mode.
If `use_fallback = true`, the node will fallback to listening mode if it is not able to connect to the remote host.
periodic_signaling:
type: integer
default: <out.max_wrs / 2>
description: |
If a sample is sent inline, no Completion Queue Entry (CQE) is generated.
However, once a while, a CQE must be generated to prevent the Send Queue from overflowing.
Therefore, every `out.periodic_signaling`th sample will be sent normally with signaling.
It turns out that the ideal value in most cases is `out.max_wrs / 2`.
Hence, usually, it is not necessary to explicitly set this value.
- $ref: ../node.yaml

View file

@ -4,7 +4,15 @@
allOf:
- type: object
properties:
format:
$ref: ../format_spec.yaml
server:
type: string
description: A hostname/port combination of the InfluxDB database server.
key:
type: string
description: |
The key is the measurement name and any optional tags separated by commas.
See also: [InfluxDB documentation](https://docs.influxdata.com/influxdb/v0.9/write_protocols/line/#key).
- $ref: ../node.yaml

View file

@ -3,8 +3,80 @@
allOf:
- type: object
required:
- server
- client_id
properties:
format:
$ref: ../format_spec.yaml
server:
type: string
description: |
The bootstrap server `{ip}:{port}` of the Kafka message brokers cluster.
protocol:
type: string
enum:
- PLAINTEXT
- SASL_PLAINTEXT
- SASL_SSL
- SSL
description: |
The [security protocol](https://kafka.apache.org/24/javadoc/org/apache/kafka/common/security/auth/SecurityProtocol.html) which is used for authentication with the Kafka cluster.
client_id:
type: string
description: The Kafka client identifier.
ssl:
type: object
properties:
ca:
type: string
description: Path to a Certificate Authority (CA) bundle which is used to validate broker server certificate.
sasl:
type: object
description: |
An object for configuring the SASL authentication against the broker.
This setting is used if the `protocol` setting is on of `SASL_PLAINTEXT` or `SASL_SSL`.
properties:
mechanisms:
type: string
username:
type: string
password:
type: string
in:
type: object
properties:
consume:
type: string
description: The Kafka topic to which this node-type will subscribe for receiving messages.
group_id:
type: string
description: The group id of the Kafka client used for receiving messages.
out:
type: object
properties:
produce:
type: string
description: The Kafka topic to which this node-type will publish messages.
timeout:
type: number
description: A timeout in seconds for the broker connection.
default: 1.0
- $ref: ../node.yaml

View file

@ -3,8 +3,24 @@
allOf:
- type: object
required:
- endpoint
properties:
format:
$ref: ../format_spec.yaml
endpoint:
type: string
format: uri
entity_id:
type: string
entity_type:
type: string
ssl_verify:
type: boolean
timeout:
type: number
- $ref: ../node.yaml

View file

@ -4,7 +4,130 @@
allOf:
- type: object
properties:
format:
$ref: ../format_spec.yaml
mode:
type: string
enum:
- key
- hash
- channel
default: key
description: |
- `key`: [Get](https://redis.io/commands/get)/[Set](https://redis.io/commands/set) of [Redis strings](https://redis.io/topics/data-types#strings)
- The implementation uses the Redis `MSET` and `MGET` commands.
- `hash`: Hashtables using [hash data-type](https://redis.io/topics/data-types#hashes)
- The implementation uses the Redis `HMSET` and `HGETALL` commands.
- `channel`: [Publish/subscribe](https://redis.io/topics/pubsub)
- The implementation uses the Redis `PUBLISH` and `SUBSCRIBE` commands.
uri:
type: string
format: uri
description: |
A Redis connection URI in the form of: `redis://<user>:<password>@<host>:<post>/<db>`.
host:
type: string
default: localhost
description: |
The hostname or IP address of the Redis server.
You can also connect to Redis server with a URI:
- `tcp://[[username:]password@]host[:port][/db]`
- `unix://[[username:]password@]path-to-unix-domain-socket[/db]`
port:
type: integer
description: The port number of the Redis server to connect to.
default: 6379
path:
type: string
description: A path of a Unix socket which should be used for the connection.
user:
type: string
default: default
description: |
The username which should be used for authentication.
See: https://redis.io/commands/auth
password:
type: string
description: |
The password which should be used for authentication.
See: https://redis.io/commands/auth
db:
type: integer
default: 0
description: |
The logical database which should be used by the Redis client.
See: https://redis.io/commands/select
timeout:
type: object
properties:
connect:
type: number
description: The timeout in seconds for the initial connection establishment.
socket:
type: number
description: The timeout in seconds for executing commands against the Redis server.
keepalive:
type: boolean
default: false
description: Enable periodic keepalive packets.
key:
type: string
default: <node-name>
description: The key which this node will use in the Redis keyspace.
channel:
type: string
default: <node-name>
description: The channel which this node will use when `mode` setting is `channel`.
notify:
type: boolean
default: true
description: |
Use [Redis keyspace notifications](https://redis.io/topics/notifications) to listen for new updates.
This setting is only used if setting `mode` is set to `key` or `hash`.
ssl:
type: object
properties:
enabled:
type: boolean
default: true
description: If enabled the connection to the Redis server will be encrypted via SSL/TLS.
cacert:
type: string
description: A path to a CA certificate file.
cacertdir:
type: string
description: A path to a directory containing CA certificates.
cert:
type: string
description: A path to a client certificate file.
key:
type: string
description: A path to the private key file.
- $ref: ../node.yaml

View file

@ -7,4 +7,10 @@ allOf:
format:
$ref: ../format_spec.yaml
out:
type: object
properties:
netem:
$ref: ../netem.yaml
- $ref: ../node.yaml

View file

@ -4,7 +4,52 @@
allOf:
- type: object
properties:
format:
$ref: ../format_spec.yaml
queuelen:
type: integer
default: <DEFAULT_SHMEM_QUEUELEN>
description: Length of the input and output queues in elements.
samplelen:
type: integer
description: Maximum number of data elements in a single `struct Sample`` for the samples handled by this node.
default: <DEFAULT_SHMEM_SAMPLELEN>
mode:
type: string
default: pthread
enum:
- pthread
- polling
description: |
If set to `pthread`, POSIX condition variables (CV) are used to signal writes between processes.
If set to `polling`, no CV's are used, meaning that blocking writes have to be implemented using polling, leading to performance improvements at a cost of unnecessary CPU usage.
exec:
description: |
Optional name and command-line arguments (as passed to `execve`) of a command to be executed during node startup.
This can be used to start the external program directly from VILLASNode. If unset, no command is executed.
type: array
items:
type: string
in:
type: object
properties:
name:
type: string
description: |
Name of the POSIX shared memory object.
Must start with a forward slash (/).
The same name should be passed to the external program somehow in its configuration or command-line arguments.
out:
type: object
properties:
name:
type: string
description: |
Name of the POSIX shared memory object.
Must start with a forward slash (/).
The same name should be passed to the external program somehow in its configuration or command-line arguments.
- $ref: ../node.yaml

View file

@ -3,8 +3,103 @@
allOf:
- type: object
required:
- signal
properties:
format:
$ref: ../format_spec.yaml
signal:
type: string
enum:
- random
- sine
- square
- triangle
- ramp
- counter
- constant
- mixed
- pulse
description: |
The type of signal which should be generated:
- `random`: a random walk with normal distributed step sizes will be generated.
- `sine`: a sine signal will be generated.
- `square`: a square / rectangle wave will be generated.
- `triangle`: a triangle wave will be generated.
- `ramp`: the generator will produce a ramp signal in the interval `[ 0, 1 / f ]`.
- `counter`: increasing integer counter is generated.
- `constant`: a constant value generated.
- `mixed`: the signals of of each sample are generated by cycling over all remaining signal types.
- `pulse`: generates pulses with a set frequency, phase and width
values:
type: integer
default: 1
description: The number of signals which each of the generated samples should contain.
rate:
type: integer
description: The rate at which sample should be generated by the node.
default: 10
amplitude:
type: number
description: The amplitude of the signal when the `signal` setting is one of `sine`, `square` or `triangle`.
default: 1.0
frequency:
type: number
description: The frequency of the signal when the `signal` setting is one of `sine`, `square`, `triangle`,`pulse` or `ramp`.
default: 1.0
phase:
type: number
default: 0.0
description: Tha pase of the signal when the `signal` setting is one of `sine` or `pulse`.
pulse_width:
type: number
default: 1.0
description: The width of the pulse, with respect to the rate
pulse_low:
type: number
default: 0.0
description: The low value of the pulse signal.
pulse_high:
type: number
default: 1.0
description: The high value of the pulse signal.
stddev:
type: number
default: 0.2
description: The standard deviation of the normal distributed steps if the `signal` setting is set to `random`.
offset:
type: number
default: 0.0
description: Adds a constant offset to each of the generated signals.
limit:
type: integer
default: -1
description: |
Limit the number of generated output samples by this node-type.
A negative number disables the limitation.
realtime:
type: boolean
default: true
description: Wait `1 / rate` seconds between emitting each sample.
monitor_missed:
type: boolean
default: true
description: |
If `true`, the `signal` node-type will count missed steps and warn the user during every iteration about missed steps.
Especially at high rates, it can be beneficial for performance to set this flag to `false`.
Warnings would namely cause system calls which will slow the node down even more, and thus cause even more missed steps.
- $ref: ../node.yaml

View file

@ -4,7 +4,26 @@
allOf:
- type: object
properties:
format:
$ref: ../format_spec.yaml
calibration:
type: object
properties:
scale:
type: number
default: 1.0
description: A scaling factor for calibrating the sensor.
offset:
type: number
default: 0.0
description: An offset for calibrating the sensor.
bus:
type: integer
description: A filter applied to the USB bus number for selecting a specific sensor if multiple are available.
port:
type: integer
description: A filter applied to the USB port number for selecting a specific sensor if multiple are available.
- $ref: ../node.yaml