The node blocks a certain amount of samples to use in its queues.
Before this commit, the only moment to release them to the framwork was
during ib_read()/ib_write().
But, there were a couple of problems. In the following I will take
ib_read() as example, but ib_write() will be analogous.
The first problem was:
1. If a QP disconnect, all Work Requests get invalidated and will be
"flushed" to a Completion Queue.
A possible solution would be, to save them in an intermediate buffer.
We could then "exchange" these samples with the framework as soon as the node
connects again and ib_read() is called again. So, we would get valid
samples from the framwork, post them, and give the "invalidated" samples back.
But, there is a second problem:
2. We cannot assume that ib_read() is ever called again after
ib_disconnect(). This is for example the case if the disconnect is
triggered by ib_stop() and not by an external node that disconnects.
This would result in a memory leak, since the samples would never be
returned to the framework, although the node is stopped.
Because of this second problem, I decided to return all samples with
sample_put() in the disconnect function. An additional benefit is that
this is more convenient than another buffer to temporarily safe the
invalidated samples.
Before, the node would throw an error as soon as it cannot connect to
the remote host. Now, it will throw a warning and switch to listening
mode (in which it will wait for another node to connect).
In the case that a node was already disconnected but not stopped,
rdma_cm_get_event always blocked and we coulnd't join the threads. This
is solved in this commit by registering SIGUSR1 to the CM event thread.
This bug originated in issue #152
The actual maximum size for inline mode is now returned to the user and
there is a check that inline_mode is either 0 or 1. Furthermore, this
commit includes a minor improvement in ib_write()
The user can set the maximum size of the inline data and the node checks
if a sample can be send inline. This commit doesn't contain a info
message to the user about what the final max inline size will be. (The
HCA will probably change the value set by the user.)
Now, ib_write() reads every cycle cnt values from the Completion Queue.
If it is not able to return them to the framework immediately, it
temporarily saves them on a stack.
ib_write() checks every cycle if the stack is non-empty and if it is
possible to return values from the stack to the framework.
The functions now look like this
int node_read(struct node *n, struct sample *smps[], unsigned cnt, unsigned *release);
int node_write(struct node *n, struct sample *smps[], unsigned cnt, unsigned *release);
This commit enables nodes to control how many samples will
be released by the framework through *release
