The netlink message buffer is preallocated to a page and later
expanded as needed. Everything was properly paded and zeroed
out except for the unused part at the end. Use calloc() to
allocate the buffer.
I found the following bug, where nlmsg_ok() in lib/msg.c would
incorrectly return 'true' when the input argument 'remaining' was a negative
number. This happens when the message is not aligned the way that libnl
expects (although it is still legal).
In the comparison of the signed and unsigned numbers on line 284, the signed
number gets converted to an unsigned number, which is unexpected and
naturally produces a bug. My patch is below. The cast is ugly, but it
fixes the problem.
In order for the interface to become more thread safe, the error
handling was revised to no longer depend on a static errno and
error string buffer.
This patch converts all error paths to return a libnl specific
error code which can be translated to a error message using
nl_geterror(int error). The functions nl_error() and
nl_get_errno() are therefore obsolete.
This change required various sets of function prototypes to be
changed in order to return an error code, the most prominent
are:
struct nl_cache *foo_alloc_cache(...);
changed to:
int foo_alloc_cache(..., struct nl_cache **);
struct nl_msg *foo_build_request(...);
changed to:
int foo_build_request(..., struct nl_msg **);
struct foo *foo_parse(...);
changed to:
int foo_parse(..., struct foo **);
This pretty much only leaves trivial allocation functions to
still return a pointer object which can still return NULL to
signal out of memory.
This change is a serious API and ABI breaker, sorry!
As pointed out by Regis Hanna, a considerable performance gain can be
achieved by using malloc() over calloc() when allocating netlink message
buffers. This is likely due to the fact that we use a complete page for
each message.
Patrick McHardy reported a problem where pointers to the
payload of a netlink message as returned by f.e. the
nesting helpers become stale when the payload data
chunk is reallocated.
In order to avoid further problems, the payload chunk is
no longer extended on the fly. Instead the allocation is
made during netlink message object allocation time with
a default size of a page which should be fine for the
majority of all users. Additionally the functions
nlmsg_alloc_size() and nlmsg_set_default_size() have been
added to allocate messages of a particular length and to
modify the default message size.
