It is confusing to name the option what will wait for packets
on the file descriptor, for --poll, just because the function
call have this name. It confusing as AF_XDP users often want
to busy-poll for packets (for max performance reasons).
Name the new option --wakeup instead of --wait as the effect
of waiting for the FD is that the kernel needs to wakeup
the userspace process.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
The macros XSK_BTF_READ_xxx doesn't handle or report on errors
if the input or field were wrong, which is not uncommon for macros.
The bad behavior is that the macro continue to dereference the
pointer even-when xsk_btf__read_xxx() functions returns an error.
This often leads to crashing the application.
Change macro to only dereference when no errors were reported.
Side-note: The compiler warnings will detect if API user didn't
init the 'dest' value with a default value. As the effect of
the change is that the 'dest' value will not be touch, and thus
contain the value before the macro was used.
Example:
warning: ‘mark’ may be used uninitialized in this function
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Zero init xdp_hints_rx_time, which means xdp_hints_rx_time.btf_type_id
is zero if setup_btf_info() didn't find the struct name.
In print_meta_info_via_btf() we also exit if btf_id is zero, as
this isn't a valid BTF type id.
Thus, we don't need to explicitly handle "error" case of a btf_type_id
being zero, it will just naturally not match when userspace could not
find the struct name.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
The 'struct xdp_hints_rx_time' exist both in kernel-side (af_xdp_kern.c)
and userspace program af_xdp_user.c, but are very different.
The kernel-side is the actual XDP-hints memory layout used in metadata area.
The userspace side uses the same struct name, but contains BTF info
that allow us to access the struct members from kernel-side struct.
The BTF info is extracted on userspace startup, from the BPF-prog
ELF data BTF section.
Caching BTF lookups at startup is done, as target application
is a real-time application.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
The local BTF code in af_xdp_user.c was only used for
debugging the BTF structures while developing and testing
the lib_xsk_extend.c code.
If is confusing for other reading the code, so simply
remove thisi, as the lib_xsk_extend.c should hide
these details fpr API users.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Program fails on newer kernels, with error message:
libbpf: Netlink-based XDP prog detected, please unload it in order to launch AF_XDP prog
ERROR: Can't setup AF_XDP socket "Invalid argument"
Since kernel v5.13 libbpf version, when bpf_link support is
detected then libbpf/xsk require XDP/BPF programs use this feature.
See kernel commit 10397994d30f ("libbpf: xsk: Use bpf_link").
To continue using our netlink-based XDP attach approach,
instruct libbpf/xsk to not be in change of loading the
BPF-prog. As our XDP-prog is special it also makes
to control this ourselves.
This is achived with the flag XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD
xsk_cfg.libbpf_flags = XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD
Remember to update the xskmap manually.
Fixes: 10397994d30f ("libbpf: xsk: Use bpf_link")
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Add a large comments to the most central piece of code that
access the right offset into the metadata based on the BTF
info we have extracted.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
The root cause of the slowdown on the first packet with timestamps,
as primary related to refilling the fill-queue, which happend before
process_packet.
Primary cause, as first packet still see a slowdown of around 4 usec
while it was around 9 usec before.
In commit 5df5332f23 ("AF_XDP-interaction: config AF_XDP frame_headroom")
the fill-queue size was increase to be larger. This caused the
first call to handle_receive_packets() to refill too many frames
into the fill-queue.
Patch split out refill into function restock_receive_fill_queue()
which is now called *after* process_packet step, but before
releasing RX packets via xsk_ring_cons__release(&xsk->rx).
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
This API xsk_btf__read_member() is faster, but it was not the
root cause of the slowdown on the first packet with timestamps.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
With the current xsk_btf__read() we are seeing a slowdown on the
first packet with timestamps. The theory is this is caused
by the allocation for the cached hashmap entry.
meta-time rx_ktime:870394156129518 time_now:870394156139039 diff:9521 ns
meta-time rx_ktime:870396208293894 time_now:870396208295098 diff:1204 ns
meta-time rx_ktime:870398256286553 time_now:870398256287772 diff:1219 ns
Create a new API that can access struct members more directly
via caching the xsk_btf_member offset + size in the C-code API user.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
The program segfaulted if using the XSK_BTF_READ_INTO macro
with field as a C-string e.g. "rx_ktime".
Add comments to highlight this happens.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Based on:
Subj: libbpf: Helpers to access XDP hints based on BTF definitions
https://lore.kernel.org/all/20210803010331.39453-15-ederson.desouza@intel.com/
From: Ederson de Souza <ederson.desouza@intel.com>
Ederson says:
A new set of functions to help get the BTF definition of XDP hints
structure and get the information based on it.
`xsk_umem__btf_id` helps retrieve the BTF id of XDP metadata.
`xsk_btf__init` sets up a context based on the BTF, including a hashmap,
so that subsequent queries are faster.
`xsk_btf__read` returns a pointer to the position in the XDP metadata
containing a given field.
`xsk_btf__has_field` checks the presence of a field in the BTF.
`xsk_btf__free` frees up the context.
Besides those, a macro `XSK_BTF_READ_INTO` acts as a convenient helper
to read the field contents into a given variable.
Note that currently, the hashmap used to speed-up offset location into
the BTF doesn't use the field name as a string as key to the hashmap. It
directly uses the pointer value instead, as it is expected that most of
time, field names will be addressed by a shared constant string residing
on read-only memory, thus saving some time. If this assumption is not
entirely true, this optimisation needs to be rethought (or discarded
altogether).
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
This is a copy of the kernels/libbpf hashmap
tools/lib/bpf/hashmap.{h,c}
The plan is to prototype an AF_XDP/xsk userspace API for accessing
BTF information, that should be moved to libbpf (or libxdp). Thus,
this hashmap code will become avail if successful.
Original kernel commit 553db8ba73df ("libbpf: add resizable non-thread
safe internal hashmap") (Author: Andrii Nakryiko). Thus, giving Andrii
author credit in this git commit.
Andrii Nakryiko said:
libbpf: add resizable non-thread safe internal hashmap
There is a need for fast point lookups inside libbpf for multiple use
cases (e.g., name resolution for BTF-to-C conversion, by-name lookups in
BTF for upcoming BPF CO-RE relocation support, etc). This patch
implements simple resizable non-thread safe hashmap using single linked
list chains.
Four different insert strategies are supported:
- HASHMAP_ADD - only add key/value if key doesn't exist yet;
- HASHMAP_SET - add key/value pair if key doesn't exist yet; otherwise,
update value;
- HASHMAP_UPDATE - update value, if key already exists; otherwise, do
nothing and return -ENOENT;
- HASHMAP_APPEND - always add key/value pair, even if key already exists.
This turns hashmap into a multimap by allowing multiple values to be
associated with the same key. Most useful read API for such hashmap is
hashmap__for_each_key_entry() iteration. If hashmap__find() is still
used, it will return last inserted key/value entry (first in a bucket
chain).
For HASHMAP_SET and HASHMAP_UPDATE, old key/value pair is returned, so
that calling code can handle proper memory management, if necessary.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
Don't allow loading the default xsk BPF-prog if not specifying
any BPF-prog. We have a need for our own BPf-prog.
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
To verify the contents of the incomming packets add a
function print_pkt_info() that decode part of the packet
headers and print IP-header src+dst (both IPv4 and IPv6).
Signed-off-by: Jesper Dangaard Brouer <brouer@redhat.com>
