Refactor code for how events are handled in the user space
application. Preparation for adding an additional event type which
should not be handled by the normal functions for printing RTT and
flow events.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Wait with sending a flow open message until a reply has been seen for
the flow. Likewise, only emit a flow closing event if the flow has
first been opened (that is, a reply has been seen).
This introduces potential (but unlikely) concurrency issues for flow
opening/closing messages which are further described in the README.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Perform both timestamping and matching on both ingress and egress
hooks. This makes it more similar to Kathie's pping, allowing the tool
to capture RTTs in both directions when deployed on just a single
interface.
Like Kathie's pping, by default filter out RTTs for packets going to
the local machine (will only include local processing delays). This
behavior can be disabled by passing the -l/--include-local option.
As packets that are timestamped on ingress and matched on egress will
include the local machines processing delay, add the "match_on_egress"
member to the JSON output that can be used to differentiate between
RTTs that include the local processing delay, and those which don't.
Finally, report the source and destination addresses from the perspective
of the reply packet, rather than the timestamped packet, to be
consistent with Kathie's pping.
Overall, refactor large parts of pping_kern to allow both timestamping
and matching, as well as updating both the flow and reverse flow and
handle flow-events related to them, in one go. Also update README to
reflect changes.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add an option (-R, --rtt-rate) to adapt the rate sampling based on the
RTT of the flow. The sampling rate will be C * RTT, where C is a
configurable constant (ex 1.0 to get one sample every RTT), and RTT
is either the current minimum (default) or smoothed RTT of the
flow (chosen via the -t or --rtt-type option).
The smoothed RTT (sRTT) is updated for each calculated RTT, and is
calculated in a similar manner to srtt in the kernel's TCP stack. The
sRTT is a moving average of all RTTs, and is calculated according to
the formula:
srtt = 7/8 * prev_srtt + 1/8 * rtt
To allow the user to pass a non-integer C (ex 0.1 to get 10 RTT
samples for every RTT-period), fixed-point arithmetic has been used
in the eBPF programs (due to lack of support for floats). The maximum
value for C has been limited to 10000 in order for it to be unlikely
that the C * RTT calculation will overflow (with C = 10000, overflow
will only occur if RTT > 28 seconds).
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Only push flow events for opening/closing flows if the
creation/deletion of the flow-state was successful (as indicated by
the bpf_map_*_elem() return value). This should avoid outputting
several flow creation/deletion messages in case multiple instances are
trying to create/delete a flow concurrently, as could theoretically
occur previously.
Also set the last_timestamp value before creating a new flow, to avoid
a race condition where the userspace cleanup might incorrectly
determine that a flow is old before the last_timestamp value can be
set. Explicitly skip the rate-limit for the first packet of a new flow
to avoid it failing the rate-limit. This also fixes an issue where the
first packet of a new flow would previously fail the rate-limit if the
rate-limit was higher than current time uptime (CLOCK_MONOTONIC).
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add command-line flags for each protocol that pping should attempt to
parse and report RTTs for (currently -T/--tcp and -C/--icmp). If no
protocol is specified assume TCP. To clarify this, output a message
before start on how ePPing has been configured (stating output format,
tracked protocols and which interface to run on).
Additionally, as the ppviz format was only designed for TCP it does
not have any field for which protocol an entry belongs to. Therefore,
emit a warning in case the user selects the ppviz format with anything
other than TCP.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Allow pping to passivly monitor RTT for ICMP echo request/reply
flows. Use the echo identifier as ports, and echo sequence as packet
identifier.
Additionally, add protocol to standard output format in order to be
able to distinguish between TCP and ICMP flows.
The ppviz format does not include protocol, making it impossible to
distinguish between TCP and ICMP traffic. Will add warning if ppviz
format is used together with ICMP traffic in the future.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
The echoed TCP timestamp (TSecr) is only valid if the ACK flag is
set. So make sure to only attempt to match on ACK packets.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Also intercept SIGTERM (in addition the the previously intercepted
SIGINT) and perform graceful shutdown.
Perhaps it also makes sense to perform graceful shutdown on some
additional signals, like SIGHUP and SIGQUIT?
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
The libbpf API has deprecated a number of functions used by the pping
loader. While a couple of functions have simply been renamed,
bpf_object__find_program_by_title has been completely deprecated in
favor of bpf_object__find_program_by_name. Therefore, change so that
BPF programs are found based on the C function names rather than
section names.
Also remove defines of section names as they are no longer used, and
change the section names in pping_kern.c to use "tc" instead of
"classifier/ingress" and "classifier/egress".
Finally replace the flags json_format and json_ppviz in pping_config
with a single enum for the different output formats. This makes the
logic for which output format to use clearer compared to relying on
multiple (supposedly) mutually exclusive flags (and implicitly
assuming standard format if neither flag was set).
One potential concern with this commit is that it introduces some
"magical strings". In case the function names in pping_kern.c are
changed it will require multiple changes in pping.c.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
The rate-limit and cleanup-interval arguments were only verified to be
positive. Add a check for an upper bound to avoid user being able to
pass values that result in an internal overflow. The limits for both
rate-limit and cleanup-interval have been set to one week which should
be more then enough for any reasonable user.
Additionally, disable the period cleanup entirely if the value 0 is
passed to cleanup-interval.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Make several changes to functions related to attaching and detaching
the BPF programs:
- Check the BPF program id when detaching programs to ensure that the
correct programs are removed.
- When attaching tc-programs, keep track of if the clsact qdisc was
created or existed previously. Attempt to delete the qdisc if it was
created and attaching failed. If the --force argument was given, also
attempt to delete qdisc on shutdown in case it did not previously
exist.
- Rely on XDP flags to replace existing XDP program if --force is used
rather than explicitly detaching any XDP program first.
- Print out hints for why pping might have failed attaching the XDP
program.
Also, use libbpf_strerror instead of strerror to better display
libbpf-specific error codes, and for more reliable error handling in
general (don't need to ensure the error codes are positive).
Finally, change return codes of tc programs to TC_ACT_UNSPEC from
TC_ACT_OK to allow other TC-BPF programs to be used on the same
interface as pping.
Concerns with this commit:
- When attaching a tc program libbpf will emit a warning if the
clsact qdisc already exists on the interface. The fact that the
clsact already exists is not an issue, and is handled in tc_attach
by checking for EEXIST, so the warning could be a bit
misleading/confusing for the user.
- The tc_attach and xdp_attach functions attempt to return the u32
prog_id in an int. In case the programs are assigned a very high
id (> 2^31) this may cause it to be interpreted as an error instead.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
For some machines, XDP may not be suitable due to ex. lack of XDP
support in NIC drivers or another program already being attached to
the XDP hook on the desired interface. Therefore, add an option to use
the tc-ingress hook instead of XDP to attach the pping ingress BPF
program on.
In practice, this adds an additional BPF program to the object file (a
TC ingress program). To avoid loading an unnecessary BPF program, also
explicitly disable autoloading for the ingress program not selected.
Also, change the tc programs to return TC_ACT_OK instead of
BPF_OK. While both should be compatible, the TC_ACT_* return codes
seem to be more commonly used for TC-BPF programs.
Concerns with this commit:
- The error messages for XDP attach failure has gotten slightly less
descriptive. I plan to improve the code for attaching and detaching
XDP programs in a separate commit, and will then address that.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
libbpf v0.4 added an API for attaching/detaching TC-BPF programs. So
use the new API to attach the tc program instead of calling on an
external script (which uses the tc command line utility).
Avoid removing the clsact qdisc on program shutdown or error, as
there's currently no convenient way to ensure the qdisc isn't used by
other programs as well. This means pping will not completely clean up
after itself, but this is a safer alternative than always destroying
the qdsic as done by the external script, which may pull the rug out
underneath other programs using the qdisc.
Finally, remove the pin_dir member from the configuration as pping no
longer pins any programs or maps, and remove deleted tc loading
scripts from README.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add cleanup code to load_attach_bpfprogs function, so it should always
unpin tc program, and additionally detach the tc and xdp programs in
case of any failure.
Also unpin tc program directly after attaching it (rather than on
program shutdown), so that multiple instances of pping can be run
simultaneously (on different interfaces).
Finally, rename some of the functions for attaching/detaching tc
programs to be more consistent with the xdp ones.
Note: Still need to keep a copy of most of the cleanup code in main as
well, as the tc and xdp programs also need to be detached on program
shutdown or if later functions fail.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Update README, mainly add a new section with a brief descriptions and
some examples of the output formats.
Also, update the files and maps list to reflect recent changes (BPF
programs can now push flow-events, and the map rtt_events has been
renamed to just events.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Simplify the three output functions by breaking them up into smaller
helper functions. Also introduce the pping_event union, which can hold
either an rtt_event or flow_event.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Make the flow_timeout function call the current output function to
simulate a flow-closing event. Also some other minor cleanup/fixes.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add "flow events" (flow opening or closing so far) which will trigger
a printout of message.
Note: The ppviz format will only print out the traditional rtt events
as the format does not include opening/closing messages.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Use a JSON-writer library from iproute instead of complicated printf
statement. Also output timestamp, rtt and min_rtt as integers in
nanoseconds, rather than floats in seconds.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Change order of parameters for format_ip_address to follow the
convention of the printf functions where buffer is placed first,
instead of the conventions of the inet_ntop functions where buffer is
placed last.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add per-flow tracking of number of packets and bytes
sent/received. Add these to the JSON output format.
Also update README regarding concurrency issue when updating these
statistics.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Also, remove comments about concurrency issues from code in
pping_kern.c as it is now documented in README.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
The format option can take the values "standard" (default), "json" and
ppviz (new name for "machine-friendly").
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add Kathie's "machine friendly" as an optional output format when
passing '-m' or '--machine-friendly' to pping. This format can be used
together with Kathie's ppviz tool to visaulize the output.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add the option to output in JSON format by passing '-j' or '--json' to
pping. Include the protocol in the JSON format, and fix so kernel-side
actually stores the protocol in the flow_address struct.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
To add timestamp to output, push the timestamp when packet was
processed from kernel as part of the rtt-event. Also keep track of
minimum encountered RTT for each flow in kernel, and also push that as
part of the RTT-event.
Additionally, avoid pushing RTT messages at all if no flow-state
information can be found (due to ex. being deleted from egress side),
as no valid min-RTT can then be given. Furthermore, no longer delete
flow-information once seeing the FIN-flag on egress in order to keep
useful flow-state around for RTT-messages longer. Due to the
FIN-handshake process, it is sufficient if the ingress program deletes
the flow-state upon seeing FIN. However, still delete flow-state from
either ingress or egress upon seeing RST flag, as RST does not have a
handshake process allowing for delayed deletion.
While minimum RTT could also be tracked from the userspace process,
userspace is not aware of when the flow is closed so would have to add
additional logic to keep track of minimum RTT for each flow and
periodically clean them up. Furthermore, keeping RTT statistics in the
flow-state map is useful for implementing future features, such as an
RTT-based sampling interval. It would also be useful in case pping is
changed to no longer have a long-running userspace process printing
out all the calculated RTTs, but instead simply occasionally looks up
the RTT from the flow-state map.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
When both BPF programs are kept in the same file, no longer need to
pin the maps in order to share them between the programs.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Update documentation to reflect the current state of pping (after
merging pping_kern_tc and pping_kern_xdp into a single file).
Also add another point to the TODO list that has been discussed at a
previous meeting.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Reduce IPV6_EXT_MAX_CHAIN to 3 to avoid hitting the verifier limit of
processing 1 million instructions, This results in fewer loops in
parsing_helpers.h/skip_ip6hdrnext which simplifies the verifier
analysis. IPv6 extension headers do not appear to be that common, so
this is unlikely to cause a considerable limitation.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Merge the pping_kern_tc.c, pping_kern_xdp.c and pping_helpers.h into
the single file pping_kern.c. Do not change any of the BPF code,
except renaming the map ts_start to packet_ts.
To handle both BPF programs kept in single ELF-file, change loading
mechanism to extract and attach both tc and XDP programs from it. Also
refactor main-method into several smaller functions to reduce its
size.
Finally, added the --force (-f) and --cleanup-interval (-c) options to
the argument parsing, and improved the parsing of the
--rate-limit (-r) option.
NOTE: The verifier rejects program in it's current state as too
large (over 1 million instructions). Setting the TCP_MAX_OPTIONS in
pping_kern.c to 5 (or less) solves this. Unsure at the moment what
causes the verifier to think the program is so large, as the code in
pping_kern.c is identical to the one from the three files it was
merged from.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add a check that opt_size is at least 2 in
pping_helpers.h/prase_tcp_ts, otherwise terminate the loop
unsucessfully. Only check the lower bound of opt_size, the upper
bound will be checked in the first step of the next loop iteration
anyways.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Declare opt_size in pping_helpers.h/parse_tcp_ts volatile to ensure
compiler always reads it from stack as u8, which avoids confusing the
verifier into thinking it might have a negative value.
Old solution of having &=0x3f before adding opt_size to pos could
potentially cause weird behavior if a packet with an invalid TCP
option size arrived (for example, if opt_size was 64 it would be
interpreted as 0, and the loop would simply check the same position
again on each iteration). Simply changing the check to 0xff was not
possible because the compiler would optimize that away (as it knows
that to have no effect on a u8).
Also change check that TCP timestamp is not outside of boundaries from
pos+opt_size to pos+10. Before declaring opt_size as volatile compiler
automatically did this transformation, but now have to explicitly do
this. If this conversion is not done the verifier will reject the
program as it due to its goldfish memory isn't sure that opt_size has
to be 10 at this point.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Refactor init_rodata to search for the first map with ".rodata" in its
name. Should be more robust than previous solution which first tried
to construct the name for the rodata map, and then find the map by
name.
Also remove some outcommented code that was not used.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Update the README, the pping diagram (eBPF_pping_design.png) and TODO
to be more up to date with the current implementation.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Implement basic mechanic for parsing arguments from userspace and
passing them to a global config variable in the BPF programs.
This also changes the basic use of the program from:
$./pping interface
to:
$./pping -i interface
Also, revert to using the memset solution for the map_ipv4_to_ipv6
function to avoid the ipv4_prefix constant being stored in the .rodata
section. This makes it easier to set the value for the global config
variable from userspace, as the only thing left in the .rodata section
is the config struct.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add a check that to ensure verifier that opt_size is positive in case
its been read in from stack. Also enable (uncomment) the flow-state
cleanup from the XDP program as the added check avoids verifier
rejection.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Verifier might have rejected XDP program due to opt_size being loaded
from memory, see
https://blog.path.net/ebpf-xdp-and-network-security. Add check of
opt_size to attempt to convince verifier that it's not a negative
value or anything else crazy. Leads to verifier instead thinking the
program is too large (over 1m instructions).
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add parsing of TCP FIN/RST to determine if connection is being closed,
and if so delete state directly from BPF programs.
Only enabled on tc-program, as verifier is unhappy about in on XDP
side for some reason.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Remove some out-commented code. Also use bpf_object__unpin_maps
instead of manually unpinning the ts_start map. Additionally, change
map_ipv4_to_ipv6 to use clearer implementation (that now also works
for tc due to always using libbpf to load program).
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Load and pin the tc-bpf program in pping.c using libbpf, and only
attach the pinned program using iproute. That way, can use features
that are not supported by the old iproute loader, even if iproute does
not have libbpf support.
To support this change, extend bpf_egress_loader with option to load
pinned program. Additionally, remove configure script and parts of
Makefile that are no longer needed. Furthermore, remove multiple
definitions of ts_start map, and place singular definition in
pping_helpers.h which is included by both BPF programs.
Also, some minor fixes based on Toke's review.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Update SAMPLING_DESIGN.md, partly based on discussions during the
meeting with Red Hat on 2021-03-01.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add a per-flow rate limit, limiting how often new timestamp entries
can be created. As part of this, add per-flow state keeping track
of when last timestamp was created and last seen identifier for each
flow.
Additionally, remove timestamp entry as soon as RTT is
calculated, as last seen identifier is used to find first unique value
instead. Furthermore, remove packet_timestamp struct and only use
__u64 as timestamp, as used memeber is no longer needed.
This initial commit lacks cleanup of flow-state, user-configuration of
rate limit, mechanism to handle bursts etc.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
Add a document outlining my thoughts for how to implement
sampling. Intended both as a basis for discussion, as well as being a
form of documentation.
Signed-off-by: Simon Sundberg <simon.sundberg@kau.se>
