xdp-project-bpf-examples/ktrace-CO-RE

Example using CO-RE to access ktrace

• Compiling kprobes for different CPU architectures
  • Gotcha with target bpf
• Bpftrace

This is an example that show how CO-RE came be utilised… to learn CO-RE and BTF better.

Compiling kprobes for different CPU architectures

When BPF-tracing kprobes then the header files need to use the correct asm-level C-calling convention for the architecture that this is getting compiled for, as BPF will be hook/tap directly into the kernels function call. These are the raw CPU registers, which we later via BTF demonstrate howto use to CO-RE to get the C-type back.

The header include file <bpf/bpf_tracing.h> takes care of defining macros for accessing the members in kernels struct pt_regs, as the struct member names differ between architectures. This header file is provided by libbpf (or distros libbpf-devel package).

The architecture specific definition of struct pt_regs can be extracted via generating a vmlinux.h file on the ARCH target. To ease getting the structs this git repo contains the arch include files here: ../headers/vmlinux/arch/

You should not include them directly, but instead include vmlinux_local.h to get the surrounding types defined.

Gotcha with target bpf

This git-repo compile with clang -target bpf. This cause issues for ktrace, because the fallback mechanism in header file <bpf/bpf_tracing.h> cannot determine the ARCH from compiler defines.

To solve this our makefile system in ../lib/common.mk detect the ARCH (via uname -m) and defines the __TARGET_ARCH_$(ARCH) in BPF_CFLAGS.

Without this fix the following compile warnings occurs:

 warning: implicit declaration of function 'PT_REGS_PARM1' is invalid in C99
 int BPF_KPROBE(udp_send_skb, struct sk_buff *skb)
     ^
 /usr/include/bpf/bpf_tracing.h:385:20: note: expanded from macro 'BPF_KPROBE'

When loading via libbpf the following error happens:

 libbpf: failed to find BTF for extern 'PT_REGS_PARM1': -2

Bpftrace

Pre-investigations with bpftrace.

Simple bpftrace command to look at callstack for udp_send_skb:

• bpftrace -e 'kprobe:udp_send_skb { @[kstack] = count() }'

$ sudo bpftrace -e 'kprobe:udp_send_skb { @[kstack] = count() }'
Attaching 1 probe...
^C

@[
    udp_send_skb+1
    udp_sendmsg+2407
    sock_sendmsg+87
    __sys_sendto+238
    __x64_sys_sendto+37
    do_syscall_64+51
    entry_SYSCALL_64_after_hwframe+68
]: 70