Date:   Thu, 13 Dec 2018 10:03:59 -0300
From:   Arnaldo Carvalho de Melo <acme@...nel.org>
To:     Quentin Monnet <quentin.monnet@...ronome.com>
Cc:     Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>, netdev@...r.kernel.org,
        oss-drivers@...ronome.com,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Stanislav Fomichev <sdf@...gle.com>
Subject: Re: [PATCH bpf-next 0/8] tools: bpftool: add probes for system and
 device

Em Thu, Dec 13, 2018 at 12:19:14PM +0000, Quentin Monnet escreveu:
> Hi,
> This set add a new command to bpftool in order to dump a list of
> eBPF-related parameters for the system (or for a specific network
> device) to the console. Once again, this is based on a suggestion from
> Daniel.
> 
> At this time, output includes:
> 
>     - Availability of bpf() system call
>     - Availability of bpf() system call for unprivileged users
>     - JIT status (enabled or not, with or without debugging traces)
>     - JIT hardening status
>     - JIT kallsyms exports status
>     - Status of kernel compilation options related to BPF features
>     - Release number of the running kernel
>     - Availability of known eBPF program types
>     - Availability of known eBPF map types
>     - Availability of known eBPF helper functions
> 
> There are three different ways to dump this information at this time:
> 
>     - Plain output dumps probe results in plain text. It is the most
>       flexible options for providing descriptive output to the user, but
>       should not be relied upon for parsing the output.
>     - JSON output is supported.
>     - A third mode, available through the "macros" keyword appended to
>       the command line, dumps the parameters as a series of "#define"
>       directives, that can be included into a C header file for example.
> 
> If the user does not have root privileges (or more precisely, the
> CAP_SYS_ADMIN capability) detection will be erroneous for most
> parameters. Therefore, forbid non-root users to run the command.

Interesting, and helps developers to figure out what is present and help
the user to tell why some specific feature is not working.

Something else we've discussed while in LPC was to present the users
with hints about what to do when something fails, IIRC Joe Stringer
suggested to have hints pointing to the documentation about that
specific error, helping users to setup the system.

In perf we have something similar, for instance, when a !root user is
wanting to use 'perf trace'

$ perf trace
Error:	No permissions to read /sys/kernel/debug/tracing/events/raw_syscalls/sys_(enter|exit)
Hint:	Try 'sudo mount -o remount,mode=755 /sys/kernel/debug/'
$ sudo mount -o remount,mode=755 /sys/kernel/debug/
$ perf trace sleep 1
Error:	Permission denied.
Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
Hint:	For your workloads it needs to be <= 1
Hint:	For system wide tracing it needs to be set to -1.
Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
Hint:	The current value is 2.
$ sudo sh -c "echo 1 > /proc/sys/kernel/perf_event_paranoid"
$ perf trace sleep 1
     <SNIP>
     0.708 ( 0.008 ms): sleep/24196 mmap(len: 217749968, prot: READ, flags: PRIVATE, fd: 3) = 0x7fe0863b5000
     0.728 ( 0.002 ms): sleep/24196 close(fd: 3) = 0
     0.796 (1000.111 ms): sleep/24196 nanosleep(rqtp: 0x7fff78e7ff60) = 0
  1000.953 ( 0.007 ms): sleep/24196 close(fd: 1) = 0
  1000.965 ( 0.003 ms): sleep/24196 close(fd: 2) = 0
  1000.989 (         ): sleep/24196 exit_group()
$ 
$ perf trace --all-cpus
Error:	Operation not permitted.
Hint:	Check /proc/sys/kernel/perf_event_paranoid setting.
Hint:	For system wide tracing it needs to be set to -1.
Hint:	Try: 'sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
Hint:	The current value is 1.
$ 
$ sudo sh -c "echo -1 > /proc/sys/kernel/perf_event_paranoid"'
$ perf trace -a -e open*
     0.000 ( 0.057 ms): pulseaudio/2103 openat(dfd: CWD, filename: 0x712de930, flags: CLOEXEC) = 47
     1.658 ( 0.038 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57cfd0, flags: CLOEXEC) = 47
     1.770 ( 0.013 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57cc60, flags: CLOEXEC) = 47
     1.805 ( 0.029 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57cc40, flags: RDWR|CLOEXEC) = 47
     2.038 ( 0.021 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57d290, flags: CLOEXEC) = 47
     2.085 ( 0.014 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57d0a0, flags: RDWR|CLOEXEC) = 47
     2.119 (33.276 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57d250, flags: RDWR|CLOEXEC|NONBLOCK) = 53
    35.575 ( 0.038 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57d3d0, flags: CLOEXEC) = 47
    35.698 ( 0.006 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57cf90, flags: CLOEXEC) = 47
    35.710 ( 0.003 ms): alsa-sink-ALC2/2181 openat(dfd: CWD, filename: 0x5b57cf70, flags: RDWR|CLOEXEC) = 47
   250.421 ( 0.044 ms): DNS Res~er #41/18571 openat(dfd: CWD, filename: 0x5fc2d024, flags: CLOEXEC) = 138
   365.200 ( 0.038 ms): :17437/17437 openat(dfd: CWD, filename: 0x25211b80) = 129
   365.393 ( 0.013 ms): :17437/17437 openat(dfd: CWD, filename: 0x25211b80) = 129
   366.264 ( 0.031 ms): :17437/17437 openat(dfd: CWD, filename: 0x25211b80) = 129
   366.442 ( 0.012 ms): :17437/17437 openat(dfd: CWD, filename: 0x25211b80) = 524
   367.727 ( 0.026 ms): :17437/17437 openat(dfd: CWD, filename: 0x634a3780) = 129
   426.638 ( 0.036 ms): vim/24205 openat(dfd: CWD, filename: 0xc9504770, flags: CREAT|WRONLY, mode: IRUSR|IWUSR) = 6
^C$

Which makes me realize I have to provide hints for when a !root user tries to
load a BPF program via 'perf trace':

[acme@...co perf]$ ls -la tools/perf/examples/bpf/augmented_raw_syscalls.o
-rw-r--r--. 1 root root 2224 Dec 12 16:45 tools/perf/examples/bpf/augmented_raw_syscalls.o
[acme@...co perf]$ perf trace -a -e open*,tools/perf/examples/bpf/augmented_raw_syscalls.o
event syntax error: 'tools/perf/examples/bpf/augmented_raw_syscalls.o'
                     \___ Failed to load program for unknown reason

(add -v to see detail)
Run 'perf list' for a list of valid events

 Usage: perf trace [<options>] [<command>]
    or: perf trace [<options>] -- <command> [<options>]
    or: perf trace record [<options>] [<command>]
    or: perf trace record [<options>] -- <command> [<options>]

    -e, --event <event>   event/syscall selector. use 'perf list' to list available events
[acme@...co perf]$ file tools/perf/examples/bpf/augmented_raw_syscalls.o
tools/perf/examples/bpf/augmented_raw_syscalls.o: ELF 64-bit LSB relocatable, eBPF, version 1 (SYSV), not stripped
[acme@...co perf]

running with -v ends up with:

libbpf: failed to load object 'tools/perf/examples/bpf/augmented_raw_syscalls.o'
bpf: load objects failed: err=-4009: (Incorrect kernel version)
event syntax error: 'tools/perf/examples/bpf/augmented_raw_syscalls.o'
                     \___ Failed to load program for unknown reason

ouch, even rebuildng the .c file ends up with that error, will debug :-)

[acme@...co perf]$ perf trace -v -a -e open*,tools/perf/examples/bpf/augmented_raw_syscalls.c
bpf: builtin compilation failed: -95, try external compiler
Kernel build dir is set to /lib/modules/4.20.0-rc5/build
set env: KBUILD_DIR=/lib/modules/4.20.0-rc5/build
unset env: KBUILD_OPTS
include option is set to  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/8/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h 
set env: NR_CPUS=8
set env: LINUX_VERSION_CODE=0x41400
set env: CLANG_EXEC=/usr/lib64/ccache/clang
unset env: CLANG_OPTIONS
set env: KERNEL_INC_OPTIONS= -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/8/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h 
set env: PERF_BPF_INC_OPTIONS=-I/home/acme/lib/perf/include/bpf
set env: WORKING_DIR=/lib/modules/4.20.0-rc5/build
set env: CLANG_SOURCE=/home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c
llvm compiling command template: $CLANG_EXEC -D__KERNEL__ -D__NR_CPUS__=$NR_CPUS -DLINUX_VERSION_CODE=$LINUX_VERSION_CODE $CLANG_OPTIONS $PERF_BPF_INC_OPTIONS $KERNEL_INC_OPTIONS -Wno-unused-value -Wno-pointer-sign -working-directory $WORKING_DIR -c "$CLANG_SOURCE" -target bpf $CLANG_EMIT_LLVM -O2 -o - $LLVM_OPTIONS_PIPE
llvm compiling command : /usr/lib64/ccache/clang -D__KERNEL__ -D__NR_CPUS__=8 -DLINUX_VERSION_CODE=0x41400  -I/home/acme/lib/perf/include/bpf  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/8/include -I/home/acme/git/linux/arch/x86/include -I./arch/x86/include/generated  -I/home/acme/git/linux/include -I./include -I/home/acme/git/linux/arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I/home/acme/git/linux/include/uapi -I./include/generated/uapi -include /home/acme/git/linux/include/linux/kconfig.h  -Wno-unused-value -Wno-pointer-sign -working-directory /lib/modules/4.20.0-rc5/build -c /home/acme/git/perf/tools/perf/examples/bpf/augmented_raw_syscalls.c -target bpf  -O2 -o - 
libbpf: loading object 'tools/perf/examples/bpf/augmented_raw_syscalls.c' from buffer
libbpf: section(1) .strtab, size 211, link 0, flags 0, type=3
libbpf: skip section(1) .strtab
libbpf: section(2) .text, size 0, link 0, flags 6, type=1
libbpf: skip section(2) .text
libbpf: section(3) raw_syscalls:sys_enter, size 440, link 0, flags 6, type=1
libbpf: found program raw_syscalls:sys_enter
libbpf: section(4) .relraw_syscalls:sys_enter, size 48, link 11, flags 0, type=9
libbpf: section(5) raw_syscalls:sys_exit, size 192, link 0, flags 6, type=1
libbpf: found program raw_syscalls:sys_exit
libbpf: section(6) .relraw_syscalls:sys_exit, size 32, link 11, flags 0, type=9
libbpf: section(7) maps, size 112, link 0, flags 3, type=1
libbpf: section(8) license, size 4, link 0, flags 3, type=1
libbpf: license of tools/perf/examples/bpf/augmented_raw_syscalls.c is GPL
libbpf: section(9) version, size 4, link 0, flags 3, type=1
libbpf: kernel version of tools/perf/examples/bpf/augmented_raw_syscalls.c is 41400
libbpf: section(10) .llvm_addrsig, size 8, link 11, flags 80000000, type=1879002115
libbpf: skip section(10) .llvm_addrsig
libbpf: section(11) .symtab, size 336, link 1, flags 0, type=2
libbpf: maps in tools/perf/examples/bpf/augmented_raw_syscalls.c: 4 maps in 112 bytes
libbpf: map 0 is "__augmented_syscalls__"
libbpf: map 1 is "__bpf_stdout__"
libbpf: map 2 is "pids_filtered"
libbpf: map 3 is "syscalls"
libbpf: collecting relocating info for: 'raw_syscalls:sys_enter'
libbpf: relo for 10 value 84 name 106
libbpf: relocation: insn_idx=5
libbpf: relocation: find map 3 (pids_filtered) for insn 5
libbpf: relo for 13 value 56 name 38
libbpf: relocation: insn_idx=16
libbpf: relocation: find map 2 (syscalls) for insn 16
libbpf: relo for 6 value 28 name 151
libbpf: relocation: insn_idx=48
libbpf: relocation: find map 1 (__augmented_syscalls__) for insn 48
libbpf: collecting relocating info for: 'raw_syscalls:sys_exit'
libbpf: relo for 10 value 84 name 106
libbpf: relocation: insn_idx=5
libbpf: relocation: find map 3 (pids_filtered) for insn 5
libbpf: relo for 13 value 56 name 38
libbpf: relocation: insn_idx=17
libbpf: relocation: find map 2 (syscalls) for insn 17
bpf: config program 'raw_syscalls:sys_enter'
bpf: config program 'raw_syscalls:sys_exit'
libbpf: create map __bpf_stdout__: fd=3
libbpf: create map __augmented_syscalls__: fd=4
libbpf: create map syscalls: fd=5
libbpf: create map pids_filtered: fd=6
libbpf: load bpf program failed: Operation not permitted
libbpf: failed to load program 'raw_syscalls:sys_enter'
libbpf: failed to load object 'tools/perf/examples/bpf/augmented_raw_syscalls.c'
bpf: load objects failed: err=-4009: (Incorrect kernel version)
event syntax error: 'tools/perf/examples/bpf/augmented_raw_syscalls.c'
                     \___ Failed to load program for unknown reason

(add -v to see detail)
Run 'perf list' for a list of valid events

 Usage: perf trace [<options>] [<command>]
    or: perf trace [<options>] -- <command> [<options>]
    or: perf trace record [<options>] [<command>]
    or: perf trace record [<options>] -- <command> [<options>]

    -e, --event <event>   event/syscall selector. use 'perf list' to list available events
[acme@...co perf]$ 

Ok, out to figure this out :-)

Powered by blists - more mailing lists