Message-ID: <20250225-radical-piquant-tench-4d2588-mkl@pengutronix.de>
Date: Tue, 25 Feb 2025 23:57:47 +0100
From: Marc Kleine-Budde <mkl@...gutronix.de>
To: Eduard Zingerman <eddyz87@...il.com>
Cc: Chris Ward <tjcw01@...il.com>, Alexei Starovoitov <ast@...nel.org>, 
	Daniel Borkmann <daniel@...earbox.net>, John Fastabend <john.fastabend@...il.com>, 
	linux-kernel@...r.kernel.org, Chris Ward <tjcw@...ibm.com>, bpf@...r.kernel.org
Subject: Re: eBPF verifier does not load libxdp dispatcher eBPF program

On 25.02.2025 10:21:11, Eduard Zingerman wrote:
> On Tue, 2025-02-25 at 16:55 +0100, Marc Kleine-Budde wrote:
> > On 23.01.2023 12:35:41, Chris Ward wrote:
> > > The 5.15.0 kernel (built by 'git checkout v5.15' from the kernel.org
> > > torvalds tree) fails in the same way that the 6.2.0-rc5+ kernel fails.
> > > So it seems that something Canonical did for the Ubuntu 20.04 kernel
> > > causes eBPF to work correctly.
> > >
> > > On Mon, 23 Jan 2023 at 11:06, Chris Ward <tjcw01@...il.com> wrote:
> > > >
> > > > I am trying to use the 'bleeding edge' kernel to determine whether a
> > > > problem I see has already been fixed, but with this kernel the eBPF
> > > > verifier will not load the dispatcher program that is contained within
> > > > libxdp. I am testing kernel commit hash 2475bf0 which fails, and the
> > > > kernel in Ubuntu 22.04 (5.15.0-58-generic) works properly. I am
> > > > running the test case from
> > > > https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth ; to
> > > > build it go to the AF_XDP-filter directory and type 'make', and to run
> > > > it go to the AF_XDP-filter/runscripts/iperf3-namespace directory and
> > > > type 'sudo FILTER=af_xdp_kern PORT=50000 ./run.sh' .
> > > > The lines from the run output indicating the failure are
> > > > libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
> > > > libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
> > > > Func#11 is safe for any args that match its prototype
> > > > btf_vmlinux is malformed
> > > > reg type unsupported for arg#0 function xdp_dispatcher#29
> > > > 0: R1=ctx(off=0,imm=0) R10=fp0
> > > > ; int xdp_dispatcher(struct xdp_md *ctx)
> > > > 0: (bf) r6 = r1                       ; R1=ctx(off=0,imm=0)
> > > > R6_w=ctx(off=0,imm=0)
> > > > 1: (b7) r0 = 2                        ; R0_w=2
> > > > ; __u8 num_progs_enabled = conf.num_progs_enabled;
> > > > 2: (18) r8 = 0xffffb2f6c06d8000       ; R8_w=map_value(off=0,ks=4,vs=84,imm=0)
> > > > 4: (71) r7 = *(u8 *)(r8 +0)           ; R7=1
> > > > R8=map_value(off=0,ks=4,vs=84,imm=0)
> > > > ; if (num_progs_enabled < 1)
> > > > 5: (15) if r7 == 0x0 goto pc+141      ; R7=1
> > > > ; ret = prog0(ctx);
> > > > 6: (bf) r1 = r6                       ; R1_w=ctx(off=0,imm=0)
> > > > R6=ctx(off=0,imm=0)
> > > > 7: (85) call pc+140
> > > > btf_vmlinux is malformed
> > > > R1 type=ctx expected=fp
> > > > Caller passes invalid args into func#1
> > > > processed 84 insns (limit 1000000) max_states_per_insn 0 total_states
> > > > 9 peak_states 9 mark_read 1
> > > > -- END PROG LOAD LOG --
> > > > libbpf: prog 'xdp_dispatcher': failed to load: -22
> > > > libbpf: failed to load object 'xdp-dispatcher.o'
> > > > libxdp: Failed to load dispatcher: Invalid argument
> > > > libxdp: Falling back to loading single prog without dispatcher
> > > >
> > > > Can this regression be fixed before kernel 6.2 ships ?
> >
> > I'm seeing the same failure on 32 bit ARM on v6.13.
> >
> > Have you found a solution?

> When I try the link from the discussion:
> https://github.com/tjcw/bpf-examples/tree/tjcw-explore-sameeth
> I get a 404 error from github.

I'm have the same error as Chris Ward wrote in their original mail. But
I'm using the xdp-tutorial's [1] basic01-xdp-pass/xdp_pass_user example.

[1] https://github.com/xdp-project/xdp-tutorial.git

This is my error message.

| sudo ./xdp_pass_user -d lan0
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: prog 'xdp_dispatcher': BPF program load failed: Invalid argument
| libbpf: prog 'xdp_dispatcher': -- BEGIN PROG LOAD LOG --
| btf_vmlinux is malformed
  ^^^^^^^^^^^^^^^^^^^^^^^^

Now I understand, what this error message wants to tell me. I should
recompile my kernel with CONFIG_DEBUG_INFO_BTF=y.

| 0: R1=ctx() R10=fp0
| ; int xdp_dispatcher(struct xdp_md *ctx) @ xdp-dispatcher.c:118
| 0: (bf) r6 = r1                       ; R1=ctx() R6_w=ctx()
| ; __u8 num_progs_enabled = conf.num_progs_enabled; @ xdp-dispatcher.c:120
| 1: (18) r8 = 0xc3b45cc8               ; R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
| 3: (71) r7 = *(u8 *)(r8 +2)           ; R7_w=1 R8_w=map_value(map=xdp_disp.rodata,ks=4,vs=124)
| 4: (b7) r0 = 2                        ; R0_w=2
| ; if (num_progs_enabled < 1) @ xdp-dispatcher.c:123
| 5: (15) if r7 == 0x0 goto pc+136      ; R7_w=1
| ; ret = prog0(ctx); @ xdp-dispatcher.c:125
| 6: (bf) r1 = r6                       ; R1_w=ctx() R6_w=ctx()
| 7: (85) call pc+135
| btf_vmlinux is malformed
| R1 type=ctx expected=fp
| Caller passes invalid args into func#1 ('prog0')
| processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
| -- END PROG LOAD LOG --
| libbpf: prog 'xdp_dispatcher': failed to load: -22
| libbpf: failed to load object 'xdp-dispatcher.o'
| libxdp: Failed to load dispatcher: Invalid argument
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:118) on device:lan0(ifindex:4)


With the CONFIG_DEBUG_INFO_BTF=y kernel the verifier seems to be more
happy. Now it fails with "-22":

| sudo ./xdp_pass_user -d lan0
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: prog 'xdp_pass': BPF program load failed: Invalid argument
| libbpf: prog 'xdp_pass': -- BEGIN PROG LOAD LOG --
| Extension programs should be JITed
| processed 0 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
| -- END PROG LOAD LOG --
| libbpf: prog 'xdp_pass': failed to load: -22
| libbpf: failed to load object 'xdp-dispatcher.o'
| libxdp: Compatibility check for dispatcher program failed: Invalid argument
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:20) on device:lan0(ifindex:4)


After unloading and enabling the JIT...

| ➜ (pts/0) frogger@...t:xdp-tutorial/basic01-xdp-pass (main✗) echo 1 |sudo tee /proc/sys/net/core/bpf_jit_enable                                   

... the dispatcher fails to load with "524". Yes, the number is
positive.

| ➜ (pts/0) frogger@...t:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0 --unload-all
| Success: Unloading XDP prog name: xdp_prog_simple
| ➜ (pts/0) frogger@...t:xdp-tutorial/basic01-xdp-pass (main✗) sudo ./xdp_pass_user -d lan0             
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| libxdp: Compatibility check for dispatcher program failed: Unknown error 524
| libxdp: Falling back to loading single prog without dispatcher
| libbpf: elf: skipping unrecognized data section(7) xdp_metadata
| Success: Loading XDP prog name:xdp_prog_simple(id:48) on device:lan0(ifindex:4)

strace indicates this syscalls fails:

| bpf(BPF_RAW_TRACEPOINT_OPEN, {raw_tracepoint={name=NULL, prog_fd=17}}, 16) = -1 ENOTSUPP (Unknown error 524)

I'm on a armv7l, i.e. a 32 bit ARM system. Maybe I'm missing some kernel
option or BPF_RAW_TRACEPOINT_OPEN is not supported on armv7l. Will look
deeper into the kernel config options tomorrow.

regards,
Marc

-- 
Pengutronix e.K.                 | Marc Kleine-Budde          |
Embedded Linux                   | https://www.pengutronix.de |
Vertretung Nürnberg              | Phone: +49-5121-206917-129 |
Amtsgericht Hildesheim, HRA 2686 | Fax:   +49-5121-206917-9   |

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists