netdev - Re: [PATCH v2 2/2] x86/cfi,bpf: Fix BPF JIT call

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20231204183354.GC7299@noisy.programming.kicks-ass.net>
Date: Mon, 4 Dec 2023 19:33:54 +0100
From: Peter Zijlstra <peterz@...radead.org>
To: Jiri Olsa <olsajiri@...il.com>
Cc: Alexei Starovoitov <alexei.starovoitov@...il.com>,
	Song Liu <song@...nel.org>, Song Liu <songliubraving@...a.com>,
	Paul Walmsley <paul.walmsley@...ive.com>,
	Palmer Dabbelt <palmer@...belt.com>,
	Albert Ou <aou@...s.berkeley.edu>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
	Dave Hansen <dave.hansen@...ux.intel.com>, X86 ML <x86@...nel.org>,
	"H. Peter Anvin" <hpa@...or.com>,
	"David S. Miller" <davem@...emloft.net>,
	David Ahern <dsahern@...nel.org>,
	Alexei Starovoitov <ast@...nel.org>,
	Daniel Borkmann <daniel@...earbox.net>,
	Andrii Nakryiko <andrii@...nel.org>,
	Martin KaFai Lau <martin.lau@...ux.dev>,
	Yonghong Song <yonghong.song@...ux.dev>,
	John Fastabend <john.fastabend@...il.com>,
	KP Singh <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...gle.com>,
	Hao Luo <haoluo@...gle.com>, Arnd Bergmann <arnd@...db.de>,
	Sami Tolvanen <samitolvanen@...gle.com>,
	Kees Cook <keescook@...omium.org>,
	Nathan Chancellor <nathan@...nel.org>,
	Nick Desaulniers <ndesaulniers@...gle.com>,
	linux-riscv <linux-riscv@...ts.infradead.org>,
	LKML <linux-kernel@...r.kernel.org>,
	Network Development <netdev@...r.kernel.org>,
	bpf <bpf@...r.kernel.org>, linux-arch <linux-arch@...r.kernel.org>,
	clang-built-linux <llvm@...ts.linux.dev>,
	Josh Poimboeuf <jpoimboe@...nel.org>,
	Joao Moreira <joao@...rdrivepizza.com>,
	Mark Rutland <mark.rutland@....com>
Subject: Re: [PATCH v2 2/2] x86/cfi,bpf: Fix BPF JIT call

On Mon, Dec 04, 2023 at 07:16:14PM +0100, Peter Zijlstra wrote:
> On Mon, Dec 04, 2023 at 06:25:34PM +0100, Jiri Olsa wrote:
> 
> > that boots properly for me but gives crash below when running bpf tests
> 
> OK, more funnies..
> 
> > [  482.145182][  T699] RIP: 0010:bpf_for_each_array_elem+0xbb/0x120
> > [  482.145672][  T699] Code: 4c 01 f5 89 5c 24 04 4c 89 e7 48 8d 74 24 04 48 89 ea 4c 89 fd 4c 89 f9 45 31 c0 4d 89 eb 41 ba ef 86 cd 67 45 03 53 f1 74 02 <0f> 0b 41 ff d3 0f 1f 00 48 85 c0 75 0e 48 8d 43 01 41 8b 4c 24 24
> > [  482.147221][  T699] RSP: 0018:ffffc900017e3e88 EFLAGS: 00010217
> > [  482.147702][  T699] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffffc900017e3ed8
> > [  482.152162][  T699] RDX: ffff888152eb0210 RSI: ffffc900017e3e8c RDI: ffff888152eb0000
> > [  482.152770][  T699] RBP: ffffc900017e3ed8 R08: 0000000000000000 R09: 0000000000000000
> > [  482.153350][  T699] R10: 000000004704ef28 R11: ffffffffa0012774 R12: ffff888152eb0000
> > [  482.153951][  T699] R13: ffffffffa0012774 R14: ffff888152eb0210 R15: ffffc900017e3ed8
> > [  482.154554][  T699] FS:  00007fa60d4fdd00(0000) GS:ffff88846d200000(0000) knlGS:0000000000000000
> > [  482.155138][  T699] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  482.155564][  T699] CR2: 00007fa60d7d8000 CR3: 00000001502a2005 CR4: 0000000000770ef0
> > [  482.156095][  T699] PKRU: 55555554
> > [  482.156349][  T699] Call Trace:
> > [  482.156596][  T699]  <TASK>
> > [  482.156816][  T699]  ? __die_body+0x68/0xb0
> > [  482.157138][  T699]  ? die+0xba/0xe0
> > [  482.157456][  T699]  ? do_trap+0xa5/0x180
> > [  482.157826][  T699]  ? bpf_for_each_array_elem+0xbb/0x120
> > [  482.158277][  T699]  ? bpf_for_each_array_elem+0xbb/0x120
> > [  482.158711][  T699]  ? do_error_trap+0xc4/0x140
> > [  482.159052][  T699]  ? bpf_for_each_array_elem+0xbb/0x120
> > [  482.159506][  T699]  ? handle_invalid_op+0x2c/0x40
> > [  482.159906][  T699]  ? bpf_for_each_array_elem+0xbb/0x120
> > [  482.160990][  T699]  ? exc_invalid_op+0x38/0x60
> > [  482.161375][  T699]  ? asm_exc_invalid_op+0x1a/0x20
> > [  482.161788][  T699]  ? 0xffffffffa0012774
> > [  482.162149][  T699]  ? 0xffffffffa0012774
> > [  482.162513][  T699]  ? bpf_for_each_array_elem+0xbb/0x120
> > [  482.162905][  T699]  bpf_prog_ca45ea7f9cb8ac1a_inner_map+0x94/0x98
> > [  482.163471][  T699]  bpf_trampoline_6442549234+0x47/0x1000
> 
> Looks like this trips an #UD, I'll go try and figure out what this
> bpf_for_each_array_elem() does to cause this. Looks like it has an
> indirect call, could be the callback_fn thing has a CFI mis-match.

So afaict this is used through bpf_for_each_map_elem(), where the
argument still is properly callback_fn. However, in the desriptor
bpf_for_each_map_elem_proto the argument gets described as:
ARG_PTR_TO_FUNC, which in turn has a comment like:

  ARG_PTR_TO_FUNC,        /* pointer to a bpf program function */

Which to me sounds like there is definite type punning involved. The
call in bpf_for_each_array_elem() is a regular C indirect call, which
gets adorned with the kCFI magic.

But I doubt the BPF function that gets used gets the correct matching
bits on.

TL;DR, I think this is a pre-existing problem with kCFI + eBPF and not
caused by my patches.

Could any of you bpf knowledgeable folks please explain me exactly what
gets used as the function pointer in this case? -- I'm not sure I can
follow along well enough to begin looking for a solution at this point
:/