linux-kernel - Re: BUG: unable to handle kernel paging request in bpf_dispatcher

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <Y5NSStSi7h9Vdo/j@krava>
Date:   Fri, 9 Dec 2022 16:20:42 +0100
From:   Jiri Olsa <olsajiri@...il.com>
To:     Jiri Olsa <olsajiri@...il.com>
Cc:     Alexei Starovoitov <alexei.starovoitov@...il.com>,
        Song Liu <song@...nel.org>, Hao Sun <sunhao.th@...il.com>,
        Peter Zijlstra <peterz@...radead.org>,
        bpf <bpf@...r.kernel.org>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        John Fastabend <john.fastabend@...il.com>,
        Andrii Nakryiko <andrii@...nel.org>,
        Martin KaFai Lau <martin.lau@...ux.dev>,
        Yonghong Song <yhs@...com>, KP Singh <kpsingh@...nel.org>,
        Stanislav Fomichev <sdf@...gle.com>,
        Hao Luo <haoluo@...gle.com>,
        David Miller <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        Jesper Dangaard Brouer <hawk@...nel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        netdev <netdev@...r.kernel.org>,
        Thorsten Leemhuis <regressions@...mhuis.info>
Subject: Re: BUG: unable to handle kernel paging request in bpf_dispatcher_xdp

On Fri, Dec 09, 2022 at 02:50:55PM +0100, Jiri Olsa wrote:
> On Fri, Dec 09, 2022 at 12:22:37PM +0100, Jiri Olsa wrote:
> 
> SBIP
> 
> > > > > > > >
> > > > > > > > I'm trying to understand the severity of the issues and
> > > > > > > > whether we need to revert that commit asap since the merge window
> > > > > > > > is about to start.
> > > > > > > 
> > > > > > > Jiri, Peter,
> > > > > > > 
> > > > > > > ping.
> > > > > > > 
> > > > > > > cc-ing Thorsten, since he's tracking it now.
> > > > > > > 
> > > > > > > The config has CONFIG_X86_KERNEL_IBT=y.
> > > > > > > Is it related?
> > > > > > 
> > > > > > sorry for late reply.. I still did not find the reason,
> > > > > > but I did not try with IBT yet, will test now
> > > > > 
> > > > > no difference with IBT enabled, can't reproduce the issue
> > > > > 
> > > > 
> > > > ok, scratch that.. the reproducer got stuck on wifi init :-\
> > > > 
> > > > after I fix that I can now reproduce on my local config with
> > > > IBT enabled or disabled.. it's something else
> > > 
> > > I'm getting the error also when reverting the static call change,
> > > looking for good commit, bisecting
> > > 
> > > I'm getting fail with:
> > >    f0c4d9fc9cc9 (tag: v6.1-rc4) Linux 6.1-rc4
> > > 
> > > v6.1-rc1 is ok
> > 
> > so far I narrowed it down between rc1 and rc3.. bisect got me nowhere so far
> > 
> > attaching some more logs
> 
> looking at the code.. how do we ensure that code running through
> bpf_prog_run_xdp will not get dispatcher image changed while
> it's being exetuted
> 
> we use 'the other half' of the image when we add/remove programs,
> but could bpf_dispatcher_update race with bpf_prog_run_xdp like:
> 
> 
> cpu 0:                                  cpu 1:
> 
> bpf_prog_run_xdp
>    ...
>    bpf_dispatcher_xdp_func
>       start exec image at offset 0x0
> 
>                                         bpf_dispatcher_update
>                                                 update image at offset 0x800
>                                         bpf_dispatcher_update
>                                                 update image at offset 0x0
> 
>       still in image at offset 0x0
> 
> 
> that might explain why I wasn't able to trigger that on
> bare metal just in qemu

I tried patch below and it fixes the issue for me and seems
to confirm the race above.. but not sure it's the best fix

jirka


---
diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c
index c19719f48ce0..6a2ced102fc7 100644
--- a/kernel/bpf/dispatcher.c
+++ b/kernel/bpf/dispatcher.c
@@ -124,6 +124,7 @@ static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs)
 	}
 
 	__BPF_DISPATCHER_UPDATE(d, new ?: (void *)&bpf_dispatcher_nop_func);
+	synchronize_rcu_tasks();
 
 	if (new)
 		d->image_off = noff;