lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Message-Id: <0B62D35A-E695-4B7A-A0D4-774767544C1A@gmail.com> Date: Tue, 13 Dec 2022 10:26:13 +0800 From: Hao Sun <sunhao.th@...il.com> To: Jiri Olsa <olsajiri@...il.com> Cc: Alexei Starovoitov <alexei.starovoitov@...il.com>, Jakub Kicinski <kuba@...nel.org>, "Paul E. McKenney" <paulmck@...nel.org>, Daniel Borkmann <daniel@...earbox.net>, Yonghong Song <yhs@...a.com>, Song Liu <song@...nel.org>, Peter Zijlstra <peterz@...radead.org>, bpf <bpf@...r.kernel.org>, Alexei Starovoitov <ast@...nel.org>, John Fastabend <john.fastabend@...il.com>, Andrii Nakryiko <andrii@...nel.org>, Martin KaFai Lau <martin.lau@...ux.dev>, Yonghong Song <yhs@...com>, KP Singh <kpsingh@...nel.org>, Stanislav Fomichev <sdf@...gle.com>, Hao Luo <haoluo@...gle.com>, David Miller <davem@...emloft.net>, Jesper Dangaard Brouer <hawk@...nel.org>, Linux Kernel Mailing List <linux-kernel@...r.kernel.org>, netdev <netdev@...r.kernel.org>, Thorsten Leemhuis <regressions@...mhuis.info> Subject: Re: BUG: unable to handle kernel paging request in bpf_dispatcher_xdp > On 12 Dec 2022, at 11:04 PM, Jiri Olsa <olsajiri@...il.com> wrote: > > On Sat, Dec 10, 2022 at 02:11:34PM +0100, Jiri Olsa wrote: >> On Fri, Dec 09, 2022 at 05:12:03PM -0800, Alexei Starovoitov wrote: >>> On Fri, Dec 9, 2022 at 4:06 PM Jiri Olsa <olsajiri@...il.com> wrote: >>>> >>>> On Fri, Dec 09, 2022 at 03:34:45PM -0800, Jakub Kicinski wrote: >>>>> On Sat, 10 Dec 2022 00:32:07 +0100 Daniel Borkmann wrote: >>>>>> fwiw, these should not be necessary, Documentation/RCU/checklist.rst : >>>>>> >>>>>> [...] One example of non-obvious pairing is the XDP feature in networking, >>>>>> which calls BPF programs from network-driver NAPI (softirq) context. BPF >>>>>> relies heavily on RCU protection for its data structures, but because the >>>>>> BPF program invocation happens entirely within a single local_bh_disable() >>>>>> section in a NAPI poll cycle, this usage is safe. The reason that this usage >>>>>> is safe is that readers can use anything that disables BH when updaters use >>>>>> call_rcu() or synchronize_rcu(). [...] >>>>> >>>>> FWIW I sent a link to the thread to Paul and he confirmed >>>>> the RCU will wait for just the BH. >>>> >>>> so IIUC we can omit the rcu_read_lock/unlock on bpf_prog_run_xdp side >>>> >>>> Paul, >>>> any thoughts on what we can use in here to synchronize bpf_dispatcher_change_prog >>>> with bpf_prog_run_xdp callers? >>>> >>>> with synchronize_rcu_tasks I'm getting splats like: >>>> https://lore.kernel.org/bpf/20221209153445.22182ca5@kernel.org/T/#m0a869f93404a2744884d922bc96d497ffe8f579f >>>> >>>> synchronize_rcu_tasks_rude seems to work (patch below), but it also sounds special ;-) >>> >>> Jiri, >>> >>> I haven't tried to repro this yet, but I feel you're on >>> the wrong path here. The splat has this: >>> ? bpf_prog_run_xdp include/linux/filter.h:775 [inline] >>> ? bpf_test_run+0x2ce/0x990 net/bpf/test_run.c:400 >>> that test_run logic takes rcu_read_lock. >>> See bpf_test_timer_enter. >>> I suspect the addition of synchronize_rcu_tasks_rude >>> only slows down the race. >>> The synchronize_rcu_tasks_trace also behaves like synchronize_rcu. >>> See our new and fancy rcu_trace_implies_rcu_gp(), >>> but I'm not sure it applies to synchronize_rcu_tasks_rude. >>> Have you tried with just synchronize_rcu() ? >>> If your theory about the race is correct then >>> the vanila sync_rcu should help. >>> If not, the issue is some place else. >> >> synchronize_rcu seems to work as well, I'll keep the test >> running for some time > > looks good, Hao Sun, could you please test change below? Hi, Tested on a latest bpf-next build. The reproducer would trigger the Oops in 5 mins without the patch. After applying the patch, the reproducer cannot trigger any issue for more than 15 mins. Seems working, tested on: HEAD commit: ef3911a3e4d6 docs/bpf: Reword docs for BPF_MAP_TYPE_SK_STORAGE git tree: bpf-next kernel config: https://pastebin.com/raw/rZdWLcgK C reproducer: https://pastebin.com/raw/GFfDn2Gk > > --- > diff --git a/kernel/bpf/dispatcher.c b/kernel/bpf/dispatcher.c > index c19719f48ce0..4b0fa5b98137 100644 > --- a/kernel/bpf/dispatcher.c > +++ b/kernel/bpf/dispatcher.c > @@ -124,6 +124,7 @@ static void bpf_dispatcher_update(struct bpf_dispatcher *d, int prev_num_progs) > } > > __BPF_DISPATCHER_UPDATE(d, new ?: (void *)&bpf_dispatcher_nop_func); > + synchronize_rcu(); > > if (new) > d->image_off = noff;
Powered by blists - more mailing lists