netdev - Re: [Patch net] lwt: disable BH too in run_lwt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAADnVQ+tRAKn4KR_k9eU-fG3iQhivzwn6d2BDGnX_44MTBrkJg@mail.gmail.com>
Date:   Thu, 3 Dec 2020 10:22:07 -0800
From:   Alexei Starovoitov <alexei.starovoitov@...il.com>
To:     Cong Wang <xiyou.wangcong@...il.com>
Cc:     Jakub Kicinski <kuba@...nel.org>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>,
        Dongdong Wang <wangdongdong@...edance.com>,
        Thomas Graf <tgraf@...g.ch>, bpf <bpf@...r.kernel.org>,
        Cong Wang <cong.wang@...edance.com>
Subject: Re: [Patch net] lwt: disable BH too in run_lwt_bpf()

On Wed, Dec 2, 2020 at 5:50 PM Alexei Starovoitov
<alexei.starovoitov@...il.com> wrote:
>
> On Wed, Dec 2, 2020 at 5:32 PM Cong Wang <xiyou.wangcong@...il.com> wrote:
> >
> > On Wed, Dec 2, 2020 at 5:10 PM Jakub Kicinski <kuba@...nel.org> wrote:
> > >
> > > On Tue,  1 Dec 2020 11:44:38 -0800 Cong Wang wrote:
> > > > From: Dongdong Wang <wangdongdong@...edance.com>
> > > >
> > > > The per-cpu bpf_redirect_info is shared among all skb_do_redirect()
> > > > and BPF redirect helpers. Callers on RX path are all in BH context,
> > > > disabling preemption is not sufficient to prevent BH interruption.
> > > >
> > > > In production, we observed strange packet drops because of the race
> > > > condition between LWT xmit and TC ingress, and we verified this issue
> > > > is fixed after we disable BH.
> > > >
> > > > Although this bug was technically introduced from the beginning, that
> > > > is commit 3a0af8fd61f9 ("bpf: BPF for lightweight tunnel infrastructure"),
> > > > at that time call_rcu() had to be call_rcu_bh() to match the RCU context.
> > > > So this patch may not work well before RCU flavor consolidation has been
> > > > completed around v5.0.
> > > >
> > > > Update the comments above the code too, as call_rcu() is now BH friendly.
> > > >
> > > > Cc: Thomas Graf <tgraf@...g.ch>
> > > > Cc: bpf@...r.kernel.org
> > > > Reviewed-by: Cong Wang <cong.wang@...edance.com>
> > > > Signed-off-by: Dongdong Wang <wangdongdong@...edance.com>
> > > > ---
> > > >  net/core/lwt_bpf.c | 8 ++++----
> > > >  1 file changed, 4 insertions(+), 4 deletions(-)
> > > >
> > > > diff --git a/net/core/lwt_bpf.c b/net/core/lwt_bpf.c
> > > > index 7d3438215f32..4f3cb7c15ddf 100644
> > > > --- a/net/core/lwt_bpf.c
> > > > +++ b/net/core/lwt_bpf.c
> > > > @@ -39,12 +39,11 @@ static int run_lwt_bpf(struct sk_buff *skb, struct bpf_lwt_prog *lwt,
> > > >  {
> > > >       int ret;
> > > >
> > > > -     /* Preempt disable is needed to protect per-cpu redirect_info between
> > > > -      * BPF prog and skb_do_redirect(). The call_rcu in bpf_prog_put() and
> > > > -      * access to maps strictly require a rcu_read_lock() for protection,
> > > > -      * mixing with BH RCU lock doesn't work.
> > > > +     /* Preempt disable and BH disable are needed to protect per-cpu
> > > > +      * redirect_info between BPF prog and skb_do_redirect().
> > > >        */
> > > >       preempt_disable();
> > > > +     local_bh_disable();
> > >
> > > Why not remove the preempt_disable()? Disabling BH must also disable
> > > preemption AFAIK.
> >
> > It seems RT kernel still needs preempt disable:
>
> No. It's the opposite. When we did RT+bpf changes we missed this function.
> It should be migrate_disable here instead of preempt_disable.
> I don't know what local_bh_disable() maps to in RT.
> Since it's used in many other places it's fine to use it here to
> prevent this race.

I guess my previous comment could be misinterpreted.
Cong,
please respin with changing preempt_disable to migrate_disable
and adding local_bh_disable.