Date:   Wed, 13 Apr 2022 16:56:12 -0700
From:   Martin KaFai Lau <kafai@...com>
To:     sdf@...gle.com
Cc:     Andrii Nakryiko <andrii.nakryiko@...il.com>,
        Networking <netdev@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Andrii Nakryiko <andrii@...nel.org>
Subject: Re: [PATCH bpf-next] bpf: move rcu lock management out of
 BPF_PROG_RUN routines

On Wed, Apr 13, 2022 at 03:52:18PM -0700, sdf@...gle.com wrote:
> On 04/13, Martin KaFai Lau wrote:
> > On Wed, Apr 13, 2022 at 12:52:53PM -0700, Andrii Nakryiko wrote:
> > > On Wed, Apr 13, 2022 at 12:39 PM <sdf@...gle.com> wrote:
> > > >
> > > > On 04/13, Andrii Nakryiko wrote:
> > > > > On Wed, Apr 13, 2022 at 11:33 AM Stanislav Fomichev <sdf@...gle.com>
> > > > > wrote:
> > > > > >
> > > > > > Commit 7d08c2c91171 ("bpf: Refactor BPF_PROG_RUN_ARRAY family of
> > macros
> > > > > > into functions") switched a bunch of BPF_PROG_RUN macros to inline
> > > > > > routines. This changed the semantic a bit. Due to arguments
> > expansion
> > > > > > of macros, it used to be:
> > > > > >
> > > > > >         rcu_read_lock();
> > > > > >         array = rcu_dereference(cgrp->bpf.effective[atype]);
> > > > > >         ...
> > > > > >
> > > > > > Now, with with inline routines, we have:
> > > > > >         array_rcu = rcu_dereference(cgrp->bpf.effective[atype]);
> > > > > >         /* array_rcu can be kfree'd here */
> > > > > >         rcu_read_lock();
> > > > > >         array = rcu_dereference(array_rcu);
> > > > > >
> > > >
> > > > > So subtle difference, wow...
> > > >
> > > > > But this open-coding of rcu_read_lock() seems very unfortunate as
> > > > > well. Would making BPF_PROG_RUN_ARRAY back to a macro which only
> > does
> > > > > rcu lock/unlock and grabs effective array and then calls static
> > inline
> > > > > function be a viable solution?
> > > >
> > > > > #define BPF_PROG_RUN_ARRAY_CG_FLAGS(array_rcu, ctx, run_prog,
> > ret_flags) \
> > > > >    ({
> > > > >        int ret;
> > > >
> > > > >        rcu_read_lock();
> > > > >        ret =
> > > > > __BPF_PROG_RUN_ARRAY_CG_FLAGS(rcu_dereference(array_rcu), ....);
> > > > >        rcu_read_unlock();
> > > > >        ret;
> > > > >    })
> > > >
> > > >
> > > > > where __BPF_PROG_RUN_ARRAY_CG_FLAGS is what
> > > > > BPF_PROG_RUN_ARRAY_CG_FLAGS is today but with __rcu annotation
> > dropped
> > > > > (and no internal rcu stuff)?
> > > >
> > > > Yeah, that should work. But why do you think it's better to hide them?
> > > > I find those automatic rcu locks deep in the call stack a bit obscure
> > > > (when reasoning about sleepable vs non-sleepable contexts/bpf).
> > > >
> > > > I, as the caller, know that the effective array is rcu-managed (it
> > > > has __rcu annotation) and it seems natural for me to grab rcu lock
> > > > while work with it; I might grab it for some other things like cgroup
> > > > anyway.
> > >
> > > If you think that having this more explicitly is better, I'm fine with
> > > that as well. I thought a simpler invocation pattern would be good,
> > > given we call bpf_prog_run_array variants in quite a lot of places. So
> > > count me indifferent. I'm curious what others think.
> 
> > Would it work if the bpf_prog_run_array_cg() directly takes the
> > 'struct cgroup *cgrp' argument instead of the array ?
> > bpf_prog_run_array_cg() should know what protection is needed
> > to get member from the cgrp ptr.  The sk call path should be able
> > to provide a cgrp ptr.  For current cgrp, pass NULL as the cgrp
> > pointer and then current will be used in bpf_prog_run_array_cg().
> > A rcu_read_lock() is needed anyway to get the current's cgrp
> > and can be done together in bpf_prog_run_array_cg().
> 
> > That there are only two remaining bpf_prog_run_array() usages
> > from lirc and bpf_trace which are not too bad to have them
> > directly do rcu_read_lock on their own struct ?
> 
> From Andrii's original commit message:
> 
>     I think BPF_PROG_RUN_ARRAY_CG would benefit from further refactoring to
> accept
>     struct cgroup and enum bpf_attach_type instead of bpf_prog_array,
> fetching
>     cgrp->bpf.effective[type] and RCU-dereferencing it internally. But that
>     required including include/linux/cgroup-defs.h, which I wasn't sure is
> ok with
>     everyone.
> 
> I guess including cgroup-defs.h/bpf-cgroup-defs.h into bpf.h might still
> be somewhat problematic?
> 
> But even if we pass the cgroup pointer, I'm assuming that this cgroup
> pointer
> is still rcu-managed, right? So the callers still have to rcu-lock.
> However, in most places we don't care and do "cgrp =
> sock_cgroup_ptr(&sk->sk_cgrp_data);"
> but seems like it depends on the fact that sockets can't (yet?)
> change their cgroup association and it's fine to not rcu-lock that
> cgroup. Seems fragile, but ok.
There is no __rcu tag in struct sock_cgroup_data, so presumably it
won't change or a lock is needed ?  seems to be the former.

> It always stumbles me when I see:
> 
> cgrp = sock_cgroup_ptr(&sk->sk_cgrp_data);
> bpf_prog_run_array_cg_flags(cgrp.bpf->effective[atype], ...)
> 
> But then, with current, it becomes:
> 
> rcu_read_lock();
> cgrp = task_dfl_cgroup(current);
> bpf_prog_run_array_cg_flags(cgrp.bpf->effective[atype], ...)
> rcu_read_unlock();
> 
> Idk, I might be overthinking it. I'll try to see if including
> bpf-cgroup-defs.h and passing cgroup_bpf is workable.
yeah, passing cgroup_bpf and bpf-cgroup-defs.h is a better option.

Powered by blists - more mailing lists