Date:   Thu, 7 Oct 2021 14:29:16 -0700
From:   Andrii Nakryiko <andrii.nakryiko@...il.com>
To:     Kumar Kartikeya Dwivedi <memxor@...il.com>
Cc:     Toke Høiland-Jørgensen <toke@...hat.com>,
        Alexei Starovoitov <alexei.starovoitov@...il.com>,
        bpf <bpf@...r.kernel.org>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Andrii Nakryiko <andrii@...nel.org>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>, Yonghong Song <yhs@...com>,
        Jesper Dangaard Brouer <brouer@...hat.com>,
        Networking <netdev@...r.kernel.org>
Subject: Re: [PATCH bpf-next v1 3/6] libbpf: Ensure that module BTF fd is
 never 0

On Thu, Oct 7, 2021 at 11:44 AM Kumar Kartikeya Dwivedi
<memxor@...il.com> wrote:
>
> On Thu, Oct 07, 2021 at 03:54:34PM IST, Toke Høiland-Jørgensen wrote:
> > Andrii Nakryiko <andrii.nakryiko@...il.com> writes:
> >
> > > On Wed, Oct 6, 2021 at 12:09 PM Alexei Starovoitov
> > > <alexei.starovoitov@...il.com> wrote:
> > >>
> > >> On Wed, Oct 6, 2021 at 9:43 AM Andrii Nakryiko
> > >> <andrii.nakryiko@...il.com> wrote:
> > >> >
> > >> > On Tue, Oct 5, 2021 at 10:24 PM Kumar Kartikeya Dwivedi
> > >> > <memxor@...il.com> wrote:
> > >> > >
> > >> > > On Wed, Oct 06, 2021 at 10:11:29AM IST, Andrii Nakryiko wrote:
> > >> > > > On Tue, Oct 5, 2021 at 5:29 PM Kumar Kartikeya Dwivedi <memxor@...il.com> wrote:
> > >> > > > >
> > >> > > > > Since the code assumes in various places that BTF fd for modules is
> > >> > > > > never 0, if we end up getting fd as 0, obtain a new fd > 0. Even though
> > >> > > > > fd 0 being free for allocation is usually an application error, it is
> > >> > > > > still possible that we end up getting fd 0 if the application explicitly
> > >> > > > > closes its stdin. Deal with this by getting a new fd using dup and
> > >> > > > > closing fd 0.
> > >> > > > >
> > >> > > > > Signed-off-by: Kumar Kartikeya Dwivedi <memxor@...il.com>
> > >> > > > > ---
> > >> > > > >  tools/lib/bpf/libbpf.c | 14 ++++++++++++++
> > >> > > > >  1 file changed, 14 insertions(+)
> > >> > > > >
> > >> > > > > diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> > >> > > > > index d286dec73b5f..3e5e460fe63e 100644
> > >> > > > > --- a/tools/lib/bpf/libbpf.c
> > >> > > > > +++ b/tools/lib/bpf/libbpf.c
> > >> > > > > @@ -4975,6 +4975,20 @@ static int load_module_btfs(struct bpf_object *obj)
> > >> > > > >                         pr_warn("failed to get BTF object #%d FD: %d\n", id, err);
> > >> > > > >                         return err;
> > >> > > > >                 }
> > >> > > > > +               /* Make sure module BTF fd is never 0, as kernel depends on it
> > >> > > > > +                * being > 0 to distinguish between vmlinux and module BTFs,
> > >> > > > > +                * e.g. for BPF_PSEUDO_BTF_ID ld_imm64 insns (ksyms).
> > >> > > > > +                */
> > >> > > > > +               if (!fd) {
> > >> > > > > +                       fd = dup(0);
> > >> > > >
> > >> > > > This is not the only place where we make assumptions that fd > 0 but
> > >> > > > technically can get fd == 0. Instead of doing such a check in every
> > >> > > > such place, would it be possible to open (cheaply) some FD (/dev/null
> > >> > > > or whatever, don't know what's the best file to open), if we detect
> > >> > > > that FD == 0 is not allocated? Can we detect that fd 0 is not
> > >> > > > allocated?
> > >> > > >
> > >> > >
> > >> > > We can, e.g. using access("/proc/self/fd/0", F_OK), but I think just calling
> > >> > > open unconditonally and doing if (ret > 0) close(ret) is better. Also, do I
> > >> >
> > >> > yeah, I like this idea, let's go with it
> > >>
> > >> FYI some production environments may detect that FDs 0,1,2 are not
> > >> pointing to stdin, stdout, stderr and will force close whatever files are there
> > >> and open 0,1,2 with canonical files.
> > >>
> > >> libbpf doesn't have to resort to such measures, but it would be prudent to
> > >> make libbpf operate on FDs > 2 for all bpf objects to make sure other
> > >> frameworks don't ruin libbpf's view of FDs.
> > >
> > > oh well, even without those production complications this would be a
> > > bit fragile, e.g., if the application temporarily opened FD 0 and then
> > > closed it.
> > >
> > > Ok, Kumar, can you please do it as a simple helper that would
> > > dup()'ing until we have FD>2, and use it in as few places as possible
> > > to make sure that all FDs (not just module BTF) are covered. I'd
> > > suggest doing that only in low-level helpers in btf.c, I think
> > > libbpf's logic always goes through those anyways (but please
> > > double-check that we don't call bpf syscall directly anywhere else).
> >
>
> Just to make sure I am on the same page:
>
> I have to...
> 1. Add a small wrapper (currently named fd_gt_2, any other suggestions?)

ensure_good_fd() or something? 2 is a tiny detail there.

>    that takes in the fd, and dups it to fd >= 3 if in range [0, 2] (and closes
>    original fd in this case).
>    Use this for all fd returning bpf syscalls in bpf.c (btf.c is a typo?).

yep, typo, I meant bpf.c

>    Audit other places directly calling syscall(__NR_bpf, ...).

yep

> 2. Assume that the situation Alexei mentioned only occurs at startup, or
>    sometime later, not in parallel (which would race with us, and not sure
>    we can deal with it). I'm thinking of a case where such an fd gets passed
>    to an exec'd binary which closes invalids fds on startup (so keeping them
>    >= 3 allows proper inheritance).

with checking it next to syscall(__NR_bpf) and fcntl suggestion from
Toke, why does it matter?

> 3. gen_loader can hit the same case, so short of adding a bpf_sys_fcntl (or the
>    helper only exposing F_DUPFD), next best option is to reserve the three fds from
>    skel_internal.h or gen_trace (in bpftool) and close later after loading is done.

Not sure, will defer to Alexei.

>
> Feedback needed on 3 (and whether a generic bpf_sys_dup providing functionality of
> existing fcntl and dup{,2,3} is better than simply reserving the three fds at
> load time).
>
> > FYI, you can use fcntl() with F_DUPFD{,_CLOEXEC} and tell it the minimum
> > fd number you're interested in for the clone. We do that in libxdp to
> > protect against fd 0:
> >
>
> Thanks, will switch the dup to fcntl in the next version.
>
> > https://github.com/xdp-project/xdp-tools/blob/master/lib/libxdp/libxdp.c#L1184
> >
> > Given Alexei's comments above, maybe we should be '3' for the last arg
> > instead of 1...
> >
> > -Toke
> >
>
> --
> Kartikeya

Powered by blists - more mailing lists