netdev - Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs using the BPF_PROG_TEST

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-Id: <20200507175512.b2a01b872750dbfe94f6642b@kernel.org>
Date:   Thu, 7 May 2020 17:55:12 +0900
From:   Masami Hiramatsu <mhiramat@...nel.org>
To:     Alexei Starovoitov <alexei.starovoitov@...il.com>
Cc:     Arnaldo Carvalho de Melo <acme@...nel.org>,
        Eelco Chaudron <echaudro@...hat.com>,
        Yonghong Song <yhs@...com>, bpf <bpf@...r.kernel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Network Development <netdev@...r.kernel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>,
        Andrii Nakryiko <andriin@...com>
Subject: Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs
 using the BPF_PROG_TEST_RUN API

On Tue, 5 May 2020 18:25:38 -0700
Alexei Starovoitov <alexei.starovoitov@...il.com> wrote:

> On Thu, Apr 30, 2020 at 7:44 PM Masami Hiramatsu <mhiramat@...nel.org> wrote:
> >
> > On Tue, 28 Apr 2020 09:19:47 -0300
> > Arnaldo Carvalho de Melo <acme@...nel.org> wrote:
> >
> > > Em Tue, Apr 28, 2020 at 12:47:53PM +0200, Eelco Chaudron escreveu:
> > > > On 28 Apr 2020, at 6:04, Alexei Starovoitov wrote:
> > > > > On Fri, Apr 24, 2020 at 02:29:56PM +0200, Eelco Chaudron wrote:
> > >
> > > > > > > But in reality I think few kprobes in the prog will be enough to
> > > > > > > debug the program and XDP prog may still process millions of
> > > > > > > packets because your kprobe could be in error path and the user
> > > > > > > may want to capture only specific things when it triggers.
> > >
> > > > > > > kprobe bpf prog will execute in such case and it can capture
> > > > > > > necessary state from xdp prog, from packet or from maps that xdp
> > > > > > > prog is using.
> > >
> > > > > > > Some sort of bpf-gdb would be needed in user space.  Obviously
> > > > > > > people shouldn't be writing such kprob-bpf progs that debug
> > > > > > > other bpf progs by hand. bpf-gdb should be able to generate them
> > > > > > > automatically.
> > >
> > > > > > See my opening comment. What you're describing here is more when
> > > > > > the right developer has access to the specific system. But this
> > > > > > might not even be possible in some environments.
> > >
> > > > > All I'm saying that kprobe is a way to trace kernel.
> > > > > The same facility should be used to trace bpf progs.
> > >
> > > > perf doesn’t support tracing bpf programs, do you know of any tools that
> > > > can, or you have any examples that would do this?
> > >
> > > I'm discussing with Yonghong and Masami what would be needed for 'perf
> > > probe' to be able to add kprobes to BPF jitted areas in addition to
> > > vmlinux and modules.
> >
> > At a grance, at first we need a debuginfo which maps the source code and
> > BPF binaries. We also need to get a map from the kernel indicating
> > which instructions the bpf code was jited to.
> > Are there any such information?
> 
> it's already there. Try 'bpftool prog dump jited id N'
> It will show something like this:
> ; data = ({typeof(errors.leaf) *leaf =
> bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &type_key); if (!leaf) {
> bpf_map_update_elem_(bpf_pseudo_fd(1, -11), &type_key, &zero,
> BPF_NOEXIST); leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &t;
>  81d:    movabs $0xffff8881a0679000,%rdi
> ; return bpf_map_lookup_elem((void *)map, key);
>  827:    mov    %rbx,%rsi
>  82a:    callq  0xffffffffe0f7f448
>  82f:    test   %rax,%rax
>  832:    je     0x0000000000000838
>  834:    add    $0x40,%rax
> ; if (!data)
>  838:    test   %rax,%rax
>  83b:    je     0x0000000000000846
>  83d:    mov    $0x1,%edi
> ; lock_xadd(data, 1);
>  842:    lock add %edi,0x0(%rax)

Hm, so bpftool or libbpf has some source-address (offset) mapping
APIs, that will help for me.

BTW, I would like to confirm that if the kernel has jited code,
it will not fall back to xlated code. Is it correct?

> > Also, I would like to know the target BPF (XDP) is running in kprobes
> > context or not. BPF tracer sometimes use the kprobes to hook the event
> > and run in the kprobe (INT3) context. That will be need more work to
> > probe it.
> > For the BPF code which just runs in tracepoint context, it will be easy
> > to probe it. (we may need to break a limitation of notrace, which we
> > already has a kconfig)
> 
> yeah. this mechanism won't be able to debug bpf progs that are
> attached to kprobes via int3. But that is rare case.
> Most kprobe+bpf are for function entry and adding int3 to jited bpf code
> will work just like for normal kernel functions.

OK, anyway, I've done the nested kprobe support on x86/arm/arm64 :)
That will make it easy.

Thank you,

-- 
Masami Hiramatsu <mhiramat@...nel.org>