netdev - Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs using the BPF_PROG_TEST

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAADnVQKyfJPujoef6+sV7hJf9kVBjZKur_yjW8GJtTYS-c_Knw@mail.gmail.com>
Date:   Tue, 5 May 2020 18:25:38 -0700
From:   Alexei Starovoitov <alexei.starovoitov@...il.com>
To:     Masami Hiramatsu <mhiramat@...nel.org>
Cc:     Arnaldo Carvalho de Melo <acme@...nel.org>,
        Eelco Chaudron <echaudro@...hat.com>,
        Yonghong Song <yhs@...com>, bpf <bpf@...r.kernel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Network Development <netdev@...r.kernel.org>,
        Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>,
        Andrii Nakryiko <andriin@...com>
Subject: Re: [RFC PATCH bpf-next 0/3] bpf: add tracing for XDP programs using
 the BPF_PROG_TEST_RUN API

On Thu, Apr 30, 2020 at 7:44 PM Masami Hiramatsu <mhiramat@...nel.org> wrote:
>
> On Tue, 28 Apr 2020 09:19:47 -0300
> Arnaldo Carvalho de Melo <acme@...nel.org> wrote:
>
> > Em Tue, Apr 28, 2020 at 12:47:53PM +0200, Eelco Chaudron escreveu:
> > > On 28 Apr 2020, at 6:04, Alexei Starovoitov wrote:
> > > > On Fri, Apr 24, 2020 at 02:29:56PM +0200, Eelco Chaudron wrote:
> >
> > > > > > But in reality I think few kprobes in the prog will be enough to
> > > > > > debug the program and XDP prog may still process millions of
> > > > > > packets because your kprobe could be in error path and the user
> > > > > > may want to capture only specific things when it triggers.
> >
> > > > > > kprobe bpf prog will execute in such case and it can capture
> > > > > > necessary state from xdp prog, from packet or from maps that xdp
> > > > > > prog is using.
> >
> > > > > > Some sort of bpf-gdb would be needed in user space.  Obviously
> > > > > > people shouldn't be writing such kprob-bpf progs that debug
> > > > > > other bpf progs by hand. bpf-gdb should be able to generate them
> > > > > > automatically.
> >
> > > > > See my opening comment. What you're describing here is more when
> > > > > the right developer has access to the specific system. But this
> > > > > might not even be possible in some environments.
> >
> > > > All I'm saying that kprobe is a way to trace kernel.
> > > > The same facility should be used to trace bpf progs.
> >
> > > perf doesn’t support tracing bpf programs, do you know of any tools that
> > > can, or you have any examples that would do this?
> >
> > I'm discussing with Yonghong and Masami what would be needed for 'perf
> > probe' to be able to add kprobes to BPF jitted areas in addition to
> > vmlinux and modules.
>
> At a grance, at first we need a debuginfo which maps the source code and
> BPF binaries. We also need to get a map from the kernel indicating
> which instructions the bpf code was jited to.
> Are there any such information?

it's already there. Try 'bpftool prog dump jited id N'
It will show something like this:
; data = ({typeof(errors.leaf) *leaf =
bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &type_key); if (!leaf) {
bpf_map_update_elem_(bpf_pseudo_fd(1, -11), &type_key, &zero,
BPF_NOEXIST); leaf = bpf_map_lookup_elem_(bpf_pseudo_fd(1, -11), &t;
 81d:    movabs $0xffff8881a0679000,%rdi
; return bpf_map_lookup_elem((void *)map, key);
 827:    mov    %rbx,%rsi
 82a:    callq  0xffffffffe0f7f448
 82f:    test   %rax,%rax
 832:    je     0x0000000000000838
 834:    add    $0x40,%rax
; if (!data)
 838:    test   %rax,%rax
 83b:    je     0x0000000000000846
 83d:    mov    $0x1,%edi
; lock_xadd(data, 1);
 842:    lock add %edi,0x0(%rax)

> Also, I would like to know the target BPF (XDP) is running in kprobes
> context or not. BPF tracer sometimes use the kprobes to hook the event
> and run in the kprobe (INT3) context. That will be need more work to
> probe it.
> For the BPF code which just runs in tracepoint context, it will be easy
> to probe it. (we may need to break a limitation of notrace, which we
> already has a kconfig)

yeah. this mechanism won't be able to debug bpf progs that are
attached to kprobes via int3. But that is rare case.
Most kprobe+bpf are for function entry and adding int3 to jited bpf code
will work just like for normal kernel functions.