Date:   Tue, 12 Apr 2022 15:59:41 -0700
From:   Andrii Nakryiko <andrii.nakryiko@...il.com>
To:     Jiri Olsa <olsajiri@...il.com>
Cc:     Jiri Olsa <jolsa@...nel.org>, Alexei Starovoitov <ast@...nel.org>,
        Daniel Borkmann <daniel@...earbox.net>,
        Andrii Nakryiko <andrii@...nel.org>,
        Masami Hiramatsu <mhiramat@...nel.org>,
        Networking <netdev@...r.kernel.org>, bpf <bpf@...r.kernel.org>,
        lkml <linux-kernel@...r.kernel.org>,
        Martin KaFai Lau <kafai@...com>,
        Song Liu <songliubraving@...com>, Yonghong Song <yhs@...com>,
        John Fastabend <john.fastabend@...il.com>,
        KP Singh <kpsingh@...omium.org>
Subject: Re: [RFC bpf-next 4/4] selftests/bpf: Add attach bench test

On Tue, Apr 12, 2022 at 8:44 AM Jiri Olsa <olsajiri@...il.com> wrote:
>
> On Mon, Apr 11, 2022 at 03:15:40PM -0700, Andrii Nakryiko wrote:
>
> SNIP
>
> > > +static int get_syms(char ***symsp, size_t *cntp)
> > > +{
> > > +       size_t cap = 0, cnt = 0, i;
> > > +       char *name, **syms = NULL;
> > > +       struct hashmap *map;
> > > +       char buf[256];
> > > +       FILE *f;
> > > +       int err;
> > > +
> > > +       /*
> > > +        * The available_filter_functions contains many duplicates,
> > > +        * but other than that all symbols are usable in kprobe multi
> > > +        * interface.
> > > +        * Filtering out duplicates by using hashmap__add, which won't
> > > +        * add existing entry.
> > > +        */
> > > +       f = fopen(DEBUGFS "available_filter_functions", "r");
> >
> > I'm really curious how did you manage to attach to everything in
> > available_filter_functions because when I'm trying to do that I fail.
>
> the new code makes the differece ;-) so the main problem I could not
> use available_filter_functions functions before were cases like:
>
>   # cat available_filter_functions | grep sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>   sys_ni_syscall
>
> which when you try to resolve you'll find just one address:
>
>   # cat /proc/kallsyms | egrep 'T sys_ni_syscall'
>   ffffffff81170020 T sys_ni_syscall
>
> this is caused by entries like:
>     __SYSCALL(156, sys_ni_syscall)
>
> when generating syscalls for given arch
>
> this is handled by the new code by removing duplicates when
> reading available_filter_functions
>
>
>
> another case is the other way round, like with:
>
>   # cat /proc/kallsyms | grep 't t_next'
>   ffffffff8125c3f0 t t_next
>   ffffffff8126a320 t t_next
>   ffffffff81275de0 t t_next
>   ffffffff8127efd0 t t_next
>   ffffffff814d6660 t t_next
>
> that has just one 'ftrace-able' instance:
>
>   # cat available_filter_functions | grep '^t_next$'
>   t_next
>
> and this is handled by calling ftrace_location on address when
> resolving symbols, to ensure each reasolved symbol lives in ftrace
>
> > available_filter_functions has a bunch of functions that should not be
> > attachable (e.g., notrace functions). Look just at __bpf_tramp_exit:
> >
> >   void notrace __bpf_tramp_exit(struct bpf_tramp_image *tr);
> >
> > So first, curious what I am doing wrong or rather why it succeeds in
> > your case ;)
> >
> > But second, just wanted to plea to "fix" available_filter_functions to
> > not list stuff that should not be attachable. Can you please take a
> > look and checks what's going on there and why do we have notrace
> > functions (and what else should *NOT* be there)?
>
> yes, seems like a bug ;-) it's in available_filter_functions
> but it does not have 'call __fentry__' at the entry..
>
> I was going to check on that, because you brought that up before,
> but did not get to it yet

yeah, see also my reply to Masami. __bpf_tramp_exit and
__bpf_tramp_enter are two specific examples. Both are marked notrace,
but one is in available_filter_functions and another is not. Neither
should be attachable, but doing this local change you can see that one
of them (__bpf_tramp_exit) is:

$ git diff
diff --git a/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
b/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
index b9876b55fc0c..77cff034d427 100644
--- a/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
+++ b/tools/testing/selftests/bpf/prog_tests/kprobe_multi_test.c
@@ -165,8 +165,8 @@ static void test_attach_api_pattern(void)
 {
        LIBBPF_OPTS(bpf_kprobe_multi_opts, opts);

-       test_attach_api("bpf_fentry_test*", &opts);
-       test_attach_api("bpf_fentry_test?", NULL);
+       test_attach_api("__bpf_tramp_enter", &opts);
+       test_attach_api("__bpf_tramp_exit", NULL);
 }


$ sudo ./test_progs -t kprobe_multi/attach_api_pattern -v
bpf_testmod.ko is already unloaded.
Loading bpf_testmod.ko...
Successfully loaded bpf_testmod.ko.
test_kprobe_multi_test:PASS:load_kallsyms 0 nsec
test_attach_api:PASS:fentry_raw_skel_load 0 nsec
libbpf: prog 'test_kprobe': failed to attach: Invalid argument
test_attach_api:FAIL:bpf_program__attach_kprobe_multi_opts unexpected error: -22
test_attach_api:PASS:fentry_raw_skel_load 0 nsec
test_attach_api:PASS:bpf_program__attach_kprobe_multi_opts 0 nsec

Quite weird.



>
> >
> >
> > > +       if (!f)
> > > +               return -EINVAL;
> > > +
> > > +       map = hashmap__new(symbol_hash, symbol_equal, NULL);
> > > +       err = libbpf_get_error(map);
> > > +       if (err)
> > > +               goto error;
> > > +
> >
> > [...]
> >
> > > +
> > > +       attach_delta_ns = (attach_end_ns - attach_start_ns) / 1000000000.0;
> > > +       detach_delta_ns = (detach_end_ns - detach_start_ns) / 1000000000.0;
> > > +
> > > +       fprintf(stderr, "%s: found %lu functions\n", __func__, cnt);
> > > +       fprintf(stderr, "%s: attached in %7.3lfs\n", __func__, attach_delta_ns);
> > > +       fprintf(stderr, "%s: detached in %7.3lfs\n", __func__, detach_delta_ns);
> > > +
> > > +       if (attach_delta_ns > 2.0)
> > > +               PRINT_FAIL("attach time above 2 seconds\n");
> > > +       if (detach_delta_ns > 2.0)
> > > +               PRINT_FAIL("detach time above 2 seconds\n");
> >
> > see my reply on the cover letter, any such "2 second" assumption are
> > guaranteed to bite us. We've dealt with a lot of timing issues due to
> > CI being slower and more unpredictable in terms of performance, I'd
> > like to avoid dealing with one more case like that.
>
> right, I'll remove the check
>
> thanks,
> jirka

Powered by blists - more mailing lists