linux-kernel - Re: [PATCH] tracing/user_events: Add eBPF interface for user

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 30 Mar 2022 15:57:26 -0400 (EDT)
From:   Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To:     Beau Belgrave <beaub@...ux.microsoft.com>
Cc:     Alexei Starovoitov <alexei.starovoitov@...il.com>,
        Song Liu <song@...nel.org>, rostedt <rostedt@...dmis.org>,
        Masami Hiramatsu <mhiramat@...nel.org>,
        linux-trace-devel <linux-trace-devel@...r.kernel.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        bpf <bpf@...r.kernel.org>, netdev <netdev@...r.kernel.org>,
        linux-arch <linux-arch@...r.kernel.org>
Subject: Re: [PATCH] tracing/user_events: Add eBPF interface for user_event
 created events

----- On Mar 30, 2022, at 3:15 PM, Beau Belgrave beaub@...ux.microsoft.com wrote:

> On Wed, Mar 30, 2022 at 11:22:32AM -0700, Alexei Starovoitov wrote:
>> On Wed, Mar 30, 2022 at 9:34 AM Beau Belgrave <beaub@...ux.microsoft.com> wrote:
>> > > >
>> > > > But you are fine with uprobe costs? uprobes appear to be much more costly
>> > > > than a syscall approach on the hardware I've run on.
>> 
>> Care to share the numbers?
>> uprobe over USDT is a single trap.
>> Not much slower compared to syscall with kpti.
>> 
> 
> Sure, these are the numbers we have from a production device.
> 
> They are captured via perf via PERF_COUNT_HW_CPU_CYCLES.
> It's running a 20K loop emitting 4 bytes of data out.
> Each 4 byte event time is recorded via perf.
> At the end we have the total time and the max seen.
> 
> null numbers represent a 20K loop with just perf start/stop ioctl costs.
> 
> null: min=2863, avg=2953, max=30815
> uprobe: min=10994, avg=11376, max=146682
> uevent: min=7043, avg=7320, max=95396
> lttng: min=6270, avg=6508, max=41951
> 
> These costs include the data getting into a buffer, so they represent
> what we would see in production vs the trap cost alone. For uprobe this
> means we created a uprobe and attached it via tracefs to get the above
> numbers.

[...]

I assume here that by "lttng" you specifically refer to lttng-ust (LTTng's
user-space tracer), am I correct ?

By removing the "null" baseline overhead, my rough calculations are that the
average overhead for lttng-ust in your results is (in cpu cycles):

6270-2863 = 3555

So I'm unsure what is the frequency of your CPU, but guessing around 3.5GHz
this is in the area of 1 microsecond. On an Intel CPU, this is much larger
than what I would expect. 

Can you share your test program, hardware characteristics, kernel version,
glibc version, and whether the program is compiled as a 32-bit or 64-bit
binary ?

Can you confirm that lttng-ust is not calling one getcpu system call per
event ? This might be the case if run a 32-bit x86 binary and have a
glibc < 2.35, or a kernel too old to provide CONFIG_RSEQ or don't have
CONFIG_RSEQ=y in your kernel configuration. You can validate this by
running your lttng-ust test program with a system call tracer.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com