linux-kernel - Re: Question about the ipi_raise filter usage and output

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <xhsmhjznigcdr.mognet@vschneid-thinkpadt14sgen2i.remote.csb>
Date: Tue, 06 Feb 2024 10:39:28 +0100
From: Valentin Schneider <vschneid@...hat.com>
To: richard clark <richard.xnu.clark@...il.com>
Cc: Mark Rutland <mark.rutland@....com>, Steven Rostedt
 <rostedt@...dmis.org>, nico@...xnic.net, mhiramat@...nel.org,
 linux-arm-kernel@...ts.infradead.org, linux-trace-kernel@...r.kernel.org,
 linux-kernel@...r.kernel.org
Subject: Re: Question about the ipi_raise filter usage and output

On 06/02/24 16:42, richard clark wrote:
> On Tue, Feb 6, 2024 at 12:05 AM Valentin Schneider <vschneid@...hat.com> wrote:
>>
>> The CPUS{} thingie only works with an event field that is either declared as a
>> cpumask (__cpumask) or a scalar. That's not the case for ipi_raise, the
>> target_cpus event field is saved as a "raw" bitmask.
>>
>> There /should/ have been a warning about the event filter though, but I
>> think it's not happening because I'm allowing more than just FILTER_CPUMASK
>> in parse_pred() to make it work for scalars. I'll go poke around some more.
>>
>> Generally for this sort of IPI investigation I'd recommend using the newer
>> trace_ipi_send_cpu() and trace_ipi_send_cpumask() (for which CPUS{}
>> filtering works).
>> If it's only the function call interrupts you're interesting in, have a
>> look at trace_csd_queue_cpu().
>
> This should be supported by newer version kernels like v6.5, but I am
> using v6.1 and this trace event has not been supported yet... so ipi
> is more suitable for me. ipi_entry and ipi_exit is ok, but seems the
> filter doesn't support a specific cpu, maybe we need to add this?
>>

You should have access to the generic fields which include the CPU from
which the event happens. Any of "CPU", "cpu" or "common_cpu" would match
this.

So if you're on a recent enough kernel (v6.6 or above AFAICT), you should
be able to do something like so:

  trace-cmd record -e 'ipi_raise' -f 'CPU & CPUS{7-42}' ./foo.sh

If you just want to match a single CPU, or are on an older kernel, this
should work as well:

  trace-cmd record -e 'ipi_raise' -f 'CPU == 42' ./foo.sh

For example on a QEMU x86 environment:

  # trace-cmd record -e 'call_function*' -f 'CPU & CPUS{3}' hackbench
  Running in process mode with 10 groups using 40 file descriptors each (== 400 tasks)
  Each sender will pass 100 messages of 100 bytes
  Time: 0.396
  CPU0 data recorded at offset=0x738000
      0 bytes in size
  CPU1 data recorded at offset=0x738000
      0 bytes in size
  CPU2 data recorded at offset=0x738000
      0 bytes in size
  CPU3 data recorded at offset=0x738000
      4096 bytes in size

  # trace-cmd report
  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=4
            <idle>-0     [003]    29.704387: call_function_single_entry: vector=251
            <idle>-0     [003]    29.704388: call_function_single_exit: vector=251
            <idle>-0     [003]    29.705950: call_function_single_entry: vector=251
            <idle>-0     [003]    29.705951: call_function_single_exit: vector=251
            <idle>-0     [003]    29.706462: call_function_single_entry: vector=251
            <idle>-0     [003]    29.706463: call_function_single_exit: vector=251
         hackbench-962   [003]    29.706501: call_function_single_entry: vector=251
         hackbench-962   [003]    29.706502: call_function_single_exit: vector=251
         hackbench-955   [003]    29.706521: call_function_single_entry: vector=251
         hackbench-955   [003]    29.706522: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101812: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101814: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101897: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101898: call_function_single_exit: vector=251
            <idle>-0     [003]    30.101985: call_function_single_entry: vector=251
            <idle>-0     [003]    30.101986: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102072: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102072: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102161: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102161: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102250: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102251: call_function_single_exit: vector=251
            <idle>-0     [003]    30.102372: call_function_single_entry: vector=251
            <idle>-0     [003]    30.102372: call_function_single_exit: vector=251


  CPU 0 is empty
  CPU 1 is empty
  CPU 2 is empty
  cpus=4
          <idle>-0     [003]  1067.718304: call_function_single_entry: vector=251
          <idle>-0     [003]  1067.718309: call_function_single_exit: vector=251

and that behaves the same as

  trace-cmd record -e 'call_function*' -f 'CPU == 3' hackbench