linux-kernel - Re: [PATCH V3 9/9] tracing: Add timerlat tracer

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20210611160340.6970e10c@gandalf.local.home>
Date:   Fri, 11 Jun 2021 16:03:40 -0400
From:   Steven Rostedt <rostedt@...dmis.org>
To:     Daniel Bristot de Oliveira <bristot@...hat.com>
Cc:     linux-kernel@...r.kernel.org, Phil Auld <pauld@...hat.com>,
        Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
        Kate Carcia <kcarcia@...hat.com>,
        Jonathan Corbet <corbet@....net>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Alexandre Chartre <alexandre.chartre@...cle.com>,
        Clark Willaims <williams@...hat.com>,
        John Kacur <jkacur@...hat.com>,
        Juri Lelli <juri.lelli@...hat.com>, linux-doc@...r.kernel.org
Subject: Re: [PATCH V3 9/9] tracing: Add timerlat tracer

On Fri, 11 Jun 2021 14:59:13 +0200
Daniel Bristot de Oliveira <bristot@...hat.com> wrote:

> ------------------ %< -----------------------------
> It is worth mentioning that the *duration* values reported
> by the osnoise: events are *net* values. For example, the
> thread_noise does not include the duration of the overhead caused
> by the IRQ execution (which indeed accounted for 12736 ns). But
> the values reported by the timerlat tracer (timerlat_latency)
> are *gross* values.
> 
> The art below illustrates a CPU timeline and how the timerlat tracer
> observes it at the top and the osnoise: events at the bottom. Each "-"
> in the timelines means 1 us, and the time moves ==>:
> 
>      External          context irq                  context thread
>       clock           timer_latency                 timer_latency
>       event              18 us                          48 us 
>         |                  ^                             ^
>         v                  |                             |
>         |------------------|                             |       <-- timerlat irq timeline
>         |------------------+-----------------------------|       <-- timerlat thread timeline
>                            ^                             ^
>  ===================== CPU timeline ======================================
>                    [timerlat/ irq]  [ dev irq ]                          
>  [another thread...^             v..^         v........][timerlat/ thread]  
>  ===================== CPU timeline ======================================
>                    |-------------|  |---------|                  <-- irq_noise timeline
>                                  |--^         v--------|         <-- thread_noise timeline
>                                  |            |        |
>                                  |            |        + thread_noise: 10 us
>                                  |            +-> irq_noise: 9 us
>                                  +-> irq_noise: 13 us
> 
>  --------------- >% --------------------------------  

That's really busy, and honestly, I can't tell what is what.

The "context irq timer_latency" is a confusing name. Could we just have
that be "timer irq latency"? And "context thread timer_latency" just be
"thread latency". Adding too much text to the name actually makes it harder
to understand. We want to simplify it, not make people have to think harder
to see it.

I think we can get rid of the "<-- .* timeline" to the right.  I don't
think they are necessary. Again, the more you add to the diagram, the
busier it looks, and the harder it is to read.

Could we switch "[timerlat/ irq]" to just "[timer irq]" and explain how
that "context irq timer_latency"/"timer irq latency" is related?

Should probably state that the "dev irq" is an unrelated device interrupt
that happened.

What's with the two CPU timeline lines? Now there I think it would be
better to have the arrow text by itself.

And finally, not sure if you plan on doing this, but have a output of the
trace that would show the above.

Thus, here's what I would expect to see:

      External         
       clock         timer irq latency                 thread latency
       event              18 us                          48 us 
         |                  ^                             ^
         v                  |                             |
         |------------------|                             |
         |------------------+-----------------------------|       
                            ^                             ^
  =========================================================================
                    [timerlat/ irq]  [ dev irq ]                             
  [another thread...^             v..^         v........][timerlat/ thread]  <-- CPU task timeline
  =========================================================================
                    |-------------|  |---------|
                                  |--^         v--------|
                                  |            |        |
                                  |            |        + thread_noise: 10 us
                                  |            +-> irq_noise: 9 us
                                  +-> irq_noise: 13 us
 
 The "[ dev irq ]" above is an interrupt from some device on the system that
 causes extra noise to the timerlat task.

I think the above may be easier to understand, especially if the trace
output that represents it is below.

Also, I have to ask, shouldn't the "thread noise" really start at the
"External clock event"?

-- Steve