linux-kernel - Re: [PATCH 2/3] tracing: export stats of ring buffers to userspace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.00.0904302313120.20374@gandalf.stny.rr.com>
Date:	Thu, 30 Apr 2009 23:23:52 -0400 (EDT)
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Frederic Weisbecker <fweisbec@...il.com>
cc:	linux-kernel@...r.kernel.org, Ingo Molnar <mingo@...e.hu>,
	Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [PATCH 2/3] tracing: export stats of ring buffers to userspace


On Fri, 1 May 2009, Frederic Weisbecker wrote:

> On Thu, Apr 30, 2009 at 10:22:12PM -0400, Steven Rostedt wrote:
> > From: Steven Rostedt <srostedt@...hat.com>
> > 
> > This patch adds stats to the ftrace ring buffers:
> > 
> >  # cat /debugfs/tracing/per_cpu/cpu0/stats
> >  entries: 42360
> >  overrun: 30509326
> >  commit overrun: 0
> >  nmi dropped: 0
> > 
> > Where entries are the total number of data entries in the buffer.
> > 
> > overrun is the number of entries not consumed and were overwritten by
> > the writer.
> > 
> > commit overrun is the number of entries dropped due to nested writers
> > wrapping the buffer before the initial writer finished the commit.
> 
> 
> I feel a bit confused with this one.
> How such a thing can happen? The write page and the commit page
> are not the same. So is that because we can have (ring-buffer inspires
> all of us to try ascii-art):
> 
> 
>           Write page                      Commit page (which becomes new write page) 
> ------------------------------------ -----------------
>             |           |          | |               |
>   Writer 1  | Writer 2  | Writer n | | Writer n + 1  | .....
>   reserved  | reserved  | reserved | | reserved      |
> -----------------------------------  ----------------
>     |                                             ^
>     |                                             |
>     ---------------- Was supposed to commit here--|
>  
> 
> I know this is silly, my picture seem to show a data copy whereas
> the ring buffer deals with page pointers.
> But the commit page on the ring buffer is a mistery for me.
> Just because you haven't drawn in in ascii in your comments :)
> 

I have a bunch of ascii art that explains all this in my documentation 
that details the lockless version.

The commit page is the page that holds the last full commit.

 ring_buffer_unlock_commit()

On ring_buffer_lock_reserve() we reserve some data after the last commit.

            commit
               |
               V
    +---+    +---+    +---+    +---+
<---|   |--->|   |--->|   |--->|   |--->
--->|   |<---|   |<---|   |<---|   |<---
    +---+    +---+    +---+    +---+
               ^
               |
             tail (writer)

We do not disable interrupts (or softirqs) between 
ring_buffer_lock_reserve and ring_buffer_unlock_commit. If we get 
preempted by an interrupt or softirq, and it writes to the ring buffer, it 
will move the tail, but not the commit. Only the outer most writer (non 
nested) can do that:

            commit     
               |
               V
    +---+    +---+    +---+    +---+
<---|   |--->|   |--->|   |--->|   |--->
--->|   |<---|   |<---|   |<---|   |<---
    +---+    +---+    +---+    +---+
                        ^
                        |
                      tail (writer)


But lets say we are running the function graph tracer along with the event 
tracer. And to save space, we shrunk the ring buffer down to a few pages. 
(more than 2)  We are writing an event and get preempted by an interrupt 
followed by several softirqs, and these softirqs perform a lot of 
functions. It can push the tail all around the buffer:

            commit
               |
               V
    +---+    +---+    +---+    +---+
<---|   |--->|   |--->|   |--->|   |--->
--->|   |<---|   |<---|   |<---|   |<---
    +---+    +---+    +---+    +---+
      ^
      |
    tail (writer)


This happens before we even finish the original write. But the tail can 
not push the commit forward, because when we fall out of this stack of 
writers, that original writer is in the process of writing into the ring 
buffer. Thus we need to drop any more entries that want to push the tail 
pointer onto the commit pointer.

Thus, when this happens we record it with commit_overrun.

-- Steve



--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/