linux-kernel - A concern about overflow ring buffer mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <20181026.104513.2239058788450235574.davem@davemloft.net>
Date:   Fri, 26 Oct 2018 10:45:13 -0700 (PDT)
From:   David Miller <davem@...emloft.net>
To:     acme@...nel.org
CC:     linux-kernel@...r.kernel.org
Subject: A concern about overflow ring buffer mode

Since the last time I looked deeply into perf I notice that
perf top now uses a new ring buffer mode by default.

Basically, events are written in reverse order, and when fetching
events the tool uses an ioctl to "pause" the ring buffer.

I understand some of the reasons for pursing this kind of scheme but I
think there may be a huge downside to this design.

Yes, if the tool can't keep up with the kernel, we'd rather see newer
rather than older events.

However, pausing the ring buffer during the fetch is going to
virtually guaratee that we lose critical events that impact
interpretation of future events in a non-recoverable way.

The thing is, the new scheme causes events to be lost even if the tool
can keep up with the kernel.

Any event that happens while the tool is fetching the ring entries
will be lost forever.  The kernel simply skips queuing up the event
and increments a lost counter.  During a kernel build, I typically see
9 or so events lost each fetch.

Ok, if this is just a SAMPLE then fine, it's not a big deal.

But what if the lost event is a FORK or an EXEC or the worst one to
lose, an MMAP?

Now we can't even match up events properly and we get tons of those
dreaded "Unknown" symbols and DSOs.  The output looks terrible and the
tool becomes useless.

And yes this happens frequently.

I think the overwrite ring buffer mode should be seriously
reconsidered.  The "I'd rather see new than old events" part is fine,
but the "pause" part is not.  You can't turn event recording off on
the kernel side while you fetch some events, because it means that
critical events that allow us to properly interpret future events will
be lost.