linux-kernel - RE: ftrace not showing the process names for all processes on syscall events

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7dec110c2dc14792ba744419a4eb907e@AcuMS.aculab.com>
Date:   Mon, 30 Mar 2020 15:34:08 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     'Steven Rostedt' <rostedt@...dmis.org>
CC:     Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: RE: ftrace not showing the process names for all processes on syscall
 events

From: Steven Rostedt
> Sent: 30 March 2020 16:09
> On Mon, 30 Mar 2020 14:28:01 +0000
> David Laight <David.Laight@...LAB.COM> wrote:
> 
> > I've just updated one of my systems to 5.6.0-rc7+ (end of last week).
> > ftrace in showing <...>-3179 in the system call events for a couple
> > of threads of the active processes.
> > Other threads of the same processes are fine.
> > The scheduler process switch events also show the full name.
> >
> > Is this a known regression?
> 
> 
> Well, that code hasn't changed in years. But can you explain more of what
> you did? Was this the "trace" file, or "trace_pipe" file? The command names
> are cached in an array (see /sys/kernel/tracing/saved_cmdlines) of the size
> that is defined by the saved_cmdlines_size file.

It was the 'trace' file - ie the text formatted by the kernel.
I've looked at a lot of traces over the last few weeks and not seen it before.
The workload is the same, so I assumed there might be a kernel change.
I've just updated from a 5.4-rc7 kernel to a 5.6-rc7 one.

> The cmdlines get updated via the sched_switch and sched_waking trace
> events. The update is protected by a spinlock, which is only taken with a
> "trylock", if the lock fails, then it does not get updated (we don't want
> to hold back the running code just to cache the name of an event), but it
> will try at the next sched event until it succeeds. This means that under
> strong contention, it may fail to cache certain names.

I've not got a silly number of context switches.
But it seemed to have consistently lost some process names even when they
went idle and then resumed.

I would have been tracing the schedule process wakeup and switch events,
system call entry/exit and irq+softirq.

> This is not a regression, it's really just the work load that can cause
> event names to be missed. I could work on something that if you have the
> sched events enabled, that the output side could do its own caching to get
> better results.

Ok if nothing has changed I'll stop worrying and work out what I've broken.

Oh, does the 'function_graph' code ignore tail calls?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)