linux-kernel - Re: [PATCH 3/4] nfsd: add some stub tracepoints around key vfs functions

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <31ce6d1c-b036-4a7d-a888-47a62a195dba@oracle.com>
Date: Thu, 6 Mar 2025 12:40:50 -0500
From: Chuck Lever <chuck.lever@...cle.com>
To: Jeff Layton <jlayton@...nel.org>, Neil Brown <neilb@...e.de>,
        Olga Kornievskaia <okorniev@...hat.com>, Dai Ngo <Dai.Ngo@...cle.com>,
        Tom Talpey <tom@...pey.com>, Trond Myklebust <trondmy@...nel.org>,
        Anna Schumaker <anna@...nel.org>,
        "David S. Miller" <davem@...emloft.net>,
        Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>
Cc: Sargun Dillon <sargun@...a.com>, linux-nfs@...r.kernel.org,
        linux-kernel@...r.kernel.org, netdev@...r.kernel.org
Subject: Re: [PATCH 3/4] nfsd: add some stub tracepoints around key vfs
 functions

On 3/6/25 11:28 AM, Jeff Layton wrote:
> On Thu, 2025-03-06 at 09:29 -0500, Chuck Lever wrote:
>> On 3/6/25 7:38 AM, Jeff Layton wrote:
>>> Sargun set up kprobes to add some of these tracepoints. Convert them to
>>> simple static tracepoints. These are pretty sparse for now, but they
>>> could be expanded in the future as needed.
>>
>> I have mixed feelings about this.

To be very clear: I'm always up for better observability! The details of
this patch are where I start to have some hesitation.

>> - Probably tracepoints should replace the existing dprintk call sites.
>>   dprintk is kind of useless for heavy traffic.
>>
> 
> I'm fine with removing dprintks as we go.

Removing them was controversial a few years ago when I first brought
this up... I would very much like to see these call sites gone, even if
we don't have immediate replacements in the form of trace points.

>> - Seems like other existing tracepoints could report most of the same
>>   information. fh_verify, for example, has a tracepoint that reports
>>   the file handle. There's an svc proc tracepoint, and an NFSv4 COMPOUND
>>   tracepoint that can report XID and procedure.
>>
> 
> The problem there is the lack of context. Yes, I can see that
> fh_verify() got called, but on a busy server it can be hard to tell why
> it got called. I see things like the fh_verify() tracepoint working in
> conjunction with these new tracepoints. IOW, you could match up the
> xids and see which fh_verify() was called for which operation.

If we're talking about NFSv3 only, sunrpc:svc_process records the XID,
nfsd thread, NFSv3 procedure name, and NFSD namespace of each incoming
RPC call. You also get the NFS client's IP address.

You can also enable nfsd:nfsd_fh_verify to capture several of those
items, plus the NFS file handle.

The kernel process information will be identical for both the svc_proc
and nfsd_fh_verify trace points -- that will tie the two records
together so you can match an XID to an NFS procedure and its file handle
argument.

If you want to see a little more you can enable the function_graph
plug-in for nfsd_dispatch().

Another approach is adding trace points in the XDR layer to capture
all of the arguments of incoming RPC calls:

https://web.git.kernel.org/pub/scm/linux/kernel/git/cel/linux.git/log/?h=topic-xdr-tracepoints

-- 
Chuck Lever