linux-kernel - Re: High load in 2.6.27, NFS / rpcauth_lookup

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20081023123628.GA18549@squirrel.roonstrasse.net>
Date:	Thu, 23 Oct 2008 14:36:28 +0200
From:	Max Kellermann <max@...mpel.org>
To:	linux-kernel@...r.kernel.org, linux-nfs@...r.kernel.org,
	trond.myklebust@....uio.no
Cc:	harry@...os.washington.edu
Subject: Re: High load in 2.6.27, NFS / rpcauth_lookup_credcache()?

On 2008/10/22 11:12, Max Kellermann <max@...mpel.org> wrote:
> after I was able to fix http://lkml.org/lkml/2008/10/17/147, the
> server which was already upgraded to 2.6.27.2 still gets very high
> load.  It is a web server with NFS file storage (NetApp), and while
> the others in the cluster (kernel 2.6.25) have a load of 1-3, 2.6.27.2
> gets 30-50.
> 
> I did an oprofile, with the following results (server just started,
> load "only" 5-10):
> 
> 87593    56.1116  (no location information)   vmlinux
> vmlinux                  rpcauth_lookup_credcache
> 16037    10.2732  auth_generic.c:0            vmlinux
> vmlinux                  generic_match
> 6460      4.1382  (no location information)   php4
> php4                     (no symbols)
> 2478      1.5874  (no location information)   libc-2.7.so
> libc-2.7.so              (no symbols)
> [...]
> 
> We havn't configured any special authentication method.  It is a NFSv3
> over UDP mount, but the kernel has NFSv4 and therefore KRB5 enabled.
> 
> Any ideas why rpcauth_lookup_credcache() goes overboard with CPU
> usage?

I have bisected the problem: 98a8e323 is the result ("SUNRPC: Add a
helper rpcauth_lookup_generic_cred()").  5c691044 is ok.

See the attached oprofile annotation data for both commits.  I guess
that the function rpcauth_lookup_credcache() is waiting for a spinlock
too often and too long.  Trond, any idea?

Harry: added you to Cc because your problem sounds similar.

Max

View attachment "annotate.98a8e3239427051f5d44f2025b398bdcc3918f37" of type "text/plain" (7840 bytes)

View attachment "annotate.5c691044ecbca04dd558fca4c754121689fe1b34" of type "text/plain" (7841 bytes)