lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Thu, 23 Oct 2008 14:36:28 +0200
From:	Max Kellermann <max@...mpel.org>
To:	linux-kernel@...r.kernel.org, linux-nfs@...r.kernel.org,
	trond.myklebust@....uio.no
Cc:	harry@...os.washington.edu
Subject: Re: High load in 2.6.27, NFS / rpcauth_lookup_credcache()?

On 2008/10/22 11:12, Max Kellermann <max@...mpel.org> wrote:
> after I was able to fix http://lkml.org/lkml/2008/10/17/147, the
> server which was already upgraded to 2.6.27.2 still gets very high
> load.  It is a web server with NFS file storage (NetApp), and while
> the others in the cluster (kernel 2.6.25) have a load of 1-3, 2.6.27.2
> gets 30-50.
> 
> I did an oprofile, with the following results (server just started,
> load "only" 5-10):
> 
> 87593    56.1116  (no location information)   vmlinux
> vmlinux                  rpcauth_lookup_credcache
> 16037    10.2732  auth_generic.c:0            vmlinux
> vmlinux                  generic_match
> 6460      4.1382  (no location information)   php4
> php4                     (no symbols)
> 2478      1.5874  (no location information)   libc-2.7.so
> libc-2.7.so              (no symbols)
> [...]
> 
> We havn't configured any special authentication method.  It is a NFSv3
> over UDP mount, but the kernel has NFSv4 and therefore KRB5 enabled.
> 
> Any ideas why rpcauth_lookup_credcache() goes overboard with CPU
> usage?

I have bisected the problem: 98a8e323 is the result ("SUNRPC: Add a
helper rpcauth_lookup_generic_cred()").  5c691044 is ok.

See the attached oprofile annotation data for both commits.  I guess
that the function rpcauth_lookup_credcache() is waiting for a spinlock
too often and too long.  Trond, any idea?

Harry: added you to Cc because your problem sounds similar.

Max

View attachment "annotate.98a8e3239427051f5d44f2025b398bdcc3918f37" of type "text/plain" (7840 bytes)

View attachment "annotate.5c691044ecbca04dd558fca4c754121689fe1b34" of type "text/plain" (7841 bytes)

Powered by blists - more mailing lists