netdev - Fwd: 2.6.26.3 kernel - progressive slowdown over NFS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <2c0942db0812091638o3d0fe68ek70744f6938949b25@mail.gmail.com>
Date:	Tue, 9 Dec 2008 16:38:21 -0800
From:	"Ray Lee" <ray-lk@...rabbit.org>
To:	LKML <linux-kernel@...r.kernel.org>, linux-nfs@...r.kernel.org,
	Netdev <netdev@...r.kernel.org>
Subject: Fwd: 2.6.26.3 kernel - progressive slowdown over NFS

[ adding netdev, linux-nfs to cc: -- rbl ]

Hi,


We have a simple python program which keeps running a C loop to lstat
NFS mounted directories.  We are seeing some weird behavior w.r.t. the
run-time of this program on 2.6.26.3 kernel vs 2.6.24 kernel.

The run-time of the following code increases over time on the 2.6.26.3
kernel, whereas remains flat (as expected) on the 2.6.24 kernel.
[See attached graphs - B1.jpg and B2.jpg]  Once the 2.6.26.3 machine
gets into this state, we need to restart the box to get back to
reasonable run-times.  Is this a known issue ?

Setup :

Machine A (2.6.26) : exports NFS directory  - /a/baz contains 10,000
directories bar0 ... bar9999
/a/baz        *(rw,sync,no_root_squash,no_all_squash,subtree_check)

Machine B1 (2.6.26.3) : mounts NFS dir RO from A - graph B1.jpg
10.x.x.x:/a/baz on /baz type nfs
(ro,vers=3,rsize=4096,wsize=4096,namlen=255,hard,nointr,nolock,proto=udp,timeo=11,retrans=2,sec=sys,mountproto=udp,addr=10.x.x.x)

Machine B2 (2.6.24) : mounts NFS dir RO from A - graph B2.jpg
10.x.x.x:/a/baz on /baz type nfs
(ro,vers=3,rsize=4096,wsize=4096,hard,nointr,nolock,proto=udp,timeo=11,retrans=2,sec=sys,addr=10.x.x.x)

Repro :

B1 and B2, run the following python program :
{{{
#!/usr/bin/env python

import os
import sys
import time

while True:
   t1 = time.time()
   rv = os.system("/a.out 10000 >& /dev/null")   # lstat the 10000
directories mounted via NFS
   t2 = time.time()
   print >> sys.stderr, "%.3f" % (t2 - t1), rv
}}}

where a.out is the following C code :

{{{
#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <unistd.h>

int main(int argc, char **argv)
{
  int i, n = atoi(argv[1]);
  char filename[64];
  struct stat statbuf;

  for (i = 0; i < n; i++) {
     sprintf(filename, "/baz/bar%d", i);
     lstat(filename, &statbuf);
  }

  printf("done\n");
  return 0;
}
}}}

Attachments :

B1.jpg and B2.jpg - runtimes on the 2 client machines -
X-axis : iteration number
Y-axis : run-time

Config files : - for the machines B1, B2 and A

b1.2.6.26.3.config
b2.2.6.24.config
a.2.6.26.config


- P

Download attachment "B1.jpg" of type "image/jpeg" (46848 bytes)

Download attachment "B2.jpg" of type "image/jpeg" (33952 bytes)

Download attachment "b1.2.6.26.3.config" of type "application/octet-stream" (39008 bytes)

Download attachment "b2.2.6.24.config" of type "application/octet-stream" (27945 bytes)

Download attachment "a.2.6.26.config" of type "application/octet-stream" (38494 bytes)