lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:	Sat, 5 Feb 2011 16:05:55 -0500 (EST)
From:	Justin Piszcz <jpiszcz@...idpixels.com>
To:	Emmanuel Florac <eflorac@...ellique.com>
cc:	linux-kernel@...r.kernel.org, linux-scsi@...r.kernel.org,
	linux-net@...r.kernel.org, Alan Piszcz <ap@...arrain.com>
Subject: Re: Supermicro X8DTH-6: Only ~250MiB/s from RAID<->RAID over
 10GbE?



On Sat, 5 Feb 2011, Emmanuel Florac wrote:

> Le Sat, 5 Feb 2011 14:35:52 -0500 (EST) vous écriviez:
>

To respond to everyone:

> Did you try launching 4 simultaneous cp operations over nfs to get to 
> 1.2 GB/s?
>  I've witnessed single stream copy performance with Samba being less than
> maximum due to Samba limitations.  Running multiple copy ops in parallel then
> usually saturates the pipe.


I tried 4 simultaenous cp's and there was little change, 250-320MiB/s.

Reader: 250MiB/s-300MiB/s
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  0  6      0 927176  16076 2886308    0    0 206416     0 26850 40663  0  4 86 10  0
  0  0      0 925620  16076 2887768    0    0 261620     0 31057 61111  0  6 86  8  0
  0  0      0 923928  16076 2889016    0    0 328852     0 40954 102136  0  8 90  2  0
  5  2      0 921112  16076 2890780    0    0 343476     0 39469 97957  0  8 90  2  0

Writer (its almost as if its caching the data and writing it out in
1.2-1.3Gbyte/sec chunks..
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  7  0    232  13980    408 3746900    0    0     0 1351926 37877 81924  2 32 61  5  0
  6  0    232  12740    404 3748308    0    0     0 1336768 38822 86505  2 31 62  5  0
  4  0    232  12524    404 3744672    0    0     0 1295000 39368 91005  1 30 63  5  0
  6  0    232  12304    404 3748148    0    0     0 1332776 39351 86929  2 31 62  5  0

I also noticed this: 'FS-Cache: Loaded'
Could this be slowing things down?

********* [Desktop test below]
When I copy on my desktop (Debian) systems, it transfers immediately:

  0  0   9172  83648 2822716 3343120    0    0    52   320 4249 3682  2  1 97  0
  0  0   9172  86532 2822716 3343176    0    0     0     0 4362 3074  0  1 99  0
  0  4   9172  62924 2822716 3363572    0    0 94212     0 5083 3044  1  3 90  7
  1  7   9172  63444 2822708 3364008    0    0 360448    32 19058 8825  0 15 48 37
  0  5   9172  61828 2821692 3367004    0    0 491520     0 26283 15282  0 24 43 32
  0  5   9172  59212 2821672 3373292    0    0 524288     0 28810 17370  0 27 33 40
  3  6   9172  57620 2821660 3355500    0    0 469364   128 25399 15825  0 21 42 36
********* [End of desktop test]

When I transfer using CentOS 5.5, there are a bunch of little reads, then it
averages out at around 250MiB/s:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  0  0      0 871088  16108 2942808    0    0  4091     3  552 1165  0  2 96  2  0
  0  0      0 871088  16108 2942808    0    0     0     0 1013 2342  0  0 100  0  0
  0  0      0 871224  16108 2942808    0    0     0     0 1031 2375  0  0 100  0  0
  0  8      0 870864  16108 2942808    0    0   288     0 1071 2396  0  0 86 14  0
  2  0      0 868636  16108 2943256    0    0   160     0 6348 18642  0  0 80 19  0
  0  0      0 868884  16116 2943248    0    0     0    28 31717 99894  0  4 96  0  0
  1  0      0 467040  16116 3343524    0    0 401444     0 40241 105777  0  4 93  4  0
  2  1      0  12300   8540 3802792    0    0 482668     0 44694 116988  0  6 88  6  0
  1  0      0  12412   3528 3805056    0    0 480124     0 44765 112272  0  6 88  6  0
  0  8      0  17560   3528 3798900    0    0 388288     0 37987 68367  0  5 85 10  0
  1  7      0  17868   3528 3802172    0    0 323296     0 33470 38398  0  5 83 11  0
  1  7      0  17260   3524 3801688    0    0 299584     0 30991 35153  0  5 83 11  0
  0  8      0  17272   3512 3802208    0    0 304352     0 31463 35400  0  5 84 11  0
  0  8      0  17228   3512 3801476    0    0 258816     0 27035 30651  0  4 84 1

Is there a way to disable the VFS/page cache some how to avoid whatever
FS-Cache is doing?

> Ok, could you do the following:
> dd if=/dev/urandom of=hugefile bs=1M count=<c>
> Where <c> is chosen so the resulting file is 2-3 times the RAM available
> in your server.
> Then redo the dd to null. Let's also check with the rsize from the nfs.
> /usr/bin/time -v dd if=hugefile of=/dev/null bs=<rsize used by NFS>
I also have tried rsize/wsize of 65536 with NFS, no difference there, rsize
and wsize back the default, 8192 I believe.

221M    hugefile
223M    hugefile
226M    hugefile
229M    hugefile
232M    hugefile

Had to skip urandom, too slow (7MiB/s or so)..

# dd if=/dev/zero of=hugefile bs=1M count=16384
16384+0 records in
16384+0 records out
17179869184 bytes (17 GB) copied, 14.6305 seconds, 1.2 GB/s


# /usr/bin/time -v dd if=hugefile of=/dev/null bs=8192
2097152+0 records in
2097152+0 records out
17179869184 bytes (17 GB) copied, 14.7656 seconds, 1.2 GB/s
         Command being timed: "dd if=hugefile of=/dev/null bs=8192"
         User time (seconds): 2.45
         System time (seconds): 8.70
         Percent of CPU this job got: 75%
         Elapsed (wall clock) time (h:mm:ss or m:ss): 0:14.76
         Average shared text size (kbytes): 0
         Average unshared data size (kbytes): 0
         Average stack size (kbytes): 0
         Average total size (kbytes): 0
         Maximum resident set size (kbytes): 0
         Average resident set size (kbytes): 0
         Major (requiring I/O) page faults: 2
         Minor (reclaiming a frame) page faults: 216
         Voluntary context switches: 57753
         Involuntary context switches: 1843
         Swaps: 0
         File system inputs: 0
         File system outputs: 0
         Socket messages sent: 0
         Socket messages received: 0
         Signals delivered: 0
         Page size (bytes): 4096
         Exit status: 0


procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
  r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
  0  1      0  17100    432 3758476    0    0  4607  1147  575 1051  0  2 95  3  0
  1  0      0  16980    432 3758888    0    0 1064760     0 9344 10698  1  4 94  1  0
  1  0      0  17100    432 3758888    0    0 1081808     0 9520 10608  1  4 94  1  0
  1  1      0  16912    440 3758908    0    0 1136764    48 9899 11305  1  4 94  1  0
  1  0      0  16972    440 3758908    0    0 1112664     0 9756 11126  1  4 94  1  0
  1  0      0  17080    440 3759040    0    0 1140564     0 9943 11492  1  4 94  1  0
  2  0      0  16976    440 3759040    0    0 1134020     0 9910 11475  1  4 94  1  0
  1  0      0  17364    440 3758940    0    0 1132380     0 9865 11236  1  4 94


Justin.


Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ