linux-kernel - Re: Strange I/O slowdown with 4.x kernels

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CACVXFVMK5SzF9+TrXZWSWkm5j03nw6euYft_Uf0mfEbzSSm+7A@mail.gmail.com>
Date:	Tue, 3 May 2016 11:48:28 +0800
From:	Ming Lei <tom.leiming@...il.com>
To:	rus@...nxsoft.com
Cc:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Strange I/O slowdown with 4.x kernels

On Fri, Apr 29, 2016 at 9:36 PM, Rus <rus@...nxsoft.com> wrote:
> Hi,
>
> After switching from 3.x to 4.x I'm observing strange filesystem (ext4
> used) slowdown just after fresh boot - it is 100% reproducible. The
> simple test just after boot at root fs first shows the good
> performance :
>
> dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
> 10240+0 records in
> 10240+0 records out
> 83886080 bytes (84 MB) copied, 0.5676 s, 150.3 MB/s
>
> But after some small i/o stress (completed updatedb command for ex)
> the i/o speed drops dramatically :
>
> dd if=/dev/zero of=/output bs=8k count=10k; rm -f /output
> 10240+0 records in
> 10240+0 records out
> 83886080 bytes (84 MB) copied, 19.5703 s, 4.3 MB/s
>
> What is interesting is that hdparm test still shows normal performance :
>
> hdparm -t /dev/sda1
>
> /dev/sda1:
>  Timing buffered disk reads: 448 MB in  3.01 seconds = 148.79 MB/sec
>
> How to debug such thing (bissect can take too much time because of big
> version range) ? The config attached, this is ordinary 32-bit system

git bisect uses bineary search which is often quick and takes
not more than 20 iterations, for example:

$ git bisect start
$ git bisect good v3.12
$ git bisect bad HEAD
Bisecting: 93973 revisions left to test after this (roughly 17 steps)
[b0d4c5ec9730f7544772f50f84dc0dc730b1d551] mwifiex: update IEs after
AP has started

> with 32GB ECC RAM.
>  The slowdown clearly triggered with last stable (4.5.2) and latest rc
> (4.6-rc5) kernels. The 3.x kernels do not have such bug.
> Can supply any additional info.
>
> TIA, Rus
>
> --
> SfinxSoft
> http://sfinxsoft.com
>
>
>
> --
> SfinxSoft
> http://sfinxsoft.com



-- 
Ming Lei