linux-kernel - Re: iowait stability regressed from kernel 2.6.22 under heavy load of multi-thread aio writing/reading

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 15 Apr 2008 09:37:03 -0400
From:	Jeff Moyer <jmoyer@...hat.com>
To:	"rae l" <crquan@...il.com>
Cc:	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: iowait stability regressed from kernel 2.6.22 under heavy load of multi-thread aio writing/reading

"rae l" <crquan@...il.com> writes:

> I found a problem with the vanilla kernel from 2.6.22 (include 23, 24):
>
> the situation is to test a customized linux distribution under heavy IO load:
> 1. the client process initiates tens of POSIX threads (using
> libpthread), each thread uses aio_write or aio_read(using librt)
> operating
>    on one small file, then close it and write another small file;
>    the whole objective is to get a maxium throughput of small files,
> by generating heavy aio stress on the system;
>
> I have tested the vanilla kernel 2.6.22/23/24.y, all these kernels
> have the common problem:
> 1. I use top, vmstat, and iostat to monitor the system performance, I
> found that the iowait time of CPU is high at most time,
>    above 60%, and not stable while the client process running,
> although the throughput is stable, the CPU iowait time not stable;
>    As a result of instability processes felt too long and unacceptable delay.
> 2. in seldom testing cases, the system even stopped: with bi/bo are 0
> and iowait 100%, all writing processes are blocking with
>    uninterruptible state (D state in ps output), these cases are all
> system running after several days writing, but cannot guarantee
>    to reproduce;
>
> First I suspect the filesystem or the storage medium have problems, I
> tried ext2/ext3/reiserfs/xfs, for different filesytems and
> SATA/SAS, IDE disks, and several commercial hard RAID card for
> different storage medium, but the bad result remained;
>
> Then I tested 2.6.21.7 kernel, this results a stable iowait CPU time
> (below 10%) but not so good throughput of small files;
>
> Now I think the improvements of IO effciency in the development of
> 2.6.22 also caused the instability of iowait time, right? but how
> to find out? Simple bisecting seems not work.

Why doesn't bisecting work?  Can you provide your test code so others
can verify your findings?

> by the way, a question is how to guarantee a kernel not regress under
> heavy multi-thread aio writing load? ltp project seems not give the
> answer:

Well, providing your test code would be a step closer to achieving this
guarantee.  If it is scriptable, then there is a chance we could
integrate it into the aio-dio regression test suite:

  http://git.kernel.org/?p=linux/kernel/git/zab/aio-dio-regress.git;a=summary

Cheers,

Jeff
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/