linux-kernel - iowait stability regressed from kernel 2.6.22 under heavy load of multi-thread aio writing/reading

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <91b13c310804110217g1d2c3ee4p69fee6fd43f4abd2@mail.gmail.com>
Date:	Fri, 11 Apr 2008 17:17:31 +0800
From:	"rae l" <crquan@...il.com>
To:	linux-fsdevel@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org
Subject: iowait stability regressed from kernel 2.6.22 under heavy load of multi-thread aio writing/reading

I found a problem with the vanilla kernel from 2.6.22 (include 23, 24):

the situation is to test a customized linux distribution under heavy IO load:
1. the client process initiates tens of POSIX threads (using
libpthread), each thread uses aio_write or aio_read(using librt)
operating
   on one small file, then close it and write another small file;
   the whole objective is to get a maxium throughput of small files,
by generating heavy aio stress on the system;

I have tested the vanilla kernel 2.6.22/23/24.y, all these kernels
have the common problem:
1. I use top, vmstat, and iostat to monitor the system performance, I
found that the iowait time of CPU is high at most time,
   above 60%, and not stable while the client process running,
although the throughput is stable, the CPU iowait time not stable;
   As a result of instability processes felt too long and unacceptable delay.
2. in seldom testing cases, the system even stopped: with bi/bo are 0
and iowait 100%, all writing processes are blocking with
   uninterruptible state (D state in ps output), these cases are all
system running after several days writing, but cannot guarantee
   to reproduce;

First I suspect the filesystem or the storage medium have problems, I
tried ext2/ext3/reiserfs/xfs, for different filesytems and
SATA/SAS, IDE disks, and several commercial hard RAID card for
different storage medium, but the bad result remained;

Then I tested 2.6.21.7 kernel, this results a stable iowait CPU time
(below 10%) but not so good throughput of small files;

Now I think the improvements of IO effciency in the development of
2.6.22 also caused the instability of iowait time, right? but how
to find out? Simple bisecting seems not work.

by the way, a question is how to guarantee a kernel not regress under
heavy multi-thread aio writing load? ltp project seems not give the
answer:
http://ltp.sourceforge.net/tooltable.php

--
Denis
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/