linux-kernel - Re: Page Cache writeback too slow, SSD/noop scheduler/ext2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <C5F2C492.D4A8%jos@hyves.nl>
Date:	Fri, 27 Mar 2009 17:59:43 +0100
From:	Jos Houtman <jos@...es.nl>
To:	Wu Fengguang <fengguang.wu@...el.com>,
	Nick Piggin <nickpiggin@...oo.com.au>
CC:	<linux-kernel@...r.kernel.org>, Jeff Layton <jlayton@...hat.com>,
	Dave Chinner <david@...morbit.com>,
	<linux-fsdevel@...r.kernel.org>, <jens.axboe@...cle.com>,
	<akpm@...ux-foundation.org>, <hch@...radead.org>,
	<linux-nfs@...r.kernel.org>
Subject: Re: Page Cache writeback too slow,   SSD/noop scheduler/ext2

Hi,

>> 
>> kupdate surely should just continue to keep trying to write back pages
>> so long as there are more old pages to clean, and the queue isn't
>> congested. That seems to be the intention anyway: MAX_WRITEBACK_PAGES
>> is just the number to write back in a single call, but you see
>> nr_to_write is set to the number of dirty pages in the system.

And when it's congested it should just wait a little bit before continuing.

>> On your system, what must be happening is more_io is not being set.
>> The logic in fs/fs-writeback.c might be busted.

I don't know about more_io, but I agree that the logic seems busted.

> 
> Hi Jos,
> 
> I prepared a debugging patch for 2.6.28. (I cannot observe writeback
> problems on my local ext2 mount.)

Thanx for the patch, but for the next time: How should I apply it?
it seems to be context aware (@@) and broke on all kernel versions I tried
2.6.28/2.6.28.7/2.6.29

Because I saw the patch only a few hour ago and didn't want to block on your
reply I decided to patch it manually and in the process ported it to 2.6.29.

As for the information the patch provided: It is most helpful.

Attached you will find a list of files containing dirty pages and the count
of there dirty pages, there is also a dmesg output where I trace the
writeback for 40 seconds.

I did some testing on my own using printk's and what I saw is that for the
inodes located on sdb1 (the database) a lot of times they would pass
http://lxr.linux.no/linux+v2.6.29/fs/fs-writeback.c#L335
And then redirty_tail would be called, I haven't had the time to dig deeper,
but that is my primary suspect for the moment.

Thanx again, 

Jos

Download attachment "filecache-27-march.txt" of type "application/octet-stream" (4636 bytes)

Download attachment "dmesg-27-march.txt" of type "application/octet-stream" (17351 bytes)