linux-ext4 - Re: fio test triggering bad data on ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4C1F3D08.4020205@fusionio.com>
Date:	Mon, 21 Jun 2010 12:20:56 +0200
From:	Jens Axboe <jaxboe@...ionio.com>
To:	Eric Sandeen <sandeen@...hat.com>
CC:	"tytso@....edu" <tytso@....edu>,
	"adilger@....com" <adilger@....com>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
Subject: Re: fio test triggering bad data on ext4

On 2010-06-18 20:14, Jens Axboe wrote:
> On 2010-06-18 20:04, Eric Sandeen wrote:
>> Jens Axboe wrote:
>>> On 2010-06-18 17:28, Eric Sandeen wrote:
>>>> Jens Axboe wrote:
>>>>> On 18/06/10 16.59, Eric Sandeen wrote:
>>>>>   
>>>>>> Eric Sandeen wrote:
>>>>>>     
>>>>>>> Jens Axboe wrote:
>>>>>>>       
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I was writing a small fio job file to do writes and read verifies on a
>>>>>>>> device. It forks 32 processes, each writing randomly to 4 files with a
>>>>>>>> block size between 4k and 16k. When it has written 1024 of those blocks,
>>>>>>>> it'll verify the oldest 512 of them. Each block is checksummed for every
>>>>>>>> 512b. It uses libaio and O_DIRECT.
>>>>>>>>
>>>>>>>> It works on ext2 and btrfs. I haven't run it to completion yet, but they
>>>>>>>> survive 15-20 minutes just fine. ext4 doesn't even go a full minutes
>>>>>>>> before this triggers:
>>>>>>>>         
>>>>>>> Jens, can you try XFS too?  Since ext3 can't do direct IO to a hole,
>>>>>>> (and I'm not sure about btrfs in that regard), ext4 may be most similar
>>>>>>> to xfs's behavior on the test ... wondering how it fares.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> -Eric
>>>>>>>       
>>>>>> Actually mingming had a patch for direct-io.c which may be related, I'll
>>>>>> test that out.
>>>>>>     
>>>>> OK, I'll try XFS tonight as well.
>>>>>
>>>>>
>>>>>   
>>>> I haven't been able to reproduce it on ext4 here, yet.
>>>>
>>>> FWIW here's the patch from mingming:
>>>>
>>>> When unaligned DIO writes, skip zero out the block if the buffer is marked
>>>> unwritten. That means there is an asynconous direct IO (append or fill the hole)
>>>> still pending.
>>>>
>>>> Signed-off-by: Mingming Cao <cmm@...ibm.com>
>>>> ---
>>>>  fs/direct-io.c |    3 ++-
>>>>  1 file changed, 2 insertions(+), 1 deletion(-)
>>>>
>>>> Index: linux-git/fs/direct-io.c
>>>> ===================================================================
>>>> --- linux-git.orig/fs/direct-io.c	2010-05-07 15:42:22.855033403 -0700
>>>> +++ linux-git/fs/direct-io.c	2010-05-07 15:44:17.695007770 -0700
>>>> @@ -740,7 +740,8 @@
>>>>  	struct page *page;
>>>>  
>>>>  	dio->start_zero_done = 1;
>>>> -	if (!dio->blkfactor || !buffer_new(&dio->map_bh))
>>>> +	if (!dio->blkfactor || !buffer_new(&dio->map_bh)
>>>> +	    || buffer_unwritten(&dio->map_bh))
>>>>  		return;
>>>>  
>>>>  	dio_blocks_per_fs_block = 1 << dio->blkfactor;
>>>>
>>>>
>>>
>>> What is this patch against?
>>>
>>
>> Applied to 2.6.32, seems to apply upstream as well.
>>
>> It hits dio_zero-block()
> 
> Irk indeed, I am blind. The patch does not fix it.

So just to confirm that this isn't a new regression, 2.6.34 fails in the
same way. If I change the test to make the random writes overwrite
existing blocks instead of filling holes, then there are no problems
either.

-- 
Jens Axboe

--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html