linux-ext4 - Re: Using O

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <6601abe90907210745k3730f74dq62f1fe6539722b4d@mail.gmail.com>
Date:	Tue, 21 Jul 2009 07:45:58 -0700
From:	Curt Wohlgemuth <curtw@...gle.com>
To:	Eric Sandeen <sandeen@...hat.com>
Cc:	Xiang Wang <xiangw@...gle.com>, linux-ext4@...r.kernel.org
Subject: Re: Using O_DIRECT in ext4

On Mon, Jul 20, 2009 at 8:41 PM, Eric Sandeen<sandeen@...hat.com> wrote:
> Xiang Wang wrote:
>> Hi,
>>
>> Recently I've been experimenting with O_DIRECT in ext4 to get a
>> feeling of how much file fragmentation will be generated.
>>
>> On a newly formatted ext4 partition(no-journal), I created a top-level
>> directory and under this top-level directory I ran a test program to
>> generate some files.
>>
>> The test program does the following:
>> -- create multiple threads(in my test case: 16 threads)
>> -- each thread creates a file with the O_DIRECT flag and keeps
>> extending the file to 1MB
>> Since these threads run concurrently, they compete in block allocation.
>>
>> After the program ran to a completion, I ran filefrag on each file and
>> measure how many extents there are in the file.
>> And here is a sample result:
>> file0: 6 extents found
>> file1: 20 extents found
>> file2: 7 extents found
>> file3: 6 extents found
>> file4: 6 extents found
>> file5: 5 extents found
>> file6: 6 extents found
>> file7: 20 extents found
>> file8: 20 extents found
>> file9: 20 extents found
>> file10: 20 extents found
>> file11: 20 extents found
>> file12: 20 extents found
>> file13: 19 extents found
>> file14: 19 extents found
>> file15: 19 extents found
>>
>> Looks like these files are quite heavily fragmented.
>
> Multiple parallel extending DIOs in a single dir is a tough case for a
> filesystem - it has no hints about what to do, and can't use delalloc to
> wait to see what's happening; it just has to allocate things as they
> come, more or less.
>
>> For comparison, I did the same experiment on an ext2 partition,
>> resulting in each file having only 1 extent.
>
> Interestinng, not sure I would have expected that.

Same with us; we're looking into more variables to understand it.

>> I also did the experiments of using buffered writes(by removing the
>> O_DIRECT flag) on ext2 and ext4, both resulting in each file having
>> only 1 extent.
>
> delayed allocation at work I suppose.
>
>> I am wondering whether this kind of file fragmentation is already a
>> known issue in ext4 when O_DIRECT is used? Is it something by design?
>> Since it seems like ext2 does not have this issue under my test case,
>> is it necessary that we make the behavior of ext4 similar to ext2
>> under situations like this?
>
> Is this representative of a real workload?

Not exactly perhaps, but we do have apps that are showing
significantly more fragmentation in their files on ext4 than with
ext2, while using O_DIRECT (e.g., 8 extents on ext4 vs 1 on ext2, as
reported by filefrag).  The experiment above is synthetic, but fairly
representative.

(Hence the related questions about fallocate, since this is one
possible, though ugly, workaround.)

Curt
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html