linux-ext4 - Re: Repeatable block allocation problem.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <3c8d9b29-d04c-a6d6-4a80-19d2e344205a@redhat.com>
Date:   Mon, 17 Sep 2018 12:28:54 -0500
From:   Eric Sandeen <sandeen@...hat.com>
To:     Joshi <joshiiitr@...il.com>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: Repeatable block allocation problem.

On 9/17/18 11:40 AM, Joshi wrote:
> It's not that I need specific values for extents/blocks for a file. It
> is just about being able to create/prepare a file of given size with
> same (repeatable) number of extents and block numbers, in a controlled
> environment.


> I thought it would happen naturally, if I keep preconditions same
> (i.e. post format with lazy initialization disabled. I had not
> disabled lazy-allocation though).
> But extent count differed, at times.
> And I wondered whether Ext4 code makes any conscious attempt to
> randomize the extent/block allocation?

find_group_orlov() randomly selects block groups for root directories,
but AFAIK there is no explicit random behavior in the block allocator.

It's more of a chaotic system than intentional random selection.  :)

> End objective of above was to create fragmented file, with repeatable
> extent-count/block-numbers.

I understand that you want repeatable allocation behavior.

I still don't know exactly what behavior you are ultimately trying to
test, though.

> I'll check xfstests as well.

The tests marked as being in the defrag group are:

tests/btrfs/005
tests/btrfs/021
tests/btrfs/062
tests/btrfs/067
tests/btrfs/070
tests/btrfs/072
tests/btrfs/074
tests/ext4/020
tests/ext4/301
tests/ext4/302
tests/ext4/303
tests/ext4/304
tests/ext4/307
tests/ext4/308
tests/generic/018
tests/generic/324

-Eric
 
> Thanks,
> 
> On Mon, Sep 17, 2018 at 9:31 PM Eric Sandeen <sandeen@...hat.com> wrote:
>>
>> On 9/17/18 10:59 AM, Joshi wrote:
>>> Sorry if i gave the impression that i needed something in user-mode only. I am fine even if change involves modifying ext4 source.
>>
>> Perhaps you can describe what defrag behavior you are trying to test,
>> and we can offer some suggestions for good ways to construct that test?
>>
>> There are already a handful of defrag tests in xfstests, fwiw.
>>
>> Thanks,
>> -Eric
>>
>>> On Mon, 17 Sep 2018, 21:14 Eric Sandeen, <esandeen@...hat.com <mailto:esandeen@...hat.com>> wrote:
>>>
>>>     On 9/17/18 6:22 AM, Joshi wrote:
>>>     > I've been doing some defrag related tests, and for that I needed to be
>>>     > able to create file with same set of block numbers (i.e. extents), for
>>>     > at least two times.
>>>
>>>     Userspace cannot control that.
>>>
>>>     > May I know if there is any randomness in Ext4 allocator, and if there
>>>     > is any, can I disable it for the purpose of getting repeatable
>>>     > block-allocation patterns.
>>>     >
>>>     > Here are experiment details -
>>>     > For a 100K file (created using dd if=/dev/zero of=/mnt/f100k bs=4K
>>>     > count=100, oflag=direct) I got extent info in one run as this -
>>>     >
>>>     > File size of /mnt/file400k is 409600 (100 blocks of 4096 bytes)
>>>     >  ext:     logical_offset:        physical_offset: length:   expected: flags:
>>>     >    0:        0..      15:      34816..     34831:     16:
>>>     >    1:       16..      99:      33824..     33907:     84:      34832: last,eof
>>>     >
>>>     > while in the the second run I got somewhat different runs -
>>>     > File size of /mnt/file400k is 409600 (100 blocks of 4096 bytes)
>>>     >  ext:     logical_offset:        physical_offset: length:   expected: flags:
>>>     >    0:        0..       0:      34816..     34816:      1:
>>>     >    1:        1..      15:      34320..     34334:     15:      34817:
>>>     >    2:       16..      99:      33824..     33907:     84:      34335: last,eof
>>>     >
>>>     > Each run beings with a mkfs.ext4 with lazy inode/journal
>>>     > initialization disabled.
>>>
>>>     Userspace doesn't get to pick physical locations for allocations, you cannot
>>>     write a test which depends on doing so.
>>>
>>>     -Eric
>>>
>>
> 
>