linux-ext4 - Re: Repeatable block allocation problem.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Tue, 18 Sep 2018 20:41:54 +0530
From:   Joshi <joshiiitr@...il.com>
To:     sandeen@...hat.com
Cc:     linux-ext4@...r.kernel.org
Subject: Re: Repeatable block allocation problem.

> > End objective of above was to create fragmented file, with repeatable
> > extent-count/block-numbers.
>
> I understand that you want repeatable allocation behavior.
>
> I still don't know exactly what behavior you are ultimately trying to
> test, though.
Defrag without doing data-copy, as described in this paper -
https://www.usenix.org/system/files/conference/atc17/atc17-hahn.pdf


On Mon, Sep 17, 2018 at 10:58 PM Eric Sandeen <sandeen@...hat.com> wrote:
>
> On 9/17/18 11:40 AM, Joshi wrote:
> > It's not that I need specific values for extents/blocks for a file. It
> > is just about being able to create/prepare a file of given size with
> > same (repeatable) number of extents and block numbers, in a controlled
> > environment.
>
>
> > I thought it would happen naturally, if I keep preconditions same
> > (i.e. post format with lazy initialization disabled. I had not
> > disabled lazy-allocation though).
> > But extent count differed, at times.
> > And I wondered whether Ext4 code makes any conscious attempt to
> > randomize the extent/block allocation?
>
> find_group_orlov() randomly selects block groups for root directories,
> but AFAIK there is no explicit random behavior in the block allocator.
>
> It's more of a chaotic system than intentional random selection.  :)
>
> > End objective of above was to create fragmented file, with repeatable
> > extent-count/block-numbers.
>
> I understand that you want repeatable allocation behavior.
>
> I still don't know exactly what behavior you are ultimately trying to
> test, though.
>
> > I'll check xfstests as well.
>
> The tests marked as being in the defrag group are:
>
> tests/btrfs/005
> tests/btrfs/021
> tests/btrfs/062
> tests/btrfs/067
> tests/btrfs/070
> tests/btrfs/072
> tests/btrfs/074
> tests/ext4/020
> tests/ext4/301
> tests/ext4/302
> tests/ext4/303
> tests/ext4/304
> tests/ext4/307
> tests/ext4/308
> tests/generic/018
> tests/generic/324
>
> -Eric
>
> > Thanks,
> >
> > On Mon, Sep 17, 2018 at 9:31 PM Eric Sandeen <sandeen@...hat.com> wrote:
> >>
> >> On 9/17/18 10:59 AM, Joshi wrote:
> >>> Sorry if i gave the impression that i needed something in user-mode only. I am fine even if change involves modifying ext4 source.
> >>
> >> Perhaps you can describe what defrag behavior you are trying to test,
> >> and we can offer some suggestions for good ways to construct that test?
> >>
> >> There are already a handful of defrag tests in xfstests, fwiw.
> >>
> >> Thanks,
> >> -Eric
> >>
> >>> On Mon, 17 Sep 2018, 21:14 Eric Sandeen, <esandeen@...hat.com <mailto:esandeen@...hat.com>> wrote:
> >>>
> >>>     On 9/17/18 6:22 AM, Joshi wrote:
> >>>     > I've been doing some defrag related tests, and for that I needed to be
> >>>     > able to create file with same set of block numbers (i.e. extents), for
> >>>     > at least two times.
> >>>
> >>>     Userspace cannot control that.
> >>>
> >>>     > May I know if there is any randomness in Ext4 allocator, and if there
> >>>     > is any, can I disable it for the purpose of getting repeatable
> >>>     > block-allocation patterns.
> >>>     >
> >>>     > Here are experiment details -
> >>>     > For a 100K file (created using dd if=/dev/zero of=/mnt/f100k bs=4K
> >>>     > count=100, oflag=direct) I got extent info in one run as this -
> >>>     >
> >>>     > File size of /mnt/file400k is 409600 (100 blocks of 4096 bytes)
> >>>     >  ext:     logical_offset:        physical_offset: length:   expected: flags:
> >>>     >    0:        0..      15:      34816..     34831:     16:
> >>>     >    1:       16..      99:      33824..     33907:     84:      34832: last,eof
> >>>     >
> >>>     > while in the the second run I got somewhat different runs -
> >>>     > File size of /mnt/file400k is 409600 (100 blocks of 4096 bytes)
> >>>     >  ext:     logical_offset:        physical_offset: length:   expected: flags:
> >>>     >    0:        0..       0:      34816..     34816:      1:
> >>>     >    1:        1..      15:      34320..     34334:     15:      34817:
> >>>     >    2:       16..      99:      33824..     33907:     84:      34335: last,eof
> >>>     >
> >>>     > Each run beings with a mkfs.ext4 with lazy inode/journal
> >>>     > initialization disabled.
> >>>
> >>>     Userspace doesn't get to pick physical locations for allocations, you cannot
> >>>     write a test which depends on doing so.
> >>>
> >>>     -Eric
> >>>
> >>
> >
> >
>


-- 
Joshi