linux-ext4 - Re: Make huge files strictly contiguous (fallocate, bigalloc, e4defrag...)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.LFD.2.00.1404181042200.2128@localhost.localdomain>
Date:	Fri, 18 Apr 2014 10:45:03 +0200 (CEST)
From:	Lukáš Czerner <lczerner@...hat.com>
To:	Contact <neitsab@....fr>
cc:	"Theodore Ts'o" <tytso@....edu>,
	"linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
Subject: Re: Make huge files strictly contiguous (fallocate, bigalloc,
 e4defrag...)

On Thu, 17 Apr 2014, Contact wrote:

> Date: Thu, 17 Apr 2014 20:41:34 +0200
> From: Contact <neitsab@....fr>
> To: Theodore Ts'o <tytso@....edu>, lczerner@...hat.com
> Cc: "linux-ext4@...r.kernel.org" <linux-ext4@...r.kernel.org>
> Subject: Re: Make huge files strictly contiguous (fallocate, bigalloc,
>     e4defrag...)
> 
> Le 17/04/2014 14:04, Lukáš Czerner a écrit :
> >
> > This is not how it it supposed to be used. Yes fallocate
> > preallocates the file, but cp will truncate it so fallocate will
> > certainly not help you in any way. In order for fallocate to be
> > useful you'll have to write into the file without actually
> > truncating it (dd can do this if you do not want to write your own
> > program)
> >
> > Also the file is probably as contiguous as it could be. Here is what
> > I get on the file system with default mkfs options.
> >
> > # e4defrag -c /mnt/test/file1
> > <File>                                         now/best      size/ext
> > /mnt/test/file1                                 10/1         120649 KB
> >
> > But that does not tell the whole story. See
> >
> > xfs_io -f -c "fiemap -v" /mnt/test/file1
> > /mnt/test/file1:
> >  EXT: FILE-OFFSET        BLOCK-RANGE       TOTAL FLAGS
> >    0: [0..262143]:       2768896..3031039 262144   0x0
> >    1: [262144..524287]:  3031040..3293183 262144   0x0
> >    2: [524288..786431]:  3293184..3555327 262144   0x0
> >    3: [786432..1048575]: 3555328..3817471 262144   0x0
> >    4: [1048576..1310719]: 3817472..4079615 262144   0x0
> >    5: [1310720..1425407]: 4079616..4194303 114688   0x0
> >    6: [1425408..1687551]: 4456448..4718591 262144   0x0
> >    7: [1687552..1949695]: 4718592..4980735 262144   0x0
> >    8: [1949696..2211839]: 4980736..5242879 262144   0x0
> >    9: [2211840..2412991]: 5242880..5444031 201152   0x1
> >
> > (Note that the output is in 512B blocks)
> >
> > As you can see the file is mostly contiguous, but it is divided into
> > several extents because of two reasons.
> >
> > 1. The extent in ext4 has a limited size of 32768 blocks for
> > initialized extent and 32767 block for unwritten extent. So when we
> > exceed that size we need another extent which might be physically
> > contiguous on disk with the previous one.
> >
> > 2. Ext4 divides disk space into allocation groups of certain size
> > (cluster size * 8)blocks. Now with flex_bg medatada such as inode
> > tables, block bitmaps and so one are packed closely together so the
> > do not have to be stored with each block group and you'll get more
> > contiguous data space.
> >
> > However we're still storing backup superblock and Groups descriptors
> > in certain groups and those are the gaps you're seeing in the fiemap
> > list.
> >
> > For detailed overview you can use dumpe2fs to see what is allocated
> > where on the file system.
> >
> 
> Thanks, that was much interesting. I had delved a bit into ext4 data
> structure before posting but was never able to get a clear grasp on the
> limitations concerning contiguity.
> 
> So, I tried the correct way you recommended about how to use fallocate:
> 
> # mkfs.ext4 -m 0 -L iso -i 67108864 -E root_owner=1000:100 /dev/sdc2
> $ fallocate -l 1589166080
> '/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'
> $ dd if='/home/neitsab/iso/Manjaro/manjaro-gnome-0.8.9-x86_64.iso'
> of='/run/media/neitsab/iso/_ISO/manjaro-gnome-0.8.9-x86_64.iso'

The way you're using dd for this purpose is wrong. I
said that dd can write to the file without truncating it first, but
it's not doing it by default. You have to use 'notrunc' see man
page. But I think that it would not have made a difference anyway.

-Lukas