linux-ext4 - Re: [PATCH] ext4: Add support for online resizing with bigalloc.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMCMpYJfJECSUOr_bp+k3o=jH18ShkHcOT9fK0o=FaHZkwUtZA@mail.gmail.com>
Date:   Mon, 30 Oct 2017 14:01:46 -0700
From:   Harshad Shirwadkar <harshads@...gle.com>
To:     "Theodore Ts'o" <tytso@....edu>
Cc:     linux-ext4@...r.kernel.org
Subject: Re: [PATCH] ext4: Add support for online resizing with bigalloc.

On Sun, Oct 29, 2017 at 8:06 AM, Theodore Ts'o <tytso@....edu> wrote:
> (If you work for a company which supports enterprise distros, please
> keep reading.  I have a question for you below.)
>
> On Tue, Oct 24, 2017 at 09:42:08PM -0700, harshads wrote:
>> This patch adds support for online resizing on bigalloc file system by
>> implementing EXT4_IOC_RESIZE_FS ioctl. Old resize interfaces (add
>> block groups and extend last block group) are left untouched. Tests
>> performed with cluster sizes of 1, 2, 4 and 8 blocks (of size 4k) per
>> cluster. I will add these tests to xfstests.
>>
>> Signed-off-by: Harshad Shirwadkar <harshads@...gle.com>
>
> Can you talk more about what sort of testing you have done?  What file
> system sizes did you use?

In my testing, I have tried to ensure that all different code paths
that resize IOCTL triggers get executed. At a high level, these
different code paths are - 1) extending last block group, 2) adding
new block groups 3) conversion of file system to meta-bg.
I performed tests with combinations of different cluster sizes (1, 2,
4, 8 block(s) per cluster) and different file system sizes. Following
csv table lists all the tests that were performed and were successful
with new resize IOCTL.

Cluster size (# 4K blocks), Original Size(# clusters), Final Size(# clusters)
1, 16384(64M), 24576(96M)
1, 24576(96M), 32767(128M)
1, 24576(96M), 32768(128M)
1, 24576(96M), 32769(128M)
1, 24576(96M), 49152(192M)
1, 24576(96M), 65536(256M)
1, 24576(96M), 491521(1.9G)
1, 24576(96M), 507904(~1.9G)
1, 24576(96M), 527488(2G)
1, 24576(96M), 5274880(20G)
2, 16384(128M), 24576(192M)
2, 24576(192M), 32767(256M)
2, 24576(192M), 32768(256M)
2, 24576(192M), 32769(256M)
2, 24576(192M), 49152(384M)
2, 24576(192M), 65536(512M)
2, 24576(192M), 491521(3.8G)
2, 24576(192M), 507904(3.8G)
2, 24576(192M), 527488(4G)
2, 24576(192M), 5274880(40G)
4, 16384(256M), 24576(384M)
4, 24576(384M), 32767(512M)
4, 24576(384M), 32768(512M)
4, 24576(384M), 32769(512M)
4, 24576(384M), 49152(768M)
4, 24576(384M), 65536(1G)
4, 24576(384M), 491521(7.6G)
4, 24576(384M), 507904(7.6G)
4, 24576(384M), 527488(8G)
4, 24576(384M), 5274880(80G)
8, 16384(512M), 24576(768M)
8, 24576(768M), 32767(1G)
8, 24576(768M), 32768(1G)
8, 24576(768M), 32769(1G)
8, 24576(768M), 49152(1.5G)
8, 24576(768M), 65536(2G)
8, 24576(768M), 491521(15.2G)
8, 24576(768M), 507904(15.2G)
8, 24576(768M), 527488(16G)
8, 24576(768M), 5274880(160G)

>
> I'm guessing you didn't test the old resize ioctls, because they are
> currently broken.  I did a quick test because the code paths that are
> touched by this patch are also used by the old resize ioctls, and the
> following is failing:

Yes, you are right. Also in the latest kernel, there are code-paths
that are present only for the old resize IOCTLs and I have not touch
them yet. Let's see what others have to say and then I can fix those
if required.

Thanks,
Harshad.

>
> mke2fs -t ext4 -Fq /dev/vdc 1G
> mount /dev/vdc
> export RESIZE2FS_KERNEL_VERSION=3.2.0
> strace -o /tmp/st-$RESIZE2FS_KERNEL_VERSION resize2fs /dev/vdc 5G
> umount /dev/vdc
>
> From a quick check, and it looks like this broke sometime between 4.1
> and 4.4.  It works on 3.18 and 4.1, and but it failed for me in 4.4.
>
> The 4.1 kernel was released in June 2015, and 4.4 kernel was released
> in January 2016.  E2fsprogs v1.42 started using the new resize ioctl,
> which was released in November 2011.  So if we take an enterprise
> distro which was initially released > 5-6 years ago, and it upgrades
> to a kernel newer than 4.1-4.4, the resize2fs from e2fsprogs 1.41.x
> (or earlier) will not be able to do online resize.
>
> The question is whether or not we care at this point.  We could use
> the observation that our lack of good testing, sometime in second half
> of 2015, we broke online resize backwards compatibility with e2fsprogs
> from before 2011, given that no one screamed --- perhaps we can just
> rip out support for the old resize ioctls entirely.
>
> Or we can try to fix the old resize ioctl's, and then add better
> resize testing to xfstests.
>
> What do the maintainers from the enterprise distro's think?
>
>                                                 - Ted