linux-ext4 - Re: [PATCH] e2fsck: do not allow initialized blocks pass i

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <7AAE0F83-C9CE-4B48-A6A5-1A21BAD26AEE@dilger.ca>
Date:   Fri, 29 Jun 2018 13:35:41 -0600
From:   Andreas Dilger <adilger@...ger.ca>
To:     Lukas Czerner <lczerner@...hat.com>
Cc:     "Theodore Y. Ts'o" <tytso@....EDU>, linux-ext4@...r.kernel.org
Subject: Re: [PATCH] e2fsck: do not allow initialized blocks pass i_size

On Jun 29, 2018, at 2:46 AM, Lukas Czerner <lczerner@...hat.com> wrote:
> 
> On Thu, Jun 28, 2018 at 09:05:04PM -0600, Andreas Dilger wrote:
>> 
>>> On Jun 28, 2018, at 7:45 PM, Theodore Y. Ts'o <tytso@....EDU> wrote:
>>> 
>>> On Thu, Jun 28, 2018 at 06:07:47PM -0600, Andreas Dilger wrote:
>>>>> But does an RDMA operation actually do a block allocation?  Really?
>>>>> And if it is willing to do a block allocation, why is it not willing
>>>>> to bump i_size?
>>>> 
>>>> It's not that the RDMA does block allocation, but rather that the RDMA
>>>> always transfers and writes the full PAGE_SIZE of data, even if i_size
>>>> is less than the end of the last block.  This simplifies the RDMA code
>>>> so that it can always write the data instead of having to stop at i_size.
> 
> Every time we write past i_size we need to extend the i_size, why is
> this case different ?
> 
>>> 
>>> Right.  So there are two choices:
>>> 
>>> 1) Keep the blocks beyond i_size marked as uninitialized.  You
>>> transfer and write the full PAGE_SIZE of data, but it simply will
>>> never be available to the user.
> 
> Yes, that's for extent mapped files.
> 
>>> 
>>> 2)  Zero the page, write it out to the file, and then extend i_size and
>>> mark the extents as uninitialized.
> 
> Except at that point you do not really need to mark the extent as
> unitialized, the blocks are allocated and written to and i_size is
> extended. That's how it needs to be done for indirect block mapped
> files.
> 
>> 
>> The end of the page would already be zeroed before write.
>> 
>>> Why is it that Lustre is choosing to keep i_size where it is, but to
>>> mark the blocks beyond it as initialized?
>> 
>> This isn't about initialized vs. uninitialized extents.  It is only about
>> allocated vs. unallocated blocks, possibly with block-mapped files.  There
>> is no way to have uninitialized blocks with a block-mapped file.
>> 
>> The code is checking whether there are any blocks allocated beyond i_size,
>> and if there are, without the patch it considers i_size broken and extends
>> it to the end of the last allocated block.  The patch allows a small number
>> of blocks to be allocated beyond i_size without triggering this heuristic.
> 
> I do not think that ext4 has any capacity to allocate initialized blocks
> beyond i_size without actually writing to it and if we write beyong i_size
> then we need to extend i_size so if we see anything like that it's a problem
> that needs fixing. I think it's as simple as that.
> 
> Unless you can show me where in upstream ext4 we can get to this
> situation I am strongly for fixing e2fsck. But maybe I am missing
> something.

Like I said previously, this is done with Lustre, which has a different IO
submission path than stock ext4.  I don't think there is any requirement that
this only be in upstream ext4, since e2fsprogs also has code to support running
on BSD, Windows, even Hurd.

>> The only difference vs. the previous code is that it correctly calculates
>> what the PAGE_SIZE aligned block number is (the old code assumed that lblock
>> was the base-1 block number instead of the base-0 block number that it is).
> 
> Right, but the test is broken when PAGE_SIZE > 4k that's how I found
> out in the first place. See me previous email about f_eofblocks and
> f_pgsize_gt_blksize failures.

I already have a patch for the f_pgsize_gt_blksize failure that should work
with both 4KB and 64KB (and other) PAGE_SIZE systems, since it is generating
the filesystem itself.  I probably need to make a script for f_eofblocks so
that it skips the test on PAGE_SIZE=64KB or uses a different expect.1 file.

Cheers, Andreas






Download attachment "signature.asc" of type "application/pgp-signature" (874 bytes)