linux-kernel - Re: [PATCH] xfs: getattr ignore blocks beyond eof

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20220331053340.GE1544202@dread.disaster.area>
Date:   Thu, 31 Mar 2022 16:33:40 +1100
From:   Dave Chinner <david@...morbit.com>
To:     wang.yi59@....com.cn
Cc:     djwong@...nel.org, linux-xfs@...r.kernel.org,
        linux-kernel@...r.kernel.org, xue.zhihong@....com.cn,
        wang.liang82@....com.cn, cheng.lin130@....com.cn
Subject: Re: [PATCH] xfs: getattr ignore blocks beyond eof

On Thu, Mar 31, 2022 at 11:28:59AM +0800, wang.yi59@....com.cn wrote:
> >> From: Cheng Lin <cheng.lin130@....com.cn>
> >>
> >> Blocks beyond EOF, which preallocated, will be reclaimed at some time.
> >> These blocks can be ignored when getattr.
> >>
> >> This patch will optimize query accuracy for getattr blocks.
> >Huh? This subtracts posteof blocks from the query results, which makes
> >the results *less accurate*. Those blocks are mapped to the file, hence
> >they are supposed to be counted in nblocks.
> >--D
> Yes, those blocks are mapped to the file. And the results including them are
> absolutely real for xfs, at the moment of query.
> 
> But, those blocks are like the credit without consumption, are unstalbe, and
> will be reclaimed at some time. This may cause trouble for the application.

What application is having trouble with this?

> e.g. in a case,
> 1. Firstly, write 100k data to file;
> 2. query the result;
> 3. close the file;
> 4. query the result.
> 
> fd stat wrt[96 @ 393216]: blks[896], size[397312].
> fd stat wrt[97 @ 397312]: blks[896], size[401408].
> fd stat wrt[98 @ 401408]: blks[896], size[405504].
> fd stat wrt[99 @ 405504]: blks[896], size[409600].
> lstat open: blks[896], size[409600].
> lstat close: blks[800], size[409600].
> 
> Here two problems:
> 1. why the result different before between after file close?
> 2. why the result not change after writing data, or a big change?

Because that's the way speculative preallocation works.

> The above problems can be explained by fs preallocation.
> If the impact of preallocation are closed in fs, not visible to
> the outside, the result is stable and real for the application.

You're a decade late to the party. Complaining about how visible
artifacts of speculative preallocation are going to cause the sky
to fall was something that happened when we fix implemented the
mechanism.

Once people realised how much better their filesystems resisted
fragmentation and hence aged more gracefully because the temporary
overallocation helped ensure large, contiguous extents got allocated
more often than not, those complaints went away.

Have you even considered what reflink/dedupe does to user visible
block accounting? That's right - it's a complete lie.

We do not, and have not ever tried to, hide allocation or block
usage artifacts from userspace because any application that depends
on specific block allocation patterns or accounting from the
filesystem is broken by design.

Every filesystem accounts blocks differently, and more often than
not the block count exposed to userspace also includes metadata
blocks (extent maps, xattr blocks, etc) and it might multiple count
other blocks (e.g. shared extents). Hence so you can't actually
use it for anything useful in userspace except reporting how many
blocks this file *might* use.

If your application is dependent on block counts exactly matching
the file data space for waht ever reason, then what speculative
preallocation does is the least of your problems.

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com