linux-kernel - Re: [patch 0/6][RFC] Cleanup FIBMAP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 31 Oct 2007 07:06:16 -0400
From:	Ric Wheeler <ric@....com>
To:	Zach Brown <zab@...bo.net>
CC:	Mike Waychison <mikew@...gle.com>,
	Chris Mason <chris.mason@...cle.com>,
	Anton Altaparmakov <aia21@....ac.uk>,
	linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [patch 0/6][RFC] Cleanup FIBMAP

Zach Brown wrote:
>> Can you clarify what you mean above with an example?  I don't really
>> follow.
> 
> Sure, take 'tar' as an example.  It'll read files in the order that
> their names are returned from directory listing.  This can produce bad
> IO patterns because the order in which the file names are returned
> doesn't match the order of the file's blocks on disk.  (htree, I'm
> looking at you!)
> 
> People have noticed that tar-like loads can be sped up greatly just by
> sorting the files by their inode number as returned by stat(), never
> mind the file blocks themselves.  One example of this is Chris Mason's
> 'acp'.
> 
>   http://oss.oracle.com/~mason/acp/
> 
> The logical extension of that is to use FIBMAP to find the order of file
> blocks on disk and then doing IO on blocks in sorted order.  It'd take
> work to write an app that does this reliably, sure.
> 
> In this use the application doesn't actually care what the absolute
> numbers are.  It cares about their ordering.  File systems would be able
> to chose whatever scheme they wanted for the actual values of the
> results from a FIBMAP-alike as long as the sorting resulted in the right
> IO patterns.
> 
> Arguing that this use is significant enough to justify an addition to
> the file system API is a stretch.  I'm just sharing the observation.
> 
> - z

I use FIBMAP support for a few different things.

The first is to exactly the case that you describe above where we can 
use the first block of a file extracted by FIBMAP to produce an optimal 
sorting for the read order.  My testing showed that the cost of the 
extra fibmap was not too high compared to the speedup, but it was not a 
huge gain over the speedup gained when the read was done in inode sorted 
order.

The second use case is to look at the physical layout of blocks on disk 
for a specific file, use Mark Lord's write_long patches to inject a disk 
error and then read that file to make sure that we are handling disk IO 
errors correctly.  A bit obscure, but really quite useful.

We have also used FIBMAP a few times to try and map an observed IO error 
back to a file. Really slow and painful to do, but should work on any 
file system when a better method is not supported.

ric
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/