linux-ext4 - Re: [RFC] Self healing extent map OR opportunistic de-fragmentation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <AANLkTi=n5HtRQY2VR9Kz7tyZQ7=PW77Ar8gj40F1aXB_@mail.gmail.com>
Date:	Tue, 9 Nov 2010 11:35:07 +0200
From:	Amir Goldstein <amir73il@...il.com>
To:	Andreas Dilger <adilger.kernel@...ger.ca>
Cc:	Theodore Tso <tytso@....edu>,
	Ext4 Developers List <linux-ext4@...r.kernel.org>
Subject: Re: [RFC] Self healing extent map OR opportunistic de-fragmentation

On Tue, Nov 9, 2010 at 10:58 AM, Andreas Dilger
<adilger.kernel@...ger.ca> wrote:
> On 2010-11-08, at 21:14, Amir Goldstein wrote:
>> I would like to propose a simple idea how to automatically de-fragment a file.
>
> [snip]
>
>> The use case for this, besides healing fragmentation caused by
>> snapshots move-on-rewrite, is an highly fragmented ext2/3 fs, which was mounted as ext4.
>> ext2/3 old files are slowly being deleted while new (still fragmented) extent mapped files are being created.
>> This viscous cycle cannot end before there is enough contiguous free
>> space for writing new files, which may never happen.
>
> This will only happen in case the free space is _very_ low.  Normally, in a situation like this, mballoc will allocate the
> largest contiguous chunks of free space, reducing the fragmentation as new files are written, and allocations to
> highly-fragmented block groups will be avoided until the chunks in those groups have grown larger.
>
>> Online de-fragmentation will not help in this case either.
>> With opportunistic de-fragmentation, if the extent mapped files are
>> being re-written, the health of the file system will constantly improve over time.
>> BTW, Is this use case relevant for upgraded google chunk servers?
>
> While this is true in theory, the problem is that in most cases files are not overwritten in place.
> Commonly, when files are "rewritten" they are truncated and new blocks allocated, or a new file is written and renamed in place of the old file.
> Only in rare cases, like databases, are files rewritten in-place.
>

Oh, I know that. Which is why it is a bit annoying to invest a lot of
effort to solve the fragmentation caused by snapshot move-on-rewrite.
In those rare use cases, the file may end up like a swiss cheese after
a while. However, I realized that the access pattern of those
applications
in not all that random. A rewrite to offset X has a high likelihood to
repeat more than once (update to DB record or write of metadata block
in virtual disk).
I figured I could use the opportunity of the subsequent rewrites to
restore the file blocks to their original location, without paying a
performance trade-off.

The question is, are there other use cases out there that can benefit
from opportunistic de-fragmentation?

Amir.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html