linux-kernel - Re: [ANNOUNCE] Btrfs: a copy on write, snapshotting FS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20070619120457.GD14108@think.oraclecorp.com>
Date:	Tue, 19 Jun 2007 08:04:57 -0400
From:	Chris Mason <chris.mason@...cle.com>
To:	Pádraig Brady <P@...igBrady.com>
Cc:	Vladislav Bolkhovitin <vst@...b.net>, linux-kernel@...r.kernel.org,
	linux-fsdevel@...r.kernel.org
Subject: Re: [ANNOUNCE] Btrfs: a copy on write, snapshotting FS

On Tue, Jun 19, 2007 at 10:11:13AM +0100, Pádraig Brady wrote:
> Vladislav Bolkhovitin wrote:
> > 
> > I would also suggest one more feature: support for block level
> > de-duplication. I mean:
> > 
> > 1. Ability for Btrfs to have blocks in several files to point to the
> > same block on disk
> > 
> > 2. Support for new syscall or IOCTL to de-duplicate as a single
> > transaction two or more blocks on disk, i.e. link them to one of them
> > and free others
> > 
> > 3. De-de-duplicate blocks on disk, i.e. copy them on write
> > 
> > I suppose that de-duplication itself would be done by some user space
> > process that would scan files, determine blocks with the same data and
> > then de-duplicate them by using syscall or IOCTL (2).
> > 
> > That would be very usable feature, which in most cases would allow to
> > shrink occupied disk space on 50-90%.
> 
> Have you references for this number?
> In my experience one gets a lot of benefit from
> the much simpler process of "de-duplication" of files.

Yes, I would expect simple hard links to be a better solution for this,
but the feature request is not that out of line.  I actually had plans
on implementing auto duplicate block reuse earlier in btrfs.

Snapshots already share duplicate blocks between files, and so all of
the reference counting needed to implement this already exists.
Snapshots are writable, and data mods are copy on write, and in general
things work.

But, to help fsck, the extent allocation tree has a back pointer to the
inode that owns an extent.  If you're doing snapshots, all of the owners
of the extent have the same inode number.   If you're sharing duplicate
blocks, the owners can have any inode number, and fsck becomes much more
complex.

In general, when I have to decide between fsck and a feature, I'm going
to pick fsck.  The features are much more fun, but fsck is one of the
main motivations for doing this work.

Thanks for the input,
Chris
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/