linux-ext4 - Re: Some thoughts about providing data block checksumming for ext4

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141127003923.GQ10043@birch.djwong.org>
Date:	Wed, 26 Nov 2014 16:39:23 -0800
From:	"Darrick J. Wong" <darrick.wong@...cle.com>
To:	Mike Snitzer <snitzer@...hat.com>
Cc:	"Theodore Ts'o" <tytso@....edu>, dm-devel@...hat.com,
	linux-ext4@...r.kernel.org
Subject: Re: Some thoughts about providing data block checksumming for ext4

On Wed, Nov 26, 2014 at 07:07:22PM -0500, Mike Snitzer wrote:
> On Wed, Nov 26 2014 at  6:47pm -0500,
> Darrick J. Wong <darrick.wong@...cle.com> wrote:
> 
> > Sigh...
> > 
> > Well, I wrote up a preliminary version of dm-checksum and then
> > realized that I've pretty much just built a crappier version of
> > dm-dedupe, but without the dedupe part.  Given that it stores
> > checksums in a btree which claims to be robust through failures and
> > gives us automatic deduplication, I wonder if it we could achieve our
> > aims by modifying dm-dedupe to verify the checksums on the read path?
> > 
> > I guess it would be interesting to see how bad the performance hit is
> > with the online dedupe part enabled or disabled.  dm-dedupe v2 went
> > out on the mailing list last August, which I missed. :(
> > 
> > Unless... there's a specific reason nobody mentioned dm-dedupe here?
> 
> As you may have seen in the dm-dedup thread, we need to actively
> review/test that target

It was in fact today's exchange on that thread that made me slap myself on
the forehead and utter "D'oh!".

> (if your initial review focus is on extending it
> to _optionally_ verify the checksums on the read path then so be it).

Yes, sorry, I meant to say "optionally to verify" in there.  Adding a minor
feature like that might be a good check to make sure I actually understand
what's going on. :)

> See: https://www.redhat.com/archives/dm-devel/2014-November/msg00114.html
> Specifically, the git branch that builds on v2 based on my initial
> review of v2:
> 
> git://git.fsl.cs.stonybrook.edu/scm/git/linux-dmdedup
> branch: dm-dedup-devel
> 
> Your help on getting dm-dedup upstream would be very much appreciated.

<-- reading the OLS paper, as a start.  What happens to the metadata btree if
someone sets the chunk size to 4KB?  Will it become ungainly huge?  The thing
that I wrote simply wrote a block's worth of checksums inline with the data,
which required a certain amount of slicing and dicing of bios but wasn't too
horrible with performance.

--D

> 
> Thanks,
> Mike
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html