linux-ext4 - Re: created ext4 disk image differs depending on the underlying filesystem

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <171540568260.2626447.10970955416649779876@localhost>
Date: Sat, 11 May 2024 07:34:42 +0200
From: Johannes Schauer Marin Rodrigues <josch@...ter-muffin.de>
To: Theodore Ts'o <tytso@....edu>
Cc: linux-ext4@...r.kernel.org
Subject: Re: created ext4 disk image differs depending on the underlying filesystem

Hi Ted,

thank you very much for your (as usual) very detailed and comprehensive reply!
:D

Quoting Theodore Ts'o (2024-05-05 02:10:20)
> If your goal is to create completely reproducible image files, one question
> is whether keeping the checksums identical is enough, or do you care about
> whether the underlying file is being more efficiently stored by using sparse
> files or extents marked unitialized?
> 
> Depending on how much you care about reproducibility versus file
> storage efficiency, I could imagine adding some kind of option which
> disables the zeroout function, and forces e2fsprogs to always write
> zeros, even if that increases the write wearout rate of the underlying
> flash file system, and increasing the size of the image file.  Or I
> could imageine some kind of extended option which hacks mke2fs to zero out
> the lifetime writes counter.;

the good news is, that the fix in my situation is very simple: create the
filesystem on a tmpfs first and then copy it into 9p fs afterwards. Tada, the
created images will be reproducible. I think there are multiple ways forward
with which I'd be happy with:

 1. leave everything as it is. It's just one more copy operation on my end.  I
    can just document that if your underlying file system is stupid, you might
    not get the same identical image as somebody with a more intelligent
    filesystem does.

 2. allow resetting fs->super->s_kbytes_written to zero. This patch worked for
    me:

--- a/lib/ext2fs/closefs.c
+++ b/lib/ext2fs/closefs.c
@@ -504,6 +504,7 @@ errcode_t ext2fs_close2(ext2_filsys fs, int flags)
                                (fs->blocksize / 1024);
                if ((fs->flags & EXT2_FLAG_DIRTY) == 0)
                        fs->flags |= EXT2_FLAG_SUPER_ONLY | EXT2_FLAG_DIRTY;
+               fs->super->s_kbytes_written = 0;
        }
        if (fs->flags & EXT2_FLAG_DIRTY) {
                retval = ext2fs_flush2(fs, flags);


    If my goal is to create disk images, one could argue that what the end user
    is interested in, is the filesystem writes that *they* performed and that
    the disk image they receive should therefor have the counter start at zero.

 3. Somehow do magic with the zeroout function. If anybody has too much
    free-time... ;)

As an end-user I am very interested in keeping the functionality of mke2fs
which keeps track of which parts are actually sparse and which ones are not.
This functionality can be used with tools like "bmaptool" (a more clever dd) to
only copy those parts of the image to the flash drive which are actually
supposed to contain data.

Would you be happy about a patch for (2.)? If yes, I can send something over
once I find some time. :)

Thanks!

cheers, josch
Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)