linux-ext4 - Re: [RFC][PATCH] fstest: regression test for ext4 crash consistency bug

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAOQ4uxgRafGec=HFq1strxC5NM0jwNgbu5ZyP1g+c64=b399+A@mail.gmail.com>
Date:   Thu, 31 Aug 2017 07:05:18 +0300
From:   Amir Goldstein <amir73il@...il.com>
To:     Ashlie Martinez <ashmrtn@...xas.edu>
Cc:     tytso@....edu, Eryu Guan <eguan@...hat.com>,
        Josef Bacik <jbacik@...com>,
        Vijay Chidambaram <vvijay03@...il.com>,
        fstests <fstests@...r.kernel.org>,
        Ext4 <linux-ext4@...r.kernel.org>
Subject: Re: [RFC][PATCH] fstest: regression test for ext4 crash consistency bug

On Thu, Aug 31, 2017 at 4:28 AM, Ashlie Martinez <ashmrtn@...xas.edu> wrote:
> Amir,
>
> I have been working on CrashMonkey more and I have jerry-rigged together a
> test in CrashMonkey that calls into `fsx` with the minimal test case you
> made. I am able to reproduce the ext4 error that you found along with a few
> other potential errors.
>
> A quick point, I run fsck with `-yf` instead of `-nf` that xfstests runs
> with. The reason for this is that CrashMonkey would like to report on
> fixable and unfixable errors in the future.
>

That makes sense, but keep in mind that 'fixable' error may still loose data
when fixing them with -y. Perhaps you should consider running fsck is auto
fixing mode (i.e. e2fsck -p) when available, to classify errors as
'safely fixable'
I believe the error these test encountered are 'safely fixable', but
didn't check.

> Running the ported test case, I find that CrashMonkey encounters the
> following errors:
> 1. Incorrect inode size and incorrect free data block and inode counts
> (fixable)
> 2. incorrect free data block and inode counts (fixable)
> 3. `Superblock needs_recovery flag is clear, but journal has data` notice
> along with errors present in case 1
> 4. `Superblock needs_recovery flag is clear, but journal has data` notice
> with no other errors
>
> For the incorrect i_size errors, I get the output `Inode 12, i_size is
> 147456, should be 163840.` which I can also reproduce with your 501 xfstests
> test case.
>
> When free data blocks and inode errors occur, the message is `Free blocks
> count wrong (8795, counted=8714).` and `Free inodes count wrong (2549,
> counted=2546).`
>
> I have not had a chance to look into the above errors to find their root
> causes.
>

I believe this is what you get when you fsck -yf before trying to mount when
the orphan list is not empty. You should avoid doing that.

See what the greatest ext4 crash test experiment of them all is doing
and read the comment to understand why:
https://android.googlesource.com/platform/system/core/+/marshmallow-mr1-dev/fs_mgr/fs_mgr.c#96
1. mount -o  errors=remount-ro; umount
2. e2fsck -y

So upstream Android never runs e2fsck -f. It will only check fs if kernel marked
that fs has errors.
Although Cyanogenmod did add -f and I imagine that many vendors do as well.

As one who hacked and crashed a lot of Android devices, I can attest that I have
observed both data loss and corrupted (non booting) fs, but the rest
of the 2 billion
crash test monkeys don't seem to be bothered ;-)

> In total, CrashMonkey ran 1000 different tests. Of those, 344 passed without
> fsck complaining. The remaining 656 tests saw fsck complain about something.
> All of these tests consisted of unique sequences of bios, but may contain
> equivalent crash states.
>
> The larger range of test results is due to the fact that CrashMonkey runs
> many tests from just the single workload you made. These tests consist of
> replaying some number of bio write operations, so it tests states different
> than you 500 xfstest which I believe only replays to sync operations (i.e.
> it never stops replay before a recorded fsync).

That is correct. test 500 (temporary name) is mostly focused on checking
data consistency of files after fsync. detecting metadata consistency errors
is a by product. I do intend to add more tests focused on metadata consistency.
Josef already wrote an fsstress script that should be converted to an xfstest
which replays the log to every FUA and fsck.

>
> If you're interested, you can find the CrashMonkey code (and branch) at
> https://github.com/utsaslab/crashmonkey/tree/ext4_regression_bug. If you
> would like to run it, you should clone and build you xfstest in your home
> directory so that the jerry-rigged CrashMonkey test case can find it.
> Directions for running this test case in CrashMonkey should be at the top of
> the README.

You seem to have misspelled 'fsx' in README and in the code as 'xfs'.
Funny, I always mistype it as 'sfx' :)

Cheers,
Amir.