linux-ext4 - Re: [PATCH] fstests: generic, fsync fuzz tester with fsstress

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20190517034249.GM15846@desktop>
Date:   Fri, 17 May 2019 11:42:49 +0800
From:   Eryu Guan <guaneryu@...il.com>
To:     fdmanana@...nel.org
Cc:     fstests@...r.kernel.org, linux-btrfs@...r.kernel.org,
        linux-ext4@...r.kernel.org, jack@...e.cz,
        Filipe Manana <fdmanana@...e.com>
Subject: Re: [PATCH] fstests: generic, fsync fuzz tester with fsstress

On Wed, May 15, 2019 at 04:02:21PM +0100, fdmanana@...nel.org wrote:
> From: Filipe Manana <fdmanana@...e.com>
> 
> Run fsstress, fsync every file and directory, simulate a power failure and
> then verify the all files and directories exist, with the same data and
> metadata they had before the power failure.
> 
> This tes has found already 2 bugs in btrfs, that caused mtime and ctime of
> directories not being preserved after replaying the log/journal and loss
> of a directory's attributes (such a UID and GID) after replaying the log.
> The patches that fix the btrfs issues are titled:
> 
>   "Btrfs: fix wrong ctime and mtime of a directory after log replay"
>   "Btrfs: fix fsync not persisting changed attributes of a directory"
> 
> Running this test 1000 times:
> 
> - on xfs, no issues were found
> 
> - on ext4 it has resulted in about a dozen journal checksum errors (on a
>   5.0 kernel) that resulted in failure to mount the filesystem after the
>   simulated power failure with dmflakey, which produces the following
>   error in dmesg/syslog:
> 
>     [Mon May 13 12:51:37 2019] JBD2: journal checksum error
>     [Mon May 13 12:51:37 2019] EXT4-fs (dm-0): error loading journal
> 
> Signed-off-by: Filipe Manana <fdmanana@...e.com>
> ---
>  tests/generic/547     | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/547.out |  2 ++
>  tests/generic/group   |  1 +
>  3 files changed, 75 insertions(+)
>  create mode 100755 tests/generic/547
>  create mode 100644 tests/generic/547.out
> 
> diff --git a/tests/generic/547 b/tests/generic/547
> new file mode 100755
> index 00000000..577b0e9b
> --- /dev/null
> +++ b/tests/generic/547
> @@ -0,0 +1,72 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2019 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test No. 547
> +#
> +# Run fsstress, fsync every file and directory, simulate a power failure and
> +# then verify the all files and directories exist, with the same data and
> +# metadata they had before the power failure.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +	_cleanup_flakey
> +	cd /
> +	rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/dmflakey
> +
> +# real QA test starts here
> +_supported_fs generic
> +_supported_os Linux
> +_require_scratch

As we save fssum file to $TEST_DIR, it'd be better to _require_test too.

> +_require_fssum
> +_require_dm_target flakey
> +
> +rm -f $seqres.full
> +
> +fssum_files_dir=$TEST_DIR/generic-test-$seq
> +rm -fr $fssum_files_dir
> +mkdir $fssum_files_dir
> +
> +_scratch_mkfs >>$seqres.full 2>&1
> +_require_metadata_journaling $SCRATCH_DEV
> +_init_flakey
> +_mount_flakey
> +
> +mkdir $SCRATCH_MNT/test
> +args=`_scale_fsstress_args -p 4 -n 100 $FSSTRESS_AVOID -d $SCRATCH_MNT/test`
> +args="$args -f mknod=0 -f symlink=0"
> +echo "Running fsstress with arguments: $args" >>$seqres.full
> +$FSSTRESS_PROG $args >>$seqres.full
> +
> +# Fsync every file and directory.
> +find $SCRATCH_MNT/test -type f,d -exec $XFS_IO_PROG -c "fsync" {} \;

My 'find' on Fedora 29 vm (find (GNU findutils) 4.6.0) doesn't support
"-type f,d" syntax

find: Arguments to -type should contain only one letter

I have to change this to

find $SCRATCH_MNT/test \( -type f -o -type d \) -exec $XFS_IO_PROG -c "fsync" {} \;

Otherwise looks good to me, thanks!

Eryu

> +# Compute a digest of the filesystem (using the test directory only, to skip
> +# fs specific directories such as "lost+found" on ext4 for example).
> +$FSSUM_PROG -A -f -w $fssum_files_dir/fs_digest $SCRATCH_MNT/test
> +
> +# Simulate a power failure and mount the filesystem to check that all files and
> +# directories exist and have all data and metadata preserved.
> +_flakey_drop_and_remount
> +
> +# Compute a new digest and compare it to the one we created previously, they
> +# must match.
> +$FSSUM_PROG -r $fssum_files_dir/fs_digest $SCRATCH_MNT/test
> +
> +_unmount_flakey
> +
> +status=0
> +exit
> diff --git a/tests/generic/547.out b/tests/generic/547.out
> new file mode 100644
> index 00000000..0f6f1131
> --- /dev/null
> +++ b/tests/generic/547.out
> @@ -0,0 +1,2 @@
> +QA output created by 547
> +OK
> diff --git a/tests/generic/group b/tests/generic/group
> index 47e81d96..49639fc9 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -549,3 +549,4 @@
>  544 auto quick clone
>  545 auto quick cap
>  546 auto quick clone enospc log
> +547 auto quick log
> -- 
> 2.11.0
>