| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aIokiNnO3ibUf3G5@li-dc0c254c-257c-11b2-a85c-98b6c1322444.ibm.com>
Date: Wed, 30 Jul 2025 19:26:44 +0530
From: Ojaswin Mujoo <ojaswin@...ux.ibm.com>
To: "Darrick J. Wong" <djwong@...nel.org>
Cc: Zorro Lang <zlang@...hat.com>, fstests@...r.kernel.org,
Ritesh Harjani <ritesh.list@...il.com>, john.g.garry@...cle.com,
tytso@....edu, linux-xfs@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-ext4@...r.kernel.org
Subject: Re: [PATCH v3 10/13] ext4/061: Atomic writes stress test for
bigalloc using fio crc verifier
On Tue, Jul 29, 2025 at 12:47:08PM -0700, Darrick J. Wong wrote:
> On Sat, Jul 12, 2025 at 07:42:52PM +0530, Ojaswin Mujoo wrote:
> > From: "Ritesh Harjani (IBM)" <ritesh.list@...il.com>
> >
> > We brute force all possible blocksize & clustersize combinations on
> > a bigalloc filesystem for stressing atomic write using fio data crc
> > verifier. We run nproc * $LOAD_FACTOR threads in parallel writing to
> > a single $SCRATCH_MNT/test-file. With atomic writes this test ensures
> > that we never see the mix of data contents from different threads on
> > a given bsrange.
Hi Darrick, thanks for the reviews.
>
> Err, how does this differ from the next patch? It looks like this one
> creates one IO thread, whereas the next one creates 8? If so, what does
> this test add over ext4/062?
Yes these 2 tests are similar however this one uses fallocate=native +
_scratch_mkfs_ext4 to test whether atomic writes on preallocated file
via multiple threads works correctly.
The other one uses fallocate=truncate + _scratch_mkfs_sized 360MB +
'multiple jobs each writing to a different file' to ensure we are
extensively stressing the allocation logic in low space scenarios.
>
> (and now that I look at it, ext4/062 says "FS QA Test 061"...)
Ahh I missed it somehow, thanks I'll fix it.
Regards,
ojaswin
>
> --D
>
> > Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@...il.com>
> > Signed-off-by: Ojaswin Mujoo <ojaswin@...ux.ibm.com>
> > ---
> > tests/ext4/061 | 130 +++++++++++++++++++++++++++++++++++++++++++++
> > tests/ext4/061.out | 2 +
> > 2 files changed, 132 insertions(+)
> > create mode 100755 tests/ext4/061
> > create mode 100644 tests/ext4/061.out
> >
> > diff --git a/tests/ext4/061 b/tests/ext4/061
> > new file mode 100755
> > index 00000000..a0e49249
> > --- /dev/null
> > +++ b/tests/ext4/061
> > @@ -0,0 +1,130 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2025 IBM Corporation. All Rights Reserved.
> > +#
> > +# FS QA Test 061
> > +#
> > +# Brute force all possible blocksize clustersize combination on a bigalloc
> > +# filesystem for stressing atomic write using fio data crc verifier. We run
> > +# nproc * 2 * $LOAD_FACTOR threads in parallel writing to a single
> > +# $SCRATCH_MNT/test-file. With fio aio-dio atomic write this test ensures that
> > +# we should never see the mix of data contents from different threads for any
> > +# given fio blocksize.
> > +#
> > +
> > +. ./common/preamble
> > +. ./common/atomicwrites
> > +
> > +_begin_fstest auto rw stress atomicwrites
> > +
> > +_require_scratch_write_atomic
> > +_require_aiodio
> > +
> > +FIO_LOAD=$(($(nproc) * 2 * LOAD_FACTOR))
> > +SIZE=$((100*1024*1024))
> > +fiobsize=4096
> > +
> > +# Calculate fsblocksize as per bdev atomic write units.
> > +bdev_awu_min=$(_get_atomic_write_unit_min $SCRATCH_DEV)
> > +bdev_awu_max=$(_get_atomic_write_unit_max $SCRATCH_DEV)
> > +fsblocksize=$(_max 4096 "$bdev_awu_min")
> > +
> > +function create_fio_configs()
> > +{
> > + create_fio_aw_config
> > + create_fio_verify_config
> > +}
> > +
> > +function create_fio_verify_config()
> > +{
> > +cat >$fio_verify_config <<EOF
> > + [aio-dio-aw-verify]
> > + direct=1
> > + ioengine=libaio
> > + rw=randwrite
> > + bs=$fiobsize
> > + fallocate=native
> > + filename=$SCRATCH_MNT/test-file
> > + size=$SIZE
> > + iodepth=$FIO_LOAD
> > + numjobs=$FIO_LOAD
> > + atomic=1
> > + group_reporting=1
> > +
> > + verify_only=1
> > + verify_state_save=0
> > + verify=crc32c
> > + verify_fatal=1
> > + verify_write_sequence=0
> > +EOF
> > +}
> > +
> > +function create_fio_aw_config()
> > +{
> > +cat >$fio_aw_config <<EOF
> > + [aio-dio-aw]
> > + direct=1
> > + ioengine=libaio
> > + rw=randwrite
> > + bs=$fiobsize
> > + fallocate=native
> > + filename=$SCRATCH_MNT/test-file
> > + size=$SIZE
> > + iodepth=$FIO_LOAD
> > + numjobs=$FIO_LOAD
> > + group_reporting=1
> > + atomic=1
> > +
> > + verify_state_save=0
> > + verify=crc32c
> > + do_verify=0
> > +
> > +EOF
> > +}
> > +
> > +# Let's create a sample fio config to check whether fio supports all options.
> > +fio_aw_config=$tmp.aw.fio
> > +fio_verify_config=$tmp.verify.fio
> > +fio_out=$tmp.fio.out
> > +
> > +create_fio_configs
> > +_require_fio $fio_aw_config
> > +
> > +for ((fsblocksize=$fsblocksize; fsblocksize <= $(_get_page_size); fsblocksize = $fsblocksize << 1)); do
> > + # cluster sizes above 16 x blocksize are experimental so avoid them
> > + # Also, cap cluster size at 128kb to keep it reasonable for large
> > + # blocks size
> > + fs_max_clustersize=$(_min $((16 * fsblocksize)) "$bdev_awu_max" $((128 * 1024)))
> > +
> > + for ((fsclustersize=$fsblocksize; fsclustersize <= $fs_max_clustersize; fsclustersize = $fsclustersize << 1)); do
> > + for ((fiobsize = $fsblocksize; fiobsize <= $fsclustersize; fiobsize = $fiobsize << 1)); do
> > + MKFS_OPTIONS="-O bigalloc -b $fsblocksize -C $fsclustersize"
> > + _scratch_mkfs_ext4 >> $seqres.full 2>&1 || continue
> > + if _try_scratch_mount >> $seqres.full 2>&1; then
> > + echo "== FIO test for fsblocksize=$fsblocksize fsclustersize=$fsclustersize fiobsize=$fiobsize ==" >> $seqres.full
> > +
> > + touch $SCRATCH_MNT/f1
> > + create_fio_configs
> > +
> > + cat $fio_aw_config >> $seqres.full
> > + echo >> $seqres.full
> > + cat $fio_verify_config >> $seqres.full
> > +
> > + $FIO_PROG $fio_aw_config >> $seqres.full
> > + ret1=$?
> > +
> > + $FIO_PROG $fio_verify_config >> $seqres.full
> > + ret2=$?
> > +
> > + _scratch_unmount
> > +
> > + [[ $ret1 -eq 0 && $ret2 -eq 0 ]] || _fail "fio with atomic write failed"
> > + fi
> > + done
> > + done
> > +done
> > +
> > +# success, all done
> > +echo Silence is golden
> > +status=0
> > +exit
> > diff --git a/tests/ext4/061.out b/tests/ext4/061.out
> > new file mode 100644
> > index 00000000..273be9e0
> > --- /dev/null
> > +++ b/tests/ext4/061.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 061
> > +Silence is golden
> > --
> > 2.49.0
> >
> >
Powered by blists - more mailing lists