Message-ID: <87frjs6t23.fsf@gmail.com>
Date: Wed, 05 Mar 2025 09:13:32 +0530
From: Ritesh Harjani (IBM) <ritesh.list@...il.com>
To: Dave Chinner <david@...morbit.com>
Cc: Ojaswin Mujoo <ojaswin@...ux.ibm.com>, lsf-pc@...ts.linux-foundation.org, linux-xfs@...r.kernel.org, linux-fsdevel@...r.kernel.org, djwong@...nel.org, dchinner@...hat.com, jack@...e.cz, tytso@....edu, linux-ext4@...r.kernel.org, nirjhar.roy.lists@...il.com, zlang@...nel.org
Subject: Re: [LSF/MM/BPF TOPIC] xfstests: Centralizing filesystem configs and device configs

Dave Chinner <david@...morbit.com> writes:

> On Sat, Mar 01, 2025 at 06:39:57PM +0530, Ritesh Harjani wrote:
>> > Why is having hundreds of tiny single-config-only files
>> > better than having all the configs in a single file that is
>> > easily browsed and searched?
>> >
>> > Honestly, I really don't see any advantage to re-implementing config
>> > sections as a "file per config" object farm. Yes, you can store
>> > information that way, but that doesn't make it an improvement over a
>> > single file...
>> >
>> > All that is needed is for the upstream repository to maintain a
>> > config file with all the config sections defined that people need.
>> > We don't need any new infrastructure to implement a "centralised
>> > configs" feature - all we need is an agreement that upstream will
>> > ship an update-to-date default config file instead of the ancient,
>> > stale example.config/localhost.config files....
>> >
>> 
>> If we can create 1 config for every filesystem instead of creating a lot
>> of smaller config files. i.e.  
>> - configs/ext4/config.ext4
>> - configs/xfs/config.xfs
>
> Why are directories that contain a single file needed here?
>

Agreed. Not needed we can have configs/ext4.config, configs/xfs.config
and so on...

>> Each of above can contain sections like (e.g.)
>> 
>> [xfs-b4k]
>> MKFS_OPTIONS="-b size=4k"
>> ddUNT_OPTIdd    d=""dd
>> 
>> [xfs-b64k]
>> MKFS_OPTIONS="-b size=64k"
>> MOUNT_OPTIONS=""
>> 
>> 
>> Then during make we can merge all these configs into a common config file
>> i.e. configs/.all-section-configs. We can update the current check script to
>> look for either local.config file or configs/.all-section-configs file
>> for location the section passed in the command line. 
>
> What does this complexity gain us?
>

Let me try and explain this below.


>> This will help solve all the listed problems:
>> 1. We don't have to add a new parsing logic for configs
>
> We don't need new config files and makefile/build time shenanigans
> to do this.
>

Same here.

>> 2. We don't need to create 1 file per config
>
> Ditto.
>
>> 3. We still can get all sections listed in one place under which check
>> script can parse.
>
> Ditto.
>
>> 4. Calling different filesystem sections from a common config file can work.
>
> Yes, that's the whole point of have config sections: one config file
> that supports lots of different test configurations!
>
>> So as you mentioned calling something like below should work. 
>> 
>> ./check -s xfs_4k -s ext4_4k -g quick
>> 
>> Hopefully this will require minimal changes to work. Does this sound
>> good to you?
>
> You haven't explained why we need new infrastructure to do something
> we can already do with the existing infrastructure. What problem are
> you trying to solve that the current infrastructure does not handle?
>
> i.e. we won't need to change the global config file very often once the
> common configs are defined in it; it'll only get modified when
> filesystems add new features that need specific mkfs or mount option
> support to be added, and that's fairly rare.
>
> Hence I still don't understand what new problem multiple config files
> and new infrastructure to support them is supposed to solve...


I will try and explain our reasoning here: 

1. Why have per-fs config file i.e. configs/ext4.config or 
configs/xfs.config...

Instead of 1 large config file it's easier if we have FS specific
sections in their own .config file. I agree we don't need configs/<fs>
directories for each filesystem. But it's much easier if we have
configs/<fs>.config with the necessary sections defined in it.  That
will be easy to maintain by their respective FS maintainers rather than
maintaining all sections defined in 1 large common config file.

2. Why then add the infrastructure to create a new common
configs/all-fs.config file during make?

This is a combined configs/all-fs.config file which need not be
maintained in git version control. It gets generated for our direct
use. This is also needed to run different cross filesystem tests from a
single ./check script. i.e. 

        ./check -s ext4_4k -s xfs_4k -g quick

(otherwise one cannot run ext4_4k and xfs_4k from a single ./check invocation)

I don't think this is too much burden for "make" to generate this file.
And it's easier than, for people to use configs/all-fs.config to run
cross filesystem tests (as mentioned above).

e.g. 
1. "make" will generate configs/all-fs.config
2. Define your devices.config in configs/devices.config
3. Then run 
   (. configs/devices.config; ./check -s ext4_4k -s xfs_4k -g quick)

Note:
One problem which I can think of with the sections approach as opposed
to multiple small config files (-c configs/ext4/4k.config) approach is
that, in sections approach we iterate over all the sections present in
the provided config file and search for the section passed in the
cmdline. Whereas in the small config files approach we were directly
passing the configuration options via a small config file. But hopefully
that linear search in locating the section is not as time consuming as
opposed to running the test itself.


Here is the sample change snippet which we are thinking of.
Thoughts?


>From c227f6335d09c2b482b759ac4df38d87ccedff6d Mon Sep 17 00:00:00 2001
Message-Id: <c227f6335d09c2b482b759ac4df38d87ccedff6d.1741201277.git.ritesh.list@...il.com>
From: "Ritesh Harjani (IBM)" <ritesh.list@...il.com>
Date: Wed, 5 Mar 2025 16:33:52 +0530
Subject: [RFC] configs: Add support for central fs configs using sections

This adds support for central fs configs using sections. This allows
different FS maintainers to maintain per-fs config file with different
sections to test multiple filesystem configurations like 1k, 4k, quota
etc.

During make we combine all configs/<fs>.config file into
configs/all-fs.config which is not tracked by git. check script will by
default use configs/all-fs.config file if local.config or
configs/$(HOST).config file is not present. This patch also adds
configs/devices.config.example which can be used as an example to export
default settings for SCRATCH_MNT, TEST_DIR, TEST_DEV, SCRATCH_DEV etc.

After defining default device settings say in configs/devices.config,
one can use:

(. configs/devices.config; ./check -s xfs_4k tests/selftest/001)

Note: make clean will clean the configs/all-fs.config file.

Co-developed-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@...il.com>
Signed-off-by: Nirjhar Roy (IBM) <nirjhar.roy.lists@...il.com>
Signed-off-by: Ritesh Harjani (IBM) <ritesh.list@...il.com>
---
 Makefile                       |  3 ++-
 common/config                  |  1 +
 configs/Makefile               | 26 ++++++++++++++++++++++++++
 configs/btrfs.config           |  4 ++++
 configs/devices.config.example |  9 +++++++++
 configs/ext4.config            |  9 +++++++++
 configs/xfs.config             |  9 +++++++++
 7 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100644 configs/Makefile
 create mode 100644 configs/btrfs.config
 create mode 100644 configs/devices.config.example
 create mode 100644 configs/ext4.config
 create mode 100644 configs/xfs.config

diff --git a/Makefile b/Makefile
index f955f0d3..490bb59a 100644
--- a/Makefile
+++ b/Makefile
@@ -41,8 +41,9 @@ endif
 
 LIB_SUBDIRS = include lib
 TOOL_SUBDIRS = ltp src m4 common tools
+CONFIGS_SUBDIRS = configs
 
-SUBDIRS = $(LIB_SUBDIRS) $(TOOL_SUBDIRS) $(TESTS_DIR)
+SUBDIRS = $(LIB_SUBDIRS) $(TOOL_SUBDIRS) $(TESTS_DIR) $(CONFIGS_SUBDIRS)
 
 default: include/builddefs
 ifeq ($(HAVE_BUILDDEFS), no)
diff --git a/common/config b/common/config
index 2afbda14..b4470951 100644
--- a/common/config
+++ b/common/config
@@ -554,6 +554,7 @@ known_hosts()
 {
 	[ "$HOST_CONFIG_DIR" ] || HOST_CONFIG_DIR=`pwd`/configs
 
+	[ -f $HOST_CONFIG_DIR/all-fs.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/all-fs.config
 	[ -f /etc/xfsqa.config ]             && export HOST_OPTIONS=/etc/xfsqa.config
 	[ -f $HOST_CONFIG_DIR/$HOST ]        && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST
 	[ -f $HOST_CONFIG_DIR/$HOST.config ] && export HOST_OPTIONS=$HOST_CONFIG_DIR/$HOST.config
diff --git a/configs/Makefile b/configs/Makefile
new file mode 100644
index 00000000..e91bcca1
--- /dev/null
+++ b/configs/Makefile
@@ -0,0 +1,26 @@
+TOPDIR = ..
+include $(TOPDIR)/include/builddefs
+
+CONFIGS_DIR = $(TOPDIR)/configs
+ALL_FS_CONFIG = $(CONFIGS_DIR)/all-fs.config
+EXT4_CONFIG = $(CONFIGS_DIR)/ext4.config
+XFS_CONFIG = $(CONFIGS_DIR)/xfs.config
+BTRFS_CONFIG = $(CONFIGS_DIR)/btrfs.config
+
+DIRT = $(ALL_FS_CONFIG)
+default: $(ALL_FS_CONFIG)
+
+include $(BUILDRULES)
+
+# Generate all-fs.config by concatenating required configs/.config files
+$(ALL_FS_CONFIG): $(EXT4_CONFIG) $(XFS_CONFIG) $(BTRFS_CONFIG)
+	@echo "Generating $@"
+	echo "# all-fs sections" > $@
+	echo "[default]" >> $@
+	@echo "" >> $@
+	@cat $(EXT4_CONFIG) >> $@
+	@echo "" >> $@
+	@cat $(XFS_CONFIG) >> $@
+	@echo "" >> $@
+	@cat $(BTRFS_CONFIG) >> $@
+	@echo "" >> $@
diff --git a/configs/btrfs.config b/configs/btrfs.config
new file mode 100644
index 00000000..1d85e13d
--- /dev/null
+++ b/configs/btrfs.config
@@ -0,0 +1,4 @@
+[btrfs]
+ FSTYP=btrfs
+ MKFS_OPTIONS="-f "
+ MOUNT_OPTIONS=""
diff --git a/configs/devices.config.example b/configs/devices.config.example
new file mode 100644
index 00000000..48598a4b
--- /dev/null
+++ b/configs/devices.config.example
@@ -0,0 +1,9 @@
+export TEST_DIR=/mnt1/test
+export SCRATCH_MNT=/mnt1/scratch
+export TEST_DEV=/dev/loop0
+export SCRATCH_DEV=/dev/loop1
+export TEST_LOGDEV=/dev/loop2
+export SCRATCH_LOGDEV=/dev/loop3
+export TEST_RTDEV=/dev/loop4
+export SCRATCH_RTDEV=/dev/loop5
+export LOGWRITES_DEV=/dev/loop6
diff --git a/configs/ext4.config b/configs/ext4.config
new file mode 100644
index 00000000..d5ceba3c
--- /dev/null
+++ b/configs/ext4.config
@@ -0,0 +1,9 @@
+[ext4_1k]
+ FSTYP=ext4
+ MKFS_OPTIONS="-F -b 1024"
+ MOUNT_OPTIONS=""
+
+[ext4_4k]
+ FSTYP=ext4
+ MKFS_OPTIONS="-F -b 4096"
+ MOUNT_OPTIONS=""
diff --git a/configs/xfs.config b/configs/xfs.config
new file mode 100644
index 00000000..516f35f7
--- /dev/null
+++ b/configs/xfs.config
@@ -0,0 +1,9 @@
+[xfs_1k]
+ FSTYP=xfs
+ MKFS_OPTIONS="-f -bsize=1024"
+ MOUNT_OPTIONS=""
+
+[xfs_4k]
+ FSTYP=xfs
+ MKFS_OPTIONS="-f -bsize=4096"
+ MOUNT_OPTIONS=""
-- 
2.39.5

Powered by blists - more mailing lists