Message-ID: <c6f6de48-2594-05e4-2048-9a9c59c018d7@gmx.com>
Date:   Tue, 20 Aug 2019 10:38:56 +0800
From:   Qu Wenruo <quwenruo.btrfs@....com>
To:     Chao Yu <yuchao0@...wei.com>, Gao Xiang <hsiangkao@....com>,
        "Darrick J. Wong" <darrick.wong@...cle.com>
Cc:     Christoph Hellwig <hch@...radead.org>,
        "Theodore Y. Ts'o" <tytso@....edu>,
        Eric Biggers <ebiggers@...nel.org>,
        Richard Weinberger <richard@....at>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Jan Kara <jack@...e.cz>, Dave Chinner <david@...morbit.com>,
        David Sterba <dsterba@...e.cz>, Miao Xie <miaoxie@...wei.com>,
        devel <devel@...verdev.osuosl.org>,
        Stephen Rothwell <sfr@...b.auug.org.au>,
        Amir Goldstein <amir73il@...il.com>,
        linux-erofs <linux-erofs@...ts.ozlabs.org>,
        Al Viro <viro@...iv.linux.org.uk>,
        Jaegeuk Kim <jaegeuk@...nel.org>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Li Guifu <bluce.liguifu@...wei.com>,
        Fang Wei <fangwei1@...wei.com>, Pavel Machek <pavel@...x.de>,
        linux-fsdevel <linux-fsdevel@...r.kernel.org>,
        Andrew Morton <akpm@...ux-foundation.org>,
        torvalds <torvalds@...ux-foundation.org>
Subject: Re: [PATCH] erofs: move erofs out of staging



On 2019/8/20 上午10:24, Chao Yu wrote:
> On 2019/8/20 8:55, Qu Wenruo wrote:
>> [...]
>>>>> I have made a simple fuzzer to inject messy in inode metadata,
>>>>> dir data, compressed indexes and super block,
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/xiang/erofs-utils.git/commit/?h=experimental-fuzzer
>>>>>
>>>>> I am testing with some given dirs and the following script.
>>>>> Does it look reasonable?
>>>>>
>>>>> # !/bin/bash
>>>>>
>>>>> mkdir -p mntdir
>>>>>
>>>>> for ((i=0; i<1000; ++i)); do
>>>>> 	mkfs/mkfs.erofs -F$i testdir_fsl.fuzz.img testdir_fsl > /dev/null 2>&1
>>>>
>>>> mkfs fuzzes the image? Er....
>>>
>>> Thanks for your reply.
>>>
>>> First, This is just the first step of erofs fuzzer I wrote yesterday night...
>>>
>>>>
>>>> Over in XFS land we have an xfs debugging tool (xfs_db) that knows how
>>>> to dump (and write!) most every field of every metadata type.  This
>>>> makes it fairly easy to write systematic level 0 fuzzing tests that
>>>> check how well the filesystem reacts to garbage data (zeroing,
>>>> randomizing, oneing, adding and subtracting small integers) in a field.
>>>> (It also knows how to trash entire blocks.)
>>
>> The same tool exists for btrfs, although lacks the write ability, but
>> that dump is more comprehensive and a great tool to learn the on-disk
>> format.
>>
>>
>> And for the fuzzing defending part, just a few kernel releases ago,
>> there is none for btrfs, and now we have a full static verification
>> layer to cover (almost) all on-disk data at read and write time.
>> (Along with enhanced runtime check)
>>
>> We have covered from vague values inside tree blocks and invalid/missing
>> cross-ref find at runtime.
>>
>> Currently the two layered check works pretty fine (well, sometimes too
>> good to detect older, improper behaved kernel).
>> - Tree blocks with vague data just get rejected by verification layer
>>   So that all members should fit on-disk format, from alignment to
>>   generation to inode mode.
>>
>>   The error will trigger a good enough (TM) error message for developer
>>   to read, and if we have other copies, we retry other copies just as
>>   we hit a bad copy.
>>
>> - At runtime, we have much less to check
>>   Only cross-ref related things can be wrong now. since everything
>>   inside a single tree block has already be checked.
>>
>> In fact, from my respect of view, such read time check should be there
>> from the very beginning.
>> It acts kinda of a on-disk format spec. (In fact, by implementing the
>> verification layer itself, it already exposes a lot of btrfs design
>> trade-offs)
>>
>> Even for a fs as complex (buggy) as btrfs, we only take 1K lines to
>> implement the verification layer.
>> So I'd like to see every new mainlined fs to have such ability.
> 
> Out of curiosity, it looks like every mainstream filesystem has its own
> fuzz/injection tool in their tool-set, if it's really such a generic
> requirement, why shouldn't there be a common tool to handle that, let specified
> filesystem fill the tool's callback to seek a node/block and supported fields
> can be fuzzed in inode.

It could be possible for XFS/EXT* to share the same infrastructure
without much hassle.
(If not considering external journal)

But for btrfs, it's like a regular fs on a super large dm-linear, which
further builds its chunks on different dm-raid1/dm-linear/dm-raid56.

So not sure if it's possible for btrfs, as it contains its logical
address layer bytenr (the most common one) along with per-chunk physical
mapping bytenr (in another tree).

It may depends on the granularity. But definitely a good idea to do so
in a generic way.
Currently we depend on super kind student developers/reporters on such
fuzzed images, and developers sometimes get inspired by real world
corruption (or his/her mood) to add some valid but hard-to-hit corner
case check.

Thanks,
Qu

> It can help to avoid redundant work whenever Linux
> welcomes a new filesystem....
> 
> Thanks,
> 
>>
>>>
>>> Actually, compared with XFS, EROFS has rather simple on-disk format.
>>> What we inject one time is quite deterministic.
>>>
>>> The first step just purposely writes some random fuzzed data to
>>> the base inode metadata, compressed indexes, or dir data field
>>> (one round one field) to make it validity and coverability.
>>>
>>>>
>>>> You might want to write such a debugging tool for erofs so that you can
>>>> take apart crashed images to get a better idea of what went wrong, and
>>>> to write easy fuzzing tests.
>>>
>>> Yes, we will do such a debugging tool of course. Actually Li Guifu is now
>>> developping a erofs-fuse to support old linux versions or other OSes for
>>> archiveing only use, we will base on that code to develop a better fuzzer
>>> tool as well.
>>
>> Personally speaking, debugging tool is way more important than a running
>> kernel module/fuse.
>> It's human trying to write the code, most of time is spent educating
>> code readers, thus debugging tool is way more important than dead cold code.
>>
>> Thanks,
>> Qu
>>>
>>> Thanks,
>>> Gao Xiang
>>>
>>>>
>>>> --D
>>>>
>>>>> 	umount mntdir
>>>>> 	mount -t erofs -o loop testdir_fsl.fuzz.img mntdir
>>>>> 	for j in `find mntdir -type f`; do
>>>>> 		md5sum $j > /dev/null
>>>>> 	done
>>>>> done
>>>>>
>>>>> Thanks,
>>>>> Gao Xiang
>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Gao Xiang
>>>>>>
>>



Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)

Powered by blists - more mailing lists