linux-ext4 - Re: [Regression] ext4: changes to mb_optimize

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20220817105736.n22yopqcq7badhe7@quack3>
Date:   Wed, 17 Aug 2022 12:57:36 +0200
From:   Jan Kara <jack@...e.cz>
To:     Stefan Wahren <stefan.wahren@...e.com>
Cc:     Jan Kara <jack@...e.cz>, linux-ext4@...r.kernel.org,
        Ojaswin Mujoo <ojaswin@...ux.ibm.com>,
        Harshad Shirwadkar <harshadshirwadkar@...il.com>,
        Theodore Ts'o <tytso@....edu>,
        Ritesh Harjani <riteshh@...ux.ibm.com>,
        linux-fsdevel@...r.kernel.org,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Geetika.Moolchandani1@....com, regressions@...ts.linux.dev,
        Florian Fainelli <f.fainelli@...il.com>
Subject: Re: [Regression] ext4: changes to mb_optimize_scan cause issues on
 Raspberry Pi

Hi Stefan!

On Tue 16-08-22 22:45:48, Stefan Wahren wrote:
> Am 16.08.22 um 11:34 schrieb Jan Kara:
> > Hi Stefan!
> > So this is interesting. We can see the card is 100% busy. The IO submitted
> > to the card is formed by small requests - 18-38 KB per request - and each
> > request takes 0.3-0.5s to complete. So the resulting throughput is horrible
> > - only tens of KB/s. Also we can see there are many IOs queued for the
> > device in parallel (aqu-sz columnt). This does not look like load I would
> > expect to be generated by download of a large file from the web.
> > 
> > You have mentioned in previous emails that with dd(1) you can do couple
> > MB/s writing to this card which is far more than these tens of KB/s. So the
> > file download must be doing something which really destroys the IO pattern
> > (and with mb_optimize_scan=0 ext4 happened to be better dealing with it and
> > generating better IO pattern). Can you perhaps strace the process doing the
> > download (or perhaps strace -f the whole rpi-update process) so that we can
> > see how does the load generated on the filesystem look like? Thanks!
> 
> i didn't create the strace yet, but i looked at the source of rpi-update. At
> the end the download phase is a curl call to download a tar archive and pipe
> it directly to tar.
> 
> You can find the content list of the tar file here:
> 
> https://raw.githubusercontent.com/lategoodbye/mb_optimize_scan_regress/main/rpi-firmware-tar-content-list.txt

Thanks for the details! This is indeed even better. Looking at the tar
archive I can see it consists of a lot of small files big part of them
is even below 10k. So this very much matches the workload I was examining
with reaim where I saw regression (although only ~8%) even on normal
rotating drive on x86 machine. In that case I have pretty much confirmed
that the problem is due to mb_optimize_scan=1 spreading small allocated
files more which is likely also harmful for the SD card because it requires
touching more erase blocks.

Thanks for help with debugging this, I will implement some of the
heuristics we discussed with other ext4 developers to avoid this behavior
and will send you patch for testing.

								Honza
-- 
Jan Kara <jack@...e.com>
SUSE Labs, CR