linux-ext4 - Re: Aarch64 EXT4FS inode checksum failures - seems to be weak memory ordering issues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20210106115359.GB26994@C02TD0UTHF1T.local>
Date:   Wed, 6 Jan 2021 11:53:59 +0000
From:   Mark Rutland <mark.rutland@....com>
To:     Russell King - ARM Linux admin <linux@...linux.org.uk>
Cc:     linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
        Theodore Ts'o <tytso@....edu>,
        Andreas Dilger <adilger.kernel@...ger.ca>,
        linux-ext4@...r.kernel.org, Will Deacon <will@...nel.org>
Subject: Re: Aarch64 EXT4FS inode checksum failures - seems to be weak memory
 ordering issues

On Tue, Jan 05, 2021 at 03:47:26PM +0000, Russell King - ARM Linux admin wrote:
> Hi,

Hi Russell,

> This is an update on where I am with this long standing issue at the
> current time.
> 
> Since 5.4, I have been struggling with several of my ARM64 systems, of
> different SoC vendors and differing filesystem media, were sporadically
> reporting inode checksum failures on their root filesystems.  The time
> taken to report this has been anything between a few hours and three
> months of uptime, making the problem unrealistic to bisect.

> However, over the last couple of days, a way to reproduce it has been
> found, at least for the LX2160A based system.  Power down, leave the
> machine powered off for some time. Power up, log in and run:
> 
> while :; do sleep 5; find /var /usr /bin /sbin -type f -print0 | \
> 	xargs -0 md5sum >/dev/null; done

I've just set this off on an Raspberry Pi 4 running a locally-built
arm64 v5.10 defconfig. I'm using a SATA SSD mounted via a USB-SATA
adapter. I'll try to give that a few reboots see if I can reproduce the
issue.

Just to check -- how is your ext4 fs mounted? Mine is mounted as:

 /dev/sda2 on / type ext4 (rw,relatime,errors=remount-ro)

... and are you using defconfig or something else?

[...]

> It is possible that it could be compiler related, but I don't see that;
> if the "dmb oshld" were strong enough, then it should mean that the
> subsequent reads to checksum the inode data after the inode data has
> been DMA'd into memory should be reading the correct values from memory
> already - but they aren't. And if changing "dmb oshld" to "dsb ld" means
> that the code can then read the right values, that to me points fairly
> definitively to a hardware problem.

Just in case the compiler has some impact, can you way which compilers
you've been using? On the rpi4 I'm using GCC 8.3.0 as packaged by Debian
Buster (10.7).

Thanks,
Mark.