lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ZNvCJAclBEJf7uUA@fisica.ufpr.br>
Date:   Tue, 15 Aug 2023 15:21:24 -0300
From:   Carlos Carvalho <carlos@...ica.ufpr.br>
To:     linux-raid@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: 6.5.0rc5 fs hang - ext4? raid?

Dr. David Alan Gilbert (dave@...blig.org) wrote on Mon, Aug 14, 2023 at 06:02:53PM -03:
>   I'm seeing a few hangs on a fs after upgrading to fedora 39's bleeding
> edge; which is running kernel 6.5.0-0.rc5.20230808git14f9643dc90a.37.fc39.x86_64
> It was always solid prior to that.  It seems to trigger on heavy IO
> on this fs.

Good news! No, I didn't forget the smiley... Maybe now the problem has become
sufficiently bad to be visible/solvable...

6.4.* also doesn't run in one of our machines, which has heavy I/O load. The
first symptom is that rsync downloads hang and abort with timeout. 1 or 2
days later the amount of modified pages waiting to go to disk reaches several
GB, as reported by /proc/meminfo, but disks remain idle. Finally reading from
the arrays collapses.

This is just the worst case. Since early 5.* I/O performance has dropped
absurdly. In all our disk servers this is easy to see: just generate lots of
writes quickly (for example expanding a kernel tarball). Using top I see that
kworker starts using 100% cpu but disks stay idle (as seen by dstat or sar). If
you do a sync or umount it takes looooong to reach ~0 modified pages for the
sync or umount to return.

In the server I mentioned above where 6.4.* don't stand the load, which is one
of the largest free software mirrors of the world, even sometimes 6.1
collapses: I/O becomes so slow that service (apache) stops.

The problem gets progressively worse with time after booting. It's hardly
noticeable in the first hour after boot, and easily seen after ~3-4 days of
uptime. The higher the (write) I/O load the faster it appears.

All this is with ext4 and raid6 with >~ 14 disks in the arrays.

I don't have debug info because these are production machines and I only
compile in the kernel the bare minimum essential for operation. It's always
pure kernel.org releases; gcc versions vary, for 6.4* it's gcc-13, for 6.1*
gcc-12 is used, on Debian unstable updated more than 4 times/week.

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ