linux-kernel - Re: Folio Related Stability Crashes in 6.17.5 and 6.17.6

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <erytfim3m6fy7qy3tmfwwkr6ymvyfoevhwsac4xtadimei6ux5@um6ylfu5e4ci>
Date: Thu, 6 Nov 2025 08:43:25 +0000
From: Pedro Falcato <pfalcato@...e.de>
To: Robert Dinse <nanook@...imo.com>
Cc: mhocko@...e.com, linux-mm@...ck.org, linux-kernel@...r.kernel.org
Subject: Re: Folio Related Stability Crashes in 6.17.5 and 6.17.6

On Wed, Nov 05, 2025 at 07:19:17PM -0800, Robert Dinse wrote:
>       Since 6.17.5 I have twice had one of my servers lock up in a state
> where it still routed network traffic and responded to pings but no user
> programs are running.
> 
>      These are proceeded by kernel splats:
>

Are you sure it didn't ever repro on 6.17.5?
Also, I'm wondering if you have the rest of the dmesg?

> [542551.007650] [    T238] INFO: task php8.3:1373100 blocked for more than
> 614 seconds.
> [542551.007652] [    T238]       Tainted: G S      W   E  6.17.6 #1
> [542551.007653] [    T238] "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> [542551.007654] [    T238] task:php8.3          state:D stack:0  pid:1373100
> tgid:1373100 ppid:5513   task_flags:0x400000 flags:0x00004002
> [542551.007657] [    T238] Call Trace:
> [542551.007658] [    T238]  <TASK>
> [542551.007659] [    T238]  __schedule+0x41c/0x16b0
> [542551.007661] [    T238]  ? _raw_spin_unlock_bh+0x1d/0x30
> [542551.007664] [    T238]  ? tcp_cleanup_rbuf+0x43/0xa0
> [542551.007670] [    T238]  schedule+0x20/0xe0
> [542551.007671] [    T238]  io_schedule+0x4c/0x80
> [542551.007673] [    T238]  folio_wait_bit+0x102/0x200

This does not have anything to do with folios (probably), but it looks like
swap IO is stuck. So, could possibly even be a block IO problem.

> [542551.007676] [    T238]  ? __pfx_wake_page_function+0x10/0x10
> [542551.007678] [    T238]  folio_wait_writeback+0x2e/0x80
> [542551.007680] [    T238]  shmem_swapin_folio+0x58d/0x1210
> [542551.007683] [    T238]  ? mod_memcg_lruvec_state+0xed/0x2b0
> [542551.007686] [    T238]  ? xas_load+0x11/0x100
> [542551.007688] [    T238]  ? filemap_get_entry+0x60/0x1c0
> [542551.007690] [    T238]  ? __lruvec_stat_mod_folio+0x7a/0xc0
> [542551.007693] [    T238]  shmem_get_folio_gfp+0x17a/0x5f0
> [542551.007696] [    T238]  shmem_fault+0x7a/0x390
> [542551.007698] [    T238]  __do_fault+0x38/0x1b0
> [542551.007701] [    T238]  do_fault+0x2bc/0x5b0
> [542551.007703] [    T238]  ? __x64_sys_alarm+0x61/0xb0
> [542551.007707] [    T238]  __handle_mm_fault+0x439/0xfc0
> [542551.007709] [    T238]  ? x64_sys_call+0x17e7/0x2330
> [542551.007715] [    T238]  handle_mm_fault+0xeb/0x2f0
> [542551.007718] [    T238]  do_user_addr_fault+0x203/0x690
> [542551.007720] [    T238]  exc_page_fault+0x7f/0x170
> [542551.007722] [    T238]  asm_exc_page_fault+0x27/0x30
> <snip>
> [542551.008011] [    T238] Kernel panic - not syncing: hung_task: blocked
> tasks
> [542551.008013] [    T238] CPU: 16 UID: 0 PID: 238 Comm: khungtaskd Kdump:
> loaded Tainted: G S      W   E       6.17.6 #1 NONE
> [542551.008017] [    T238] Tainted: [S]=CPU_OUT_OF_SPEC, [W]=WARN,
> [E]=UNSIGNED_MODULE

So, you have 1) CPU_OUT_OF_SPEC (wondering how you even have that?),
2) WARN before (which possibly explains this hang)

> <snip> 
>      Am running self compiled kernel on Ubuntu 24.04 system.  The .config
> used to generate the kernel is attached.  The compiler used is gcc 15.2. 
> The hardware this is running on is an i9-10980xe based machine clocked at
> 4.4Ghz all cores with -8 multiplier offset for avx512, -3 multiplier offset
> for AVX2.  The machine is equipped with 256G of RAM. There are three RAID1
> arrays, one based upon two SN770 WD Black nvme 1G drives mounted on /root,
> another based upon 13TB CMD drives on /space and a third on 13TB SMR drives
> on /bu.  We use Apache/2.4.65 (Unix) OpenSSL/3.6.0-dev and PHP from 5.6 to
> 8.3, most applications on 8.3.

So, yeah. As you mentioned on another email, a crashdump could most
definitely prove useful, but that's a PITA to setup. I think the full
dmesg could possibly be the key here. I would not be surprised if this
turns out to be a bug in another place.

-- 
Pedro