linux-kernel - Re: [BUG RESEND] Kernel panic in update_cfs_rq_h

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <aPEoLc1mDB22f2Ff@arm.com>
Date: Thu, 16 Oct 2025 18:15:25 +0100
From: Catalin Marinas <catalin.marinas@....com>
To: Cruz Zhao <CruzZhao@...ux.alibaba.com>
Cc: peterz@...radead.org, mingo@...hat.com, will@...nel.org,
	juri.lelli@...hat.com, vincent.guittot@...aro.org,
	dietmar.eggemann@....com, rostedt@...dmis.org, bsegall@...gle.com,
	mgorman@...e.de, vschneid@...hat.com, linux-kernel@...r.kernel.org
Subject: Re: [BUG RESEND] Kernel panic in update_cfs_rq_h_load() on aarch64

On Mon, Oct 13, 2025 at 03:17:58PM +0800, Cruz Zhao wrote:
> Hi, all. We are encountering a kernel panic in our production environment
> running Linux kernel version 5.10.134, on aarch64 architecture (aarch64
> architecture only).
> 
> Summary:
> - Kernel Version: 5.10.134

That's a pretty old kernel though still maintained LTS. Can you
reproduce this with the latest 5.10.x?

>  - Architecture: aarch64
>  - Problem Type: panic
>  - Reproducibility: About 40 times per week
> 
> This issue leads to system-wide unresponsiveness and requires a hard reboot.
> We have collected the panic logs and backtraces, which I will include below
> for your reference. We tried to analyze the vmcore but have not found out
> the root cause.
> 
> Backtrace from dmesg:
> [352245.601209] Unable to handle kernel paging request at virtual address 006410a84bf5c71c
> [352245.609208] Mem abort info:
> [352245.612077]   ESR = 0x96000004
> [352245.615208]   EC = 0x25: DABT (current EL), IL = 32 bits
> [352245.620595]   SET = 0, FnV = 0
> [352245.623723]   EA = 0, S1PTW = 0
> [352245.626938] Data abort info:
> [352245.629893]   ISV = 0, ISS = 0x00000004
> [352245.633803]   CM = 0, WnR = 0
> [352245.636845] [006410a84bf5c71c] address between user and kernel address ranges
> [352245.644059] Internal error: Oops: 96000004 [#1] SMP
> [352245.728419] CPU: 98 PID: 2075637 Comm: execution_task_ Kdump: loaded Tainted: G           OE     5.10.134-xxx.al8.aarch64 #1
> [352245.740578] Hardware name: Alibaba Alibaba Cloud ECS/Alibaba Cloud ECS, BIOS 1.2.M1.AL.P.153.00 05/23/2023
> [352245.750306] pstate: 82401089 (Nzcv daIf +PAN -UAO +TCO BTYPE=--)
> [352245.756400] pc : update_cfs_rq_h_load+0x80/0xb0
> [352245.761007] lr : wake_affine_weight+0xa8/0x158
[...]
> The crash occurred because the line of code "cfs_rq->h_load = load;" accessed
> an invalid pointer 006410a84bf5c71c when trying to access cfs->h_load. The
> variable cfs_rq was obtained via "cfs_rq = group_cfs_rq(se);". According to
> the assembly code, the value of se is stored in register x2, with the pointer
> ffff04128c023c00. However, upon inspecting its fields, it turns out that
> this object is not a valid sched_entity. We've analyzed the logic of code,
> and checked the cfs_rq and se in memory, which are all correct, but we cannot
> find out where the pointer ffff04128c023c00 came from. And we've also
> analyzed most of the vmcores, and identified a common pattern: all corrupted
> pointers belong to the kmalloc-1k slab type, the same as cfs_rq and
> sched_entity.
> 
> We currently suspect that this is not caused by a scheduler bug. Has anyone
> encountered a similar crash stack? Does anyone know why this issue only
> occurs on aarch64?

I haven't seen this before but I don't do large scale testing. It could
be some unrelated bug corrupting adjacent kmalloc allocations from the
same pool. Have you tried enabling some kernel sanitisers like kasan and
see if they spot anything?

-- 
Catalin