linux-kernel - Re: [bug report] WARNING: CPU: 0 PID: 49573 at kernel/sched/rt.c:802 rq_offline

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <1314703e-4251-f31f-3d43-ece5c1b68344@huawei.com>
Date: Mon, 13 May 2024 22:10:00 +0800
From: "zhaowenhui (A)" <zhaowenhui8@...wei.com>
To: Ingo Molnar <mingo@...hat.com>, Peter Zijlstra <peterz@...radead.org>,
	Juri Lelli <juri.lelli@...hat.com>, Vincent Guittot
	<vincent.guittot@...aro.org>, Dietmar Eggemann <dietmar.eggemann@....com>,
	Steven Rostedt <rostedt@...dmis.org>, Ben Segall <bsegall@...gle.com>, Mel
 Gorman <mgorman@...e.de>, Daniel Bristot de Oliveira <bristot@...hat.com>,
	Valentin Schneider <vschneid@...hat.com>, "open list:SCHEDULER"
	<linux-kernel@...r.kernel.org>
Subject: Re: [bug report] WARNING: CPU: 0 PID: 49573 at kernel/sched/rt.c:802
 rq_offline_rt+0x24d/0x260



在 2024/4/7 22:06, zhaowenhui (A) 写道:

> More information:
> 1. RT_RUNTIME_SHARE is enabled.
> 2. We continuously create and remove cpu cgroups. We use cgexec to do 
> some tasks like "tree" or "ps" in these cgroups and the rt_runtime_us in 
> these cgroups are set to 2000~6000.
> 3. There are frequent cpu offline/online operations, so it will trigger 
> __disable_runtime.
> 
> Every time we run these operations after reboot, this warning will 
> happen easily.
> 

--------reproduce shell scripts----------
[1] on_off.sh
#!/bin/bash

let cpu_end=8

while true
do
         for i in `seq 0 $cpu_end`
         do
                 echo $i
                 let j=i+1
                 echo 0 > /sys/devices/system/cpu/cpu${i}/online
                 echo 0 > /sys/devices/system/cpu/cpu${j}/online
                 sleep 0.5
                 echo 1 > /sys/devices/system/cpu/cpu${i}/online
                 echo 1 > /sys/devices/system/cpu/cpu${j}/online
         done
done

[2] rt_test.sh
#!/bin/bsh
# set -x

name=test$$
cpu_dir=/sys/fs/cgroup/cpu/$name
mkdir $cpu_dir
echo $rt_runtime_us > $cpu_dir/cpu.rt_runtime_us  &>/dev/null

function cg_change_rt_runtime_us()
{
         while true
         do
                 let tmp=2+RANDOM%4
                 let quota=tmp*1000
                 echo $quota > $cpu_dir/cpu.rt_runtime_us
         done
}

function rt_test1()
{
         while true
         do
                 cgexec -g cpu:/${name} chrt 20 tree /sys/fs/cgroup/cpu 
&>/dev/null
         done
}

function rt_test2()
{
         while true
         do
                 cgexec -g cpu:/${name} chrt 30 ps aux  &>/dev/null
         done
}

rt_test1 &
rt_test2 &
cg_change_rt_runtime_us &> /dev/null &

[3] start_repro.sh
function set_env(){
     echo RT_RUNTIME_SHARE > /sys/kernel/debug/sched/features
     echo RT_RUNTIME_SHARE > /sys/kernel/debug/sched_features
}

function run_test(){
     for i in 0 1
     do
         nohup sh rt_test.sh &
         nohup sh rt_test.sh &
         nohup sh rt_test.sh &
         nohup sh rt_test.sh &
     done

     nohup sh on_off.sh &>/dev/null &

}
set_env
run_test

--------reproduce shell scripts----------

Running the start_repro.sh can easily reproduce this issue.
And when I debug in __disable_runtime, I found that the rtrq have
borrow some runtime, because rtrq->runtime is greater than 
rtb->rt_runtime, but its span has only one cpu which is itself.
[   43.686441] [__disable_runtime] begin, rq: ff3113087e5b4ec0 , cpu: 159
[   43.686449] [__disable_runtime] rtrq: ff3112c9e9259800 , 
rtrq->runtime :3037500
[   43.686450] [__disable_runtime] rtb->period: 1000000000 , 
rtb->rt_runtime: 3000000
[   43.686451] [__disable_runtime] want original : -37500
[   43.686451] [__disable_runtime] after get want, rtrq: 
ff3112c9e9259800 , rtrq->runtime :3037500
[   43.686453] [ERROR!] span have cpu: 159

Do you have any idea about this? 😄

---
Regards
Zhao Wenhui