[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <a578623ecbae88433876381d3b28ec494f479ab3.camel@mediatek.com>
Date: Wed, 22 Mar 2023 09:37:01 +0000
From: Ryan Xiao (肖水林) <Ryan.Xiao@...iatek.com>
To: "tj@...nel.org" <tj@...nel.org>,
"peterz@...radead.org" <peterz@...radead.org>,
"vschneid@...hat.com" <vschneid@...hat.com>
CC: "linux-arm-kernel@...ts.infradead.org"
<linux-arm-kernel@...ts.infradead.org>,
wsd_upstream <wsd_upstream@...iatek.com>,
"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"linux-mediatek@...ts.infradead.org"
<linux-mediatek@...ts.infradead.org>,
Yongjun Luo (罗勇军)
<Yongjun.Luo@...iatek.com>
Subject: BUG: HANG_DETECT waiting for migration_cpu_stop() complete
Hi,
We meet the HANG_DETECT happened in T SW version with kernel-5.15.
Many tasks have been blocked for a long time.
Root cause:
migration_cpu_stop() is not complete due to is_migration_disabled(p) is
true, complete is false and complete_all() never get executed.
It let other task wait the rwsem.
Detail:
system_server waiting for cgroup_threadgroup_rwsem.
OomAdjuster is holding the cgroup_threadgroup_rwsem and waiting for
cpuset_rwsem.
cpuset_hotplug_workfn is holding the cpuset_rwsem and waiting for
affine_move_task() complete.
affine_move_task() waiting for migration_cpu_stop() complete.
The backtrace of system_server:
__switch_to
__schedule
schedule
percpu_rwsem_wait
__percpu_down_read
cgroup_css_set_fork => wait for cgroup_threadgroup_rwsem
cgroup_can_fork
copy_process
kernel_clone
The backtrace of OomAdjuster:
__switch_to
__schedule
schedule
percpu_rwsem_wait
percpu_down_write
cpuset_can_attach => wait for cpuset_rwsem
cgroup_migrate_execute
cgroup_attach_task
__cgroup1_procs_write => hold cgroup_threadgroup_rwsem
cgroup1_procs_write
cgroup_file_write
kernfs_fop_write_iter
vfs_write
ksys_write
The backtrace of cpuset_hotplug_workfn:
__switch_to
__schedule
schedule
schedule_timeout
wait_for_common
affine_move_task => wait for complete
__set_cpus_allowed_ptr_locked
update_tasks_cpumask
cpuset_hotplug_update_tasks => hold cpuset_rwsem
cpuset_hotplug_workfn
process_one_work
worker_thread
kthread
In affine_move_task() will call migration_cpu_stop() and wait for it
complete.
In normal case, if migration_cpu_stop() complete it will inform
everyone that he is done.
But there is an exception case that will not notify.
If is_migration_disabled(p) is true and complete will always is false,
then complete_all() never get executed.
static int migration_cpu_stop(void *data)
{
...
bool complete = false;
...
if (task_rq(p) == rq) {
if (is_migration_disabled(p))
goto out; => is_migration_disabled(p) = true,
so complete = false.
...
}
...
out:
...
if (complete) => complete = false,
so complete_all() never get executed.
complete_all(&pending->done);
return 0;
}
Review the code, we found that there are many places can change
is_migration_disabled() value.
(such as: __rt_spin_lock(), rt_read_lock(), rt_write_lock(), ...)
Do you have any suggestion for this issue?
Thank you.
Powered by blists - more mailing lists