| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20250815070518.1255842-1-chenridong@huaweicloud.com>
Date: Fri, 15 Aug 2025 07:05:18 +0000
From: Chen Ridong <chenridong@...weicloud.com>
To: tj@...nel.org,
hannes@...xchg.org,
mkoutny@...e.com,
lizefan@...wei.com
Cc: cgroups@...r.kernel.org,
linux-kernel@...r.kernel.org,
lujialin4@...wei.com,
chenridong@...wei.com,
hdanton@...a.com
Subject: [PATCH v3] cgroup: cgroup: drain specific subsystems when mounting/destroying a root
From: Chen Ridong <chenridong@...wei.com>
A hung task can occur during [1] LTP cgroup testing when repeatedly
mounting/unmounting perf_event and net_prio controllers with
systemd.unified_cgroup_hierarchy=1. The hang manifests in
cgroup_lock_and_drain_offline() during root destruction.
Related case:
cgroup_fj_function_perf_event cgroup_fj_function.sh perf_event
cgroup_fj_function_net_prio cgroup_fj_function.sh net_prio
Call Trace:
cgroup_lock_and_drain_offline+0x14c/0x1e8
cgroup_destroy_root+0x3c/0x2c0
css_free_rwork_fn+0x248/0x338
process_one_work+0x16c/0x3b8
worker_thread+0x22c/0x3b0
kthread+0xec/0x100
ret_from_fork+0x10/0x20
Root Cause:
CPU0 CPU1
mount perf_event umount net_prio
cgroup1_get_tree cgroup_kill_sb
rebind_subsystems // root destruction enqueues
// cgroup_destroy_wq
// kill all perf_event css
// one perf_event css A is dying
// css A offline enqueues cgroup_destroy_wq
// root destruction will be executed first
css_free_rwork_fn
cgroup_destroy_root
cgroup_lock_and_drain_offline
// some perf descendants are dying
// cgroup_destroy_wq max_active = 1
// waiting for css A to die
Problem scenario:
1. CPU0 mounts perf_event (rebind_subsystems)
2. CPU1 unmounts net_prio (cgroup_kill_sb), queuing root destruction work
3. A dying perf_event CSS gets queued for offline after root destruction
4. Root destruction waits for offline completion, but offline work is
blocked behind root destruction in cgroup_destroy_wq (max_active=1)
Solution:
Introduce ss_mask for cgroup_lock_and_drain_offline() to selectively drain
specific subsystems rather than all subsystems.
There are two primary scenarios requiring offline draining:
1. Root Operations - Draining all subsystems in cgrp_dfl_root when mounting
or destroying a cgroup root
2. Draining specific cgroup when modifying cgroup.subtree_control or
cgroup.threads
For case 1 (Root Operations), it only need to drain the specific subsystem
being mounted/destroyed, not all subsystems. The rationale for draining
cgrp_dfl_root is explained in [2].
For case 2, it's enough to drain subsystems enabled in the cgroup. Since
other subsystems cannot have descendants in this cgroup, adding ss_mask
should not have a hurt.
[1] https://github.com/linux-test-project/ltp/blob/master/runtest/controllers
[2] https://lore.kernel.org/cgroups/39e05402-40c7-4631-a87b-8e3747ceddc6@huaweicloud.com/T/#t
Fixes: 334c3679ec4b ("cgroup: reimplement rebind_subsystems() using cgroup_apply_control() and friends")
Reported-by: Gao Yingjie <gaoyingjie@...ontech.com>
Signed-off-by: Chen Ridong <chenridong@...wei.com>
---
v2->v3: add ss_mask for cgroup_lock_and_drain_offline to fix this issue.
v2: https://lore.kernel.org/cgroups/20250815024020.4579-1-hdanton@sina.com/T/#t
kernel/cgroup/cgroup-internal.h | 2 +-
kernel/cgroup/cgroup-v1.c | 4 ++--
kernel/cgroup/cgroup.c | 11 ++++++++---
3 files changed, 11 insertions(+), 6 deletions(-)
diff --git a/kernel/cgroup/cgroup-internal.h b/kernel/cgroup/cgroup-internal.h
index b14e61c64a34..3c7443d8b3b7 100644
--- a/kernel/cgroup/cgroup-internal.h
+++ b/kernel/cgroup/cgroup-internal.h
@@ -257,7 +257,7 @@ struct task_struct *cgroup_procs_write_start(char *buf, bool threadgroup,
void cgroup_procs_write_finish(struct task_struct *task, bool locked)
__releases(&cgroup_threadgroup_rwsem);
-void cgroup_lock_and_drain_offline(struct cgroup *cgrp);
+void cgroup_lock_and_drain_offline(struct cgroup *cgrp, u16 ss_mask);
int cgroup_mkdir(struct kernfs_node *parent_kn, const char *name, umode_t mode);
int cgroup_rmdir(struct kernfs_node *kn);
diff --git a/kernel/cgroup/cgroup-v1.c b/kernel/cgroup/cgroup-v1.c
index 2a4a387f867a..1be439c62fbe 100644
--- a/kernel/cgroup/cgroup-v1.c
+++ b/kernel/cgroup/cgroup-v1.c
@@ -1096,7 +1096,7 @@ int cgroup1_reconfigure(struct fs_context *fc)
int ret = 0;
u16 added_mask, removed_mask;
- cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp);
+ cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp, root->subsys_mask);
/* See what subsystems are wanted */
ret = check_cgroupfs_options(fc);
@@ -1261,7 +1261,7 @@ int cgroup1_get_tree(struct fs_context *fc)
if (!ns_capable(ctx->ns->user_ns, CAP_SYS_ADMIN))
return -EPERM;
- cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp);
+ cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp, ctx->subsys_mask);
ret = cgroup1_root_to_use(fc);
if (!ret && !percpu_ref_tryget_live(&ctx->root->cgrp.self.refcnt))
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index d8b82afed181..c76c23186fd3 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -1346,7 +1346,7 @@ static void cgroup_destroy_root(struct cgroup_root *root)
trace_cgroup_destroy_root(root);
- cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp);
+ cgroup_lock_and_drain_offline(&cgrp_dfl_root.cgrp, root->subsys_mask);
BUG_ON(atomic_read(&root->nr_cgrps));
BUG_ON(!list_empty(&cgrp->self.children));
@@ -1681,7 +1681,7 @@ struct cgroup *cgroup_kn_lock_live(struct kernfs_node *kn, bool drain_offline)
kernfs_break_active_protection(kn);
if (drain_offline)
- cgroup_lock_and_drain_offline(cgrp);
+ cgroup_lock_and_drain_offline(cgrp, cgrp->subtree_ss_mask);
else
cgroup_lock();
@@ -3147,12 +3147,14 @@ static int cgroup_update_dfl_csses(struct cgroup *cgrp)
/**
* cgroup_lock_and_drain_offline - lock cgroup_mutex and drain offlined csses
* @cgrp: root of the target subtree
+ * @ss_mask: bitmask of subsystem to be drained
*
* Because css offlining is asynchronous, userland may try to re-enable a
* controller while the previous css is still around. This function grabs
* cgroup_mutex and drains the previous css instances of @cgrp's subtree.
*/
-void cgroup_lock_and_drain_offline(struct cgroup *cgrp)
+
+void cgroup_lock_and_drain_offline(struct cgroup *cgrp, u16 ss_mask)
__acquires(&cgroup_mutex)
{
struct cgroup *dsct;
@@ -3168,6 +3170,9 @@ void cgroup_lock_and_drain_offline(struct cgroup *cgrp)
struct cgroup_subsys_state *css = cgroup_css(dsct, ss);
DEFINE_WAIT(wait);
+ if (!(ss_mask & 1U << ssid))
+ continue;
+
if (!css || !percpu_ref_is_dying(&css->refcnt))
continue;
--
2.34.1
Powered by blists - more mailing lists