| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Thu, 6 Jul 2023 14:08:20 +0800
From: Henry Wu <triangletrap12@...il.com>
To: linux-kernel@...r.kernel.org
Cc: peterz@...radead.org
Subject: Possible race in rt_mutex_adjust_prio_chain
Hi,
I found that it's not safe to change waiter->prio after waiter
dequeued from mutex's waiter rbtree because it's still on owner's
pi_waiters rbtree. From my analysis, waiters on pi_waiters rbtree
should be protected by pi_lock of task which have pi_waiters and
corresponding rt_mutex's wait_lock, but pi_waiters is shared by many
locks owned by this task, so actually we serialize changes on
pi_waiters only by pi_lock.
`rt_mutex_adjust_prio_chain' changes key of nodes of pi_waiters rbtree
without pi_lock and pi_waiters rbtree's invariant is violated. Maybe
we are enqueuing waiter on other cpu and pi_waiters rbtree will be
corrupted.
I attach a source file which can trigger this violation. I tested it
on Ubuntu 20.04 LTS with 5.4 kernel.
$ sudo ./a.out
pid: 39949
prio: 39
PID LWP TTY TIME CMD
39949 39949 pts/1 00:00:00 a.out
39949 39950 pts/1 00:00:00 waiter-0
39949 39951 pts/1 00:00:00 waiter-1
.............
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
.............
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
PID LWP TTY TIME CMD
39949 39949 pts/1 00:00:00 a.out
39949 39950 pts/1 00:00:00 waiter-0
39949 39951 pts/1 00:00:00 waiter-1
39949 39952 pts/1 00:00:00 waiter-2
39949 39953 pts/1 00:00:00 waiter-3
39949 39954 pts/1 00:00:00 waiter-4
39949 39955 pts/1 00:00:00 waiter-5
39949 39956 pts/1 00:00:00 waiter-6
39949 39957 pts/1 00:00:00 waiter-7
39949 39958 pts/1 00:00:00 waiter-8
39949 39959 pts/1 00:00:00 waiter-9
39949 39960 pts/1 00:00:00 waiter-10
39949 39961 pts/1 00:00:00 waiter-11
39949 39962 pts/1 00:00:00 waiter-12
39949 39963 pts/1 00:00:00 waiter-13
39949 39964 pts/1 00:00:00 waiter-14
39949 39965 pts/1 00:00:00 waiter-15
39949 39966 pts/1 00:00:00 waiter-16
39949 39967 pts/1 00:00:00 waiter-17
39949 39968 pts/1 00:00:00 waiter-18
39949 39969 pts/1 00:00:00 changer-0
39949 39970 pts/1 00:00:00 changer-1
39949 39971 pts/1 00:00:00 changer-2
39949 39972 pts/1 00:00:00 changer-3
39949 39973 pts/1 00:00:00 changer-4
39949 39974 pts/1 00:00:00 changer-5
39949 39975 pts/1 00:00:00 changer-6
39949 39976 pts/1 00:00:00 changer-7
39949 39977 pts/1 00:00:00 changer-8
39949 39978 pts/1 00:00:00 changer-9
39949 39979 pts/1 00:00:00 changer-10
39949 39980 pts/1 00:00:00 changer-11
39949 39981 pts/1 00:00:00 changer-12
39949 39982 pts/1 00:00:00 changer-13
39949 39983 pts/1 00:00:00 changer-14
39949 39984 pts/1 00:00:00 changer-15
39949 39985 pts/1 00:00:00 changer-16
39949 39986 pts/1 00:00:00 changer-17
39949 39987 pts/1 00:00:00 changer-18
found race, hang...
$ crash
crash> task -R prio,normal_prio,pi_waiters 39949
PID: 39949 TASK: ffff956a5c8b2f00 CPU: 2 COMMAND: "a.out"
prio = 121,
normal_prio = 139,
pi_waiters = {
rb_root = {
rb_node = 0xffffb24941f43d40
},
rb_leftmost = 0xffffb24941f8bd40
},
crash> print (struct rb_node *)0xffffb24941f43d40
$62 = (struct rb_node *) 0xffffb24941f43d40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$63 = {
tree_entry = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
pi_tree_entry = {
__rb_parent_color = 1,
rb_right = 0xffffb24941f13d40,
rb_left = 0xffffb24941f4bd40
},
task = 0xffff956a6c008000,
lock = 0xffff956920b80970,
prio = 130,
deadline = 0
}
crash> print $62->rb_left
$64 = (struct rb_node *) 0xffffb24941f4bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$65 = {
tree_entry = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
pi_tree_entry = {
__rb_parent_color = 18446658626441723201,
rb_right = 0xffffb24941f63d40,
rb_left = 0xffffb24941f5bd40
},
task = 0xffff956a594baf00,
lock = 0xffff956920b80190,
prio = 126,
deadline = 0
}
crash> print $62->rb_left->rb_left
$66 = (struct rb_node *) 0xffffb24941f5bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$67 = {
tree_entry = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
pi_tree_entry = {
__rb_parent_color = 18446658626441755968,
rb_right = 0xffffb24941f73d40,
rb_left = 0xffffb24941f2bd40
},
task = 0xffff956a64edaf00,
lock = 0xffff956a8d481670,
prio = 123,
deadline = 0
}
crash> print $62->rb_left->rb_left->rb_left
$68 = (struct rb_node *) 0xffffb24941f2bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$69 = {
tree_entry = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
pi_tree_entry = {
__rb_parent_color = 18446658626441821505,
rb_right = 0xffffb24940e47d40,
rb_left = 0xffffb24941f8bd40
},
task = 0xffff956a62ae0000,
lock = 0xffff956a76c5c250,
prio = 120,
deadline = 0
}
crash> print $62->rb_left->rb_left->rb_left->rb_left
$70 = (struct rb_node *) 0xffffb24941f8bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$71 = {
tree_entry = {
__rb_parent_color = 1,
rb_right = 0x0,
rb_left = 0x0
},
pi_tree_entry = {
__rb_parent_color = 18446658626441624896,
rb_right = 0x0,
rb_left = 0x0
},
task = 0xffff956a597dc680,
lock = 0xffff956920b802b0,
prio = 121,
deadline = 0
}
crash>
>From the last two rt_mutex_waiter we see that leftmost waiter's prio
is 121 but it's parent is 120.
Sorry for the inconvenience if I made a mistake. Thanks!
Henry
View attachment "pi.c" of type "text/x-csrc" (5424 bytes)
Powered by blists - more mailing lists