lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAG-UpRQsdL_Fs9HSEv2pDYXehJC+YXcYjiZKFLvkGBTZkkaTcg@mail.gmail.com>
Date:   Thu, 6 Jul 2023 14:08:20 +0800
From:   Henry Wu <triangletrap12@...il.com>
To:     linux-kernel@...r.kernel.org
Cc:     peterz@...radead.org
Subject: Possible race in rt_mutex_adjust_prio_chain

Hi,

I found that it's not safe to change waiter->prio after waiter
dequeued from mutex's waiter rbtree because it's still on owner's
pi_waiters rbtree. From my analysis, waiters on pi_waiters rbtree
should be protected by pi_lock of task which have pi_waiters and
corresponding rt_mutex's wait_lock, but pi_waiters is shared by many
locks owned by this task, so actually we serialize changes on
pi_waiters only by pi_lock.

`rt_mutex_adjust_prio_chain' changes key of nodes of pi_waiters rbtree
without pi_lock and pi_waiters rbtree's invariant is violated. Maybe
we are enqueuing waiter on other cpu and pi_waiters rbtree will be
corrupted.

I attach a source file which can trigger this violation. I tested it
on Ubuntu 20.04 LTS with 5.4 kernel.

$ sudo ./a.out
pid: 39949
prio: 39
    PID     LWP TTY          TIME CMD
  39949   39949 pts/1    00:00:00 a.out
  39949   39950 pts/1    00:00:00 waiter-0
  39949   39951 pts/1    00:00:00 waiter-1
.............
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
prio: 20
.............
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
prio: 21
    PID     LWP TTY          TIME CMD
  39949   39949 pts/1    00:00:00 a.out
  39949   39950 pts/1    00:00:00 waiter-0
  39949   39951 pts/1    00:00:00 waiter-1
  39949   39952 pts/1    00:00:00 waiter-2
  39949   39953 pts/1    00:00:00 waiter-3
  39949   39954 pts/1    00:00:00 waiter-4
  39949   39955 pts/1    00:00:00 waiter-5
  39949   39956 pts/1    00:00:00 waiter-6
  39949   39957 pts/1    00:00:00 waiter-7
  39949   39958 pts/1    00:00:00 waiter-8
  39949   39959 pts/1    00:00:00 waiter-9
  39949   39960 pts/1    00:00:00 waiter-10
  39949   39961 pts/1    00:00:00 waiter-11
  39949   39962 pts/1    00:00:00 waiter-12
  39949   39963 pts/1    00:00:00 waiter-13
  39949   39964 pts/1    00:00:00 waiter-14
  39949   39965 pts/1    00:00:00 waiter-15
  39949   39966 pts/1    00:00:00 waiter-16
  39949   39967 pts/1    00:00:00 waiter-17
  39949   39968 pts/1    00:00:00 waiter-18
  39949   39969 pts/1    00:00:00 changer-0
  39949   39970 pts/1    00:00:00 changer-1
  39949   39971 pts/1    00:00:00 changer-2
  39949   39972 pts/1    00:00:00 changer-3
  39949   39973 pts/1    00:00:00 changer-4
  39949   39974 pts/1    00:00:00 changer-5
  39949   39975 pts/1    00:00:00 changer-6
  39949   39976 pts/1    00:00:00 changer-7
  39949   39977 pts/1    00:00:00 changer-8
  39949   39978 pts/1    00:00:00 changer-9
  39949   39979 pts/1    00:00:00 changer-10
  39949   39980 pts/1    00:00:00 changer-11
  39949   39981 pts/1    00:00:00 changer-12
  39949   39982 pts/1    00:00:00 changer-13
  39949   39983 pts/1    00:00:00 changer-14
  39949   39984 pts/1    00:00:00 changer-15
  39949   39985 pts/1    00:00:00 changer-16
  39949   39986 pts/1    00:00:00 changer-17
  39949   39987 pts/1    00:00:00 changer-18
found race, hang...

$ crash
crash> task -R prio,normal_prio,pi_waiters 39949
PID: 39949  TASK: ffff956a5c8b2f00  CPU: 2   COMMAND: "a.out"
  prio = 121,
  normal_prio = 139,
  pi_waiters = {
    rb_root = {
      rb_node = 0xffffb24941f43d40
    },
    rb_leftmost = 0xffffb24941f8bd40
  },

crash> print (struct rb_node *)0xffffb24941f43d40
$62 = (struct rb_node *) 0xffffb24941f43d40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$63 = {
  tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0x0,
    rb_left = 0x0
  },
  pi_tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0xffffb24941f13d40,
    rb_left = 0xffffb24941f4bd40
  },
  task = 0xffff956a6c008000,
  lock = 0xffff956920b80970,
  prio = 130,
  deadline = 0
}
crash> print $62->rb_left
$64 = (struct rb_node *) 0xffffb24941f4bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$65 = {
  tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0x0,
    rb_left = 0x0
  },
  pi_tree_entry = {
    __rb_parent_color = 18446658626441723201,
    rb_right = 0xffffb24941f63d40,
    rb_left = 0xffffb24941f5bd40
  },
  task = 0xffff956a594baf00,
  lock = 0xffff956920b80190,
  prio = 126,
  deadline = 0
}
crash> print $62->rb_left->rb_left
$66 = (struct rb_node *) 0xffffb24941f5bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$67 = {
  tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0x0,
    rb_left = 0x0
  },
  pi_tree_entry = {
    __rb_parent_color = 18446658626441755968,
    rb_right = 0xffffb24941f73d40,
    rb_left = 0xffffb24941f2bd40
  },
  task = 0xffff956a64edaf00,
  lock = 0xffff956a8d481670,
  prio = 123,
  deadline = 0
}
crash> print $62->rb_left->rb_left->rb_left
$68 = (struct rb_node *) 0xffffb24941f2bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$69 = {
  tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0x0,
    rb_left = 0x0
  },
  pi_tree_entry = {
    __rb_parent_color = 18446658626441821505,
    rb_right = 0xffffb24940e47d40,
    rb_left = 0xffffb24941f8bd40
  },
  task = 0xffff956a62ae0000,
  lock = 0xffff956a76c5c250,
  prio = 120,
  deadline = 0
}
crash> print $62->rb_left->rb_left->rb_left->rb_left
$70 = (struct rb_node *) 0xffffb24941f8bd40
crash> print *(struct rt_mutex_waiter *)((void *)$ - 24)
$71 = {
  tree_entry = {
    __rb_parent_color = 1,
    rb_right = 0x0,
    rb_left = 0x0
  },
  pi_tree_entry = {
    __rb_parent_color = 18446658626441624896,
    rb_right = 0x0,
    rb_left = 0x0
  },
  task = 0xffff956a597dc680,
  lock = 0xffff956920b802b0,
  prio = 121,
  deadline = 0
}
crash>

>From the last two rt_mutex_waiter we see that leftmost waiter's prio
is 121 but it's parent is 120.

Sorry for the inconvenience if I made a mistake. Thanks!

Henry

View attachment "pi.c" of type "text/x-csrc" (5424 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ