linux-kernel - Re: [PATCH for-3.20-fixes] workqueue: fix hang involving racing cancel[_delayed]_work_sync()'s for PREEMPT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 3 Mar 2015 08:21:15 -0500
From:	Tejun Heo <tj@...nel.org>
To:	Tomeu Vizoso <tomeu.vizoso@...il.com>
Cc:	Jesper Nilsson <jesper.nilsson@...s.com>,
	Rabin Vincent <rabinv@...s.com>,
	Jesper Nilsson <jespern@...s.com>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH for-3.20-fixes] workqueue: fix hang involving racing
 cancel[_delayed]_work_sync()'s for PREEMPT_NONE

Hello, Tomeu.

On Tue, Mar 03, 2015 at 11:00:43AM +0100, Tomeu Vizoso wrote:
...
> [    7.421803] PC is at wake_bit_function+0x18/0x6c
> [    7.428168] LR is at __wake_up_common+0x5c/0x90
...
> [    7.673183] [<c028f54c>] (wake_bit_function) from [<c028ed3c>] (__wake_up_common+0x5c/0x90)
> [    7.683300] [<c028ed3c>] (__wake_up_common) from [<c028ef88>] (__wake_up+0x48/0x5c)
> [    7.692744] [<c028ef88>] (__wake_up) from [<c026fdfc>] (__cancel_work_timer+0xe8/0x1ac)
> [    7.702533] [<c026fdfc>] (__cancel_work_timer) from [<c026fedc>] (cancel_work_sync+0x1c/0x20)
> [    7.712854] [<c026fedc>] (cancel_work_sync) from [<c02ce3bc>] (css_free_work_fn+0x174/0x2ec)
> [    7.723099] [<c02ce3bc>] (css_free_work_fn) from [<c026f4d0>] (process_one_work+0x15c/0x3dc)
> [    7.733339] [<c026f4d0>] (process_one_work) from [<c026ff6c>] (worker_thread+0x54/0x4e8)
> [    7.743224] [<c026ff6c>] (worker_thread) from [<c02749d8>] (kthread+0xec/0x104)
> [    7.752339] [<c02749d8>] (kthread) from [<c0210aa0>] (ret_from_fork+0x14/0x34)
> [    7.761366] Code: e24cb004 e52de004 e8bd4000 e510400c (e5935000)

Hah, weird.  How is your machine ending up in wake_bit_function() from
there?  Can you please apply the following patch and report the dmesg
after crash?

Thanks.

---
 kernel/workqueue.c |   16 ++++++++++++++--
 1 file changed, 14 insertions(+), 2 deletions(-)

--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -2755,8 +2755,12 @@ static bool __cancel_work_timer(struct w
 		if (unlikely(ret == -ENOENT)) {
 			DEFINE_WAIT(wait);
 			prepare_to_wait(waitq, &wait, TASK_UNINTERRUPTIBLE);
-			if (work_is_canceling(work))
+			WARN_ON_ONCE(wait.func != autoremove_wake_function);
+			if (work_is_canceling(work)) {
+				printk("XXX __cancel_work_timer: %d sleeping w/ %pf, wait=%p\n",
+				       task_pid_nr(current), wait.func, &wait);
 				schedule();
+			}
 			finish_wait(waitq, &wait);
 		}
 	} while (unlikely(ret < 0));
@@ -2774,8 +2778,16 @@ static bool __cancel_work_timer(struct w
 	 * visible there.
 	 */
 	smp_mb();
-	if (waitqueue_active(waitq))
+	if (waitqueue_active(waitq)) {
+		wait_queue_t *cur;
+
+		spin_lock_irq(&waitq->lock);
+		list_for_each_entry(cur, &waitq->task_list, task_list)
+			printk("XXX __cancel_work_timer: waking up %pf, wait=%p\n",
+			       cur->func, cur);
+		spin_unlock_irq(&waitq->lock);
 		wake_up(waitq);
+	}
 
 	return ret;
 }

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/