linux-kernel - io_uring : deadlock between io

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <3d67a75b-0819-4f0e-87b6-99dd836e88e9@shopee.com>
Date: Tue, 25 Feb 2025 15:42:56 +0800
From: Haifeng Xu <haifeng.xu@...pee.com>
To: Pavel Begunkov <asml.silence@...il.com>, Jens Axboe <axboe@...nel.dk>,
 "Eric W. Biederman" <ebiederm@...ssion.com>,
 Olivier Langlois <olivier@...llion01.com>, io-uring@...r.kernel.org,
 linux-kernel@...r.kernel.org
Subject: io_uring : deadlock between io_uring and coredump

Hi masters,
	In our production environment, we found many hung tasks.

	Thead A (exit_mm)
	...
		if (core_state) {
		struct core_thread self;

		mmap_read_unlock(mm);

		self.task = current;
		if (self.task->flags & PF_SIGNALED)
			self.next = xchg(&core_state->dumper.next, &self);
		else
			self.task = NULL;
		/*
		 * Implies mb(), the result of xchg() must be visible
		 * to core_state->dumper.
		 */
		if (atomic_dec_and_test(&core_state->nr_threads))
			complete(&core_state->startup);

		for (;;) {
			set_current_state(TASK_UNINTERRUPTIBLE);
			if (!self.task) /* see coredump_finish() */
				break;
			freezable_schedule();
		}
		__set_current_state(TASK_RUNNING);
		mmap_read_lock(mm);
	}
	...

	Thead B (coredump_wait)
	...
		if (core_waiters > 0) {
		struct core_thread *ptr;

		freezer_do_not_count();
		wait_for_completion(&core_state->startup);
		freezer_count();
		/*
		 * Wait for all the threads to become inactive, so that
		 * all the thread context (extended register state, like
		 * fpu etc) gets copied to the memory.
		 */
		ptr = core_state->dumper.next;
		while (ptr != NULL) {
			wait_task_inactive(ptr->task, 0);
			ptr = ptr->next;
		}
	...

	Thead C (io_worker_exit)
	...
		if (refcount_dec_and_test(&worker->ref))
		complete(&worker->ref_done);
		wait_for_completion(&worker->ref_done);
	...
		

	Thread A is waiting Thead B to finish core dump, but Thead B found that there is still one thread which doesn't 
	step into exit_mm() to dec core_state->nr_threads. The thead is Thread C, it has submitted a task_work (create_worker_cb)
	to Thread B and then wait Thread B to execute or cancel the work. So this leads to a deadlock.

	Our kernel vesion is stable 5.15.125, and the commit 1d5f5ea7cb7d ("io-wq: remove worker to owner tw dependency") is included.
	when the last io woker exits, it doesn't find any callback. Once scheduled out, it will invoke io_wq_worker_sleeping
	to submit a task work to the master thread. If the above guess is right, the commit 1d5f5ea7cb7d won't help.

	Can we cancel the io_uring requests before dumping core?

Thanks!