Message-ID: <19c22785df1.288e39fb101919.2611884700541801815@linux.beauty>
Date: Tue, 03 Feb 2026 15:47:19 +0800
From: Li Chen <me@...ux.beauty>
To: "Jens Axboe" <axboe@...nel.dk>
Cc: "Pavel Begunkov" <asml.silence@...il.com>,
	"io-uring" <io-uring@...r.kernel.org>,
	"linux-kernel" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v1 0/2] io_uring/io-wq: let workers exit when unused

Hi Jens,

 ---- On Tue, 03 Feb 2026 10:29:50 +0800  Jens Axboe <axboe@...nel.dk> wrote --- 
 > On 2/2/26 5:37 PM, Li Chen wrote:
 > > Hi Jens,
 > > 
 > >  ---- On Mon, 02 Feb 2026 23:21:22 +0800  Jens Axboe <axboe@...nel.dk> wrote --- 
 > >  > On 2/2/26 7:37 AM, Li Chen wrote:
 > >  > > io_uring uses io-wq to offload regular file I/O. When that happens, the kernel
 > >  > > creates per-task iou-wrk-<tgid> workers (PF_IO_WORKER) via create_io_thread(),
 > >  > > so the worker is part of the process thread group and shows up under
 > >  > > /proc/<pid>/task/.
 > >  > > 
 > >  > > io-wq shrinks the pool on idle, but it intentionally keeps the last worker
 > >  > > around indefinitely as a keepalive to avoid churn. Combined with io_uring's
 > >  > > per-task context lifetime (tctx stays attached to the task until exit), a
 > >  > > process may permanently retain an idle iou-wrk thread even after it has closed
 > >  > > its last io_uring instance and has no active rings.
 > >  > > 
 > >  > > The keepalive behavior is a reasonable default(I guess): workloads may have
 > >  > > bursty I/O patterns, and always tearing down the last worker would add thread
 > >  > > churn and latency. Creating io-wq workers goes through create_io_thread()
 > >  > > (copy_process), which is not cheap to do repeatedly.
 > >  > > 
 > >  > > However, CRIU currently doesn't cope well with such workers being part of the
 > >  > > checkpointed thread group. The iou-wrk thread is a kernel-managed worker
 > >  > > (PF_IO_WORKER) running io_wq_worker() on a kernel stack, rather than a normal
 > >  > > userspace thread executing application code. In our setup, if the iou-wrk
 > >  > > thread remains present after quiescing and closing the last io_uring instance,
 > >  > > criu dump may hang while trying to stop and dump the thread group.
 > >  > > 
 > >  > > Besides the resource overhead and surprising userspace-visible threads, this is
 > >  > > a problem for checkpoint/restore. CRIU needs to freeze and dump all threads in
 > >  > > the thread group. With a lingering iou-wrk thread, we observed criu dump can
 > >  > > hang even after the ring has been quiesced and the io_uring fd closed, e.g.:
 > >  > > 
 > >  > >   criu dump -t $PID -D images -o dump.log -v4 --shell-job
 > >  > >   ps -T -p $PID -o pid,tid,comm | grep iou-wrk
 > >  > > 
 > >  > > This series is a kernel-side enabler for checkpoint/restore in the current
 > >  > > reality where userspace needs to quiesce and close io_uring rings before dump.
 > >  > > It is not trying to make io_uring rings checkpointable, nor does it change what
 > >  > > CRIU can or cannot restore (e.g. in-flight SQEs/CQEs, SQPOLL, SQE128/CQE32,
 > >  > > registered resources). Even with userspace gaining limited io_uring support,
 > >  > > this series only targets the specific "no active io_uring contexts left, but an
 > >  > > idle iou-wrk keepalive thread remains" case.
 > >  > > 
 > >  > > This series adds an explicit exit-on-idle mode to io-wq, and toggles it from
 > >  > > io_uring task context when the task has no active io_uring contexts
 > >  > > (xa_empty(&tctx->xa)). The mode is cleared on subsequent io_uring usage, so the
 > >  > > default behavior for active io_uring users is unchanged.
 > >  > > 
 > >  > > Tested on x86_64 with CRIU 4.2.
 > >  > > With this series applied, after closing the ring iou-wrk exited within ~200ms
 > >  > > and criu dump completed.
 > >  > 
 > >  > Applied with the mentioned commit message and IO_WQ_BIT_EXIT_ON_IDLE test
 > >  > placement.
 > > 
 > > Thanks a lot for your review!
 > > 
 > > If you still want a test, I'm happy to write it. Since you've already
 > > tweaked/applied the v1 series, I can send the test as a standalone
 > > follow-up patch (no v2).
 > > 
 > > If kselftest is preferred, I'll base it on the same CRIU-style workload:
 > > spawn iou-wrk-* via io_uring, quiesce/close the last ring, and check the
 > > worker exits within a short timeout.
 > 
 > That sounds like the right way to do the test. Preferably a liburing
 > test/ case would be better, we don't do a lot of in-kernel selftests so
 > far. But liburing has everything.

Thanks for your suggestion. I just adapted my local test program to liburing and posted the liburing
 PR here: https://github.com/axboe/liburing/pull/1529

Regards,
Li​

Powered by blists - more mailing lists