Message-ID: <2026012739-compile-bling-3779@gregkh>
Date: Tue, 27 Jan 2026 13:57:56 +0100
From: Greg KH <gregkh@...uxfoundation.org>
To: Geert Uytterhoeven <geert@...ux-m68k.org>
Cc: Tommaso Merciai <tommaso.merciai.xr@...renesas.com>,
	Marek Szyprowski <m.szyprowski@...sung.com>,
	Xin Zhao <jackzxcui1989@....com>, jirislaby@...nel.org,
	tj@...nel.org, hch@...radead.org, linux-kernel@...r.kernel.org,
	linux-serial@...r.kernel.org,
	Linux-Renesas <linux-renesas-soc@...r.kernel.org>
Subject: Re: [PATCH v8] tty: tty_port: add workqueue to flip TTY buffer

On Tue, Jan 27, 2026 at 01:02:44PM +0100, Geert Uytterhoeven wrote:
> On Tue, 27 Jan 2026 at 12:10, Tommaso Merciai
> <tommaso.merciai.xr@...renesas.com> wrote:
> > On Tue, Jan 27, 2026 at 11:34:32AM +0100, Marek Szyprowski wrote:
> > > On 23.12.2025 04:48, Xin Zhao wrote:
> > > > On the embedded platform, certain critical data, such as IMU data, is
> > > > transmitted through UART. The tty_flip_buffer_push() interface in the TTY
> > > > layer uses system_dfl_wq to handle the flipping of the TTY buffer.
> > > > Although the unbound workqueue can create new threads on demand and wake
> > > > up the kworker thread on an idle CPU, it may be preempted by real-time
> > > > tasks or other high-prio tasks.
> > > >
> > > > flush_to_ldisc() needs to wake up the relevant data handle thread. When
> > > > executing __wake_up_common_lock(), it calls spin_lock_irqsave(), which
> > > > does not disable preemption but disables migration in RT-Linux. This
> > > > prevents the kworker thread from being migrated to other cores by CPU's
> > > > balancing logic, resulting in long delays. The call trace is as follows:
> > > >      __wake_up_common_lock
> > > >      __wake_up
> > > >      ep_poll_callback
> > > >      __wake_up_common
> > > >      __wake_up_common_lock
> > > >      __wake_up
> > > >      n_tty_receive_buf_common
> > > >      n_tty_receive_buf2
> > > >      tty_ldisc_receive_buf
> > > >      tty_port_default_receive_buf
> > > >      flush_to_ldisc
> > > >
> > > > In our system, the processing interval for each frame of IMU data
> > > > transmitted via UART can experience significant jitter due to this issue.
> > > > Instead of the expected 10 to 15 ms frame processing interval, we see
> > > > spikes up to 30 to 35 ms. Moreover, in just one or two hours, there can
> > > > be 2 to 3 occurrences of such high jitter, which is quite frequent. This
> > > > jitter exceeds the software's tolerable limit of 20 ms.
> > > >
> > > > Introduce flip_wq in tty_port which can be set by tty_port_link_wq() or as
> > > > default linked to default workqueue allocated when tty_register_driver().
> > > > The default workqueue is allocated with flag WQ_SYSFS, so that cpumask and
> > > > nice can be set dynamically. The execution timing of tty_port_link_wq() is
> > > > not clearly restricted. The newly added function tty_port_link_driver_wq()
> > > > checks whether the flip_wq of the tty_port has already been assigned when
> > > > linking the default tty_driver's workqueue to the port. After the user has
> > > > set a custom workqueue for a certain tty_port using tty_port_link_wq(), the
> > > > system will only use this custom workqueue, even if tty_driver does not
> > > > have %TTY_DRIVER_CUSTOM_WORKQUEUE flag.
> > > >
> > > > Introduce %TTY_DRIVER_CUSTOM_WORKQUEUE flag meaning not to create the
> > > > default single tty_driver workqueue. Two reasons why need to introduce the
> > > > %TTY_DRIVER_CUSTOM_WORKQUEUE flag:
> > > > 1. If the WQ_SYSFS parameter is enabled, workqueue_sysfs_register() will
> > > > fail when trying to create a workqueue with the same name. The pty is an
> > > > example of this; if both CONFIG_LEGACY_PTYS and CONFIG_UNIX98_PTYS are
> > > > enabled, the call to tty_register_driver() in unix98_pty_init() will fail.
> > > > 2. Different tty ports may be used for different tasks, which may require
> > > > separate core binding control via workqueues. In this case, the workqueue
> > > > created by default in the tty driver is unnecessary. Enabling this flag
> > > > prevents the creation of this redundant workqueue.
> > > >
> > > > After applying this patch, we can set the related UART TTY flip buffer
> > > > workqueue by sysfs. We set the cpumask to CPU cores associated with the
> > > > IMU tasks, and set the nice to -20. Testing has shown significant
> > > > improvement in the previously described issue, with almost no stuttering
> > > > occurring anymore.
> > > >
> > > > Signed-off-by: Xin Zhao <jackzxcui1989@....com>
> > >
> > > This patch landed in linux-next as commit d000422a46aa ("tty: tty_port:
> > > add workqueue to flip TTY buffer"). In my tests I found that it causes
> > > some regressions, see the comments in the code below.
> >
> > Same here, testing on RZ/G3E looks like s2idle is broken:
> 
> > [  185.237717] Call trace:
> > [  185.240176]  __queue_work+0x20/0x474 (P)
> > [  185.244141]  queue_work_on+0x8c/0xa8
> > [  185.247753]  tty_flip_buffer_push+0x2c/0x38
> 
> Lucky you, there is a hint to tty in your trace ;-)
> 
> I see a similar crash during boot on koelsch (R-Car M2-W), and a
> lock-up during boot on salvator-xs (R-Car H3 ES2.0), with either no
> output or an rcu stall:
> 
>     rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
>     rcu:     1-...!: (0 ticks this GP) idle=1fe8/0/0x0 softirq=85/85
> fqs=1 (false positive?)
>     rcu:     2-...!: (1 ticks this GP) idle=1c78/0/0x0 softirq=77/77
> fqs=1 (false positive?)
>     rcu:     6-...!: (0 ticks this GP) idle=07b8/0/0x0 softirq=9/9
> fqs=1 (false positive?)
>     rcu:     (detected by 3, t=5260 jiffies, g=-1015, q=274 ncpus=8)
>     Sending NMI from CPU 3 to CPUs 1:
>     Sending NMI from CPU 3 to CPUs 2:
>     Sending NMI from CPU 3 to CPUs 6:
>     rcu: rcu_preempt kthread timer wakeup didn't happen for 12771
> jiffies! g-1015 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x200
>     rcu:     Possible timer handling issue on cpu=6 timer-softirq=1
>     rcu: rcu_preempt kthread starved for 12780 jiffies! g-1015 f0x0
> RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=6
>     rcu:     Unless rcu_preempt kthread gets sufficient CPU time, OOM
> is now expected behavior.
>     rcu: RCU grace-period kthread stack dump:
>     task:rcu_preempt     state:R stack:0     pid:15    tgid:15
> ppid:2      task_flags:0x208040 flags:0x00000010
>     Call trace:
>      __switch_to+0xcc/0x100 (T)
>      __schedule+0x368/0xc00
>      schedule+0x30/0x100
>      schedule_timeout+0x80/0xf8
>      rcu_gp_fqs_loop+0xfc/0x418
>      rcu_gp_kthread+0xe0/0xf4
>      kthread+0x128/0x1e0
>      ret_from_fork+0x10/0x20
> 
> Reverting commit d000422a46aad322 ("tty: tty_port: add workqueue to
> flip TTY buffer") in tty-next fixes both.

Thanks all for the reports, I'll go revert this for now, sorry for the
problems!

greg k-h

Powered by blists - more mailing lists