linux-kernel - Re: [PREEMPT_RT] 8250 IRQ lockup when flooding serial console (was Re: [ANNOUNCE] v5.4.28-rt19)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200424191945.an42attvo7pdt3qz@linutronix.de>
Date:   Fri, 24 Apr 2020 21:19:45 +0200
From:   Sebastian Andrzej Siewior <bigeasy@...utronix.de>
To:     Jiri Kosina <jikos@...nel.org>
Cc:     Thomas Gleixner <tglx@...utronix.de>,
        LKML <linux-kernel@...r.kernel.org>,
        linux-rt-users <linux-rt-users@...r.kernel.org>,
        Steven Rostedt <rostedt@...dmis.org>,
        Frederic Weisbecker <fweisbec@...il.com>,
        Matt Fleming <matt@...eblueprint.co.uk>,
        Daniel Wagner <dwagner@...e.de>
Subject: Re: [PREEMPT_RT] 8250 IRQ lockup when flooding serial console (was
 Re: [ANNOUNCE] v5.4.28-rt19)

On 2020-04-23 18:07:07 [+0200], To Jiri Kosina wrote:
> The IRQ4 is edge and in charge of ttyS0. It is handled by
> handle_edge_irq() and after ->irq_ack(), the thread is woken up and then
> we get another ->handle_edge_irq() for IRQ4. With larger PASS_LIMIT the
> thread runs longer so note_interrupt() will make less IRQ_HANDLED based
> on ->threads_handled_last. If it observes 100 handled within 100000
> interrupts then the counters are reset again. On !RT it usually manages
> to get >100 per 100000 interrupts so it appears good. On RT it gets less
> and the interrupt gets disabled.

so the EDGE interrupt shouldn't retrigger unless something happened that
*retriggers* the situation. I think the problem is that qemu is too
fast. Based on tracing: The IRQ happens, the threaded-handler fires up.
The threaded handler writes something into the TX FIFO. QEMU probably
transmits that byte immediately so the FIFO is empty again *and* another
interrupt is fired. It makes sense because I see one one byte written to
the FIFO followed by an interrupt, followed by another byte, followed by
an interrupt, ….
This does not happen on real hardware because it takes some time to
transmit the data so.

I already dropped PASS_LIMIT from the RT queue so it might not trigger
again. I think I'm going to make the chain-handler depend on ISA so it
the thread is invoked more often.
According to the comment it is/was required to handle some ISA issues.
I'm not sure if back then there was no SA_SHARED handling in the IRQ
core was limited or $REASON. As of today I don't see a reason to keep it
for !ISA hardware. On Debian CONFIG_ISA_BUS is disabled, I guess the
same is true for SUSE.

And with the hunk at the bottom it shouldn't trigger as well. So
PASS_LIMIT and no custom handler is duct tape. This really stops the IRQ
from being generated.

diff --git a/drivers/tty/serial/8250/8250_port.c b/drivers/tty/serial/8250/8250_port.c
index e31217e8dce62..eadb12bf0e90b 100644
--- a/drivers/tty/serial/8250/8250_port.c
+++ b/drivers/tty/serial/8250/8250_port.c
@@ -1741,6 +1741,7 @@ void serial8250_tx_chars(struct uart_8250_port *up)
 	}

 	count = up->tx_loadsz;
+	serial8250_clear_THRI(up);
 	do {
 		serial_out(up, UART_TX, xmit->buf[xmit->tail]);
 		xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
@@ -1755,6 +1756,7 @@ void serial8250_tx_chars(struct uart_8250_port *up)
 		    !(serial_in(up, UART_LSR) & UART_LSR_THRE))
 			break;
 	} while (--count > 0);
+	serial8250_set_THRI(up);

 	if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
 		uart_write_wakeup(port);

Sebastian