linux-kernel - Re: RFC: Serial port DTR/RTS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20251110033556.GC2988753@mit.edu>
Date: Sun, 9 Nov 2025 22:35:56 -0500
From: "Theodore Ts'o" <tytso@....edu>
To: "H. Peter Anvin" <hpa@...or.com>
Cc: linux-serial@...r.kernel.org, linux-api@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: RFC: Serial port DTR/RTS - O_NRESETDEV

On Sat, Nov 08, 2025 at 06:25:20PM -0800, H. Peter Anvin wrote:
> 
> The standard ESP32 configuration for its serial port is that asserting RTS#
> even for a moment will cause a device reset, and asserting DTR# during reset
> forces the device into boot mode. So even if you execute TIOCMSET immediately
> after opening the device, you will have glitched the output, and only the
> capacitance of the output will save you, in the best case.

IMHO, these more esoteric use cases should involve a custom kernel
driver which replaces the generic serial driver.  In practice, these
things aren't really a tty, but somethiung else weird, and trying to
do this in userspace seems really awkward.

> setserial (TIOCSSERIAL) and termios (TCSETS*) both require file descriptors,
> so that is not suitable. The 8250 driver, but *not* other serial drivers,
> allows the setserial information to be accessed via sysfs; however, this
> functionality is local to the 8250 driver.

My suggestion of using setserial to turn on some "not really a tty;
but some weird networking / cheap debugging hack" flag should work,
because you would do this at boot up.  Note that the 8250
autoconfiguration code (see drivers/tty/serial/8250/8250_port.c) is
going to mess with DTR / RTS.  This is why I asserted that trying to
claim that you can preserve "state" across reboots is Just Not
Possible.

If you have some weird setup where DTR or RTS is wierd to the
"detonate the TNT" line, might I suggest that maybe we shouldn't be
using the tty / 8250 serial driver, but it should ***really*** be a
dedicated kernel driver?

					- Ted