linux-kernel - Re: RFC: Serial port DTR/RTS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20251107173743.GA3131573@mit.edu>
Date: Fri, 7 Nov 2025 12:37:43 -0500
From: "Theodore Ts'o" <tytso@....edu>
To: "H. Peter Anvin" <hpa@...or.com>
Cc: linux-serial@...r.kernel.org, linux-api@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: RFC: Serial port DTR/RTS - O_NRESETDEV

On Thu, Nov 06, 2025 at 11:53:23PM -0800, H. Peter Anvin wrote:
> 
> I recently ran into a pretty serious issue due to the Unix/Linux
> (mis)behavior of forcing DTR and RTS asserted when a serial port is
> set, losing the pre-existing status in the process.

There's a hidden assumption in your problem statement which is that
DTR / RTS has a "state" which can be saved when the serial port is not
active, where active is one or more file descriptors holding the
serial port open.  There may be certain hardware or drivers where this
is just not possible, because nothing is defined if the serial port is
not active.  It might make sense if you are using a 8250 UART, but not
all the world is the National Semiconductor (or clones) UART.

Certainly the "state" will not be preserved across boots, since how we
autodetect the UART is going to mess with UART settings.  So
*presumably* what you are talking about is you want to be able to open
the serial port, mess with DTR / RTS, and then be able to close the
serial port, and then later on, re-open the serial port, have the DTR
/ RTS remain the same.  And it's Too Hard(tm) to have userspace
keeping a file descriptor open during the whole time?  (Which is
traditionally how Unix/Linux has required that applications do
things.)

Is that a fair summary of the requirements?

> It seems to me that this may very well be a problem beyond ttys, in
> which case a new open flag to request to a driver that the
> configuration and (observable) state of the underlying hardware
> device -- whatever it may be -- should not be disturbed by calling
> open(). This is of course already the case for many devices, not to
> mention block and non-devices, in which case this flag is a don't
> care.

I think it's going to be a lot simpler to keep this specific to serial
ports and DTR / RTS, because the concept that the hardware should not
be changed when the file descriptor is opened may simply not be
possible.  For example, it might be that until you open it, the there
might not even be power applied to the device.  The concept that all
hardware should burn battery power once the machine is booted may not
make sense, and the assumption that hardware has the extra
millicent(s) worth of silicon to maintain state when power is dropped
may again, not be something that we can assume as being possible for
all devices.

If that's the case, if you want to have something where DTR and RTS
stay the same, and for some reason we can't assume that userspace
can't just keep a process holding the tty device open, my suggestion is to use 

Given that DTR and RTS are secial port concepts, my suggesiton is to
set a serial port flag, using setserial(8).  It may be the case that
for certain types of serial device, the attempt to set the flag may be
rejected, but that's something which the ioctl used by setserial
already can do and which userspace applications such as setserial
understand may be the case.

Cheers,

						- Ted