Message-ID: <dc42f5d4-a707-4442-bda6-1c1990666f54@zytor.com>
Date: Sat, 8 Nov 2025 18:25:20 -0800
From: "H. Peter Anvin" <hpa@...or.com>
To: "Theodore Ts'o" <tytso@....edu>
Cc: linux-serial@...r.kernel.org, linux-api@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: RFC: Serial port DTR/RTS - O_NRESETDEV

On 2025-11-07 09:37, Theodore Ts'o wrote:
> On Thu, Nov 06, 2025 at 11:53:23PM -0800, H. Peter Anvin wrote:
>>
>> I recently ran into a pretty serious issue due to the Unix/Linux
>> (mis)behavior of forcing DTR and RTS asserted when a serial port is
>> set, losing the pre-existing status in the process.
> 
> There's a hidden assumption in your problem statement which is that
> DTR / RTS has a "state" which can be saved when the serial port is not
> active, where active is one or more file descriptors holding the
> serial port open.  There may be certain hardware or drivers where this
> is just not possible, because nothing is defined if the serial port is
> not active.  It might make sense if you are using a 8250 UART, but not
> all the world is the National Semiconductor (or clones) UART.
> 
> Certainly the "state" will not be preserved across boots, since how we
> autodetect the UART is going to mess with UART settings.  So
> *presumably* what you are talking about is you want to be able to open
> the serial port, mess with DTR / RTS, and then be able to close the
> serial port, and then later on, re-open the serial port, have the DTR
> / RTS remain the same.  And it's Too Hard(tm) to have userspace
> keeping a file descriptor open during the whole time?  (Which is
> traditionally how Unix/Linux has required that applications do
> things.)
> 
> Is that a fair summary of the requirements?
> 

Not really.

First of all, obviously virtual serial connections that don't fully emulate
RS232/422/485 obviously may have other requirements, however, the ones that
*do* currently have a problem, see for example:

https://lore.kernel.org/linux-serial/20220531043356.8CAB637401A9@freecalypso.org/

RS232 is rarely used these days with its original purpose, modems (except as a
virtual port for things like GSM), but the ubiquitousness of the interface
means it is used for a ton of other things.

The standard ESP32 configuration for its serial port is that asserting RTS#
even for a moment will cause a device reset, and asserting DTR# during reset
forces the device into boot mode. So even if you execute TIOCMSET immediately
after opening the device, you will have glitched the output, and only the
capacitance of the output will save you, in the best case.

The use of RTS# and DTR# as a reset and/or debug mode entry for embedded
devices is, in fact, extremely common; another example is the Atmel single pin
reset/debug interface.

Another example is when the device is connected to an RS485 interface: in that
case asserting RTS# will activate the transmitter, disrupting traffic on the
bus. The kernel will manage RTS# *once it has been configured*, but until the
kernel has been told that the port is used to drive an RS485 port, it has no
way to know.

Furthermore, *even if* the kernel already knew the state and could have
reported it with TIOCMGET, that state is now lost.

It is not correct that the state cannot be maintained across system reboots
(without power loss.) The hardware may or may not allow the state to be read
back (notably, the USB CDC ACM specification, oddly enough, has a
GET_LINE_CODING command but no GET_CONTROL_LINE_STATE) but again, for *those
that can* it should be possible. For those that cannot, there won't be any way
to get valid data to TIOCMGET, but it is still possible to not send a change
command. Either way, the power up state of write-only devices can generally be
assumed to be RTS# and DTR# deasserted, not asserted.

(USB is also a bit special because it is normal for the USB host to power
cycle the device during bus initialization.)

>> It seems to me that this may very well be a problem beyond ttys, in
>> which case a new open flag to request to a driver that the
>> configuration and (observable) state of the underlying hardware
>> device -- whatever it may be -- should not be disturbed by calling
>> open(). This is of course already the case for many devices, not to
>> mention block and non-devices, in which case this flag is a don't
>> care.
> 
> I think it's going to be a lot simpler to keep this specific to serial
> ports and DTR / RTS, because the concept that the hardware should not
> be changed when the file descriptor is opened may simply not be
> possible.  For example, it might be that until you open it, the there
> might not even be power applied to the device.  The concept that all
> hardware should burn battery power once the machine is booted may not
> make sense, and the assumption that hardware has the extra
> millicent(s) worth of silicon to maintain state when power is dropped
> may again, not be something that we can assume as being possible for
> all devices.

This is actually a great example! One should be able to open a file descriptor
to such a device to configure the driver, without needing to power up the
physical hardware.

However, I intentionally defined this as a best-effort control for two reasons:

1. As you say, the hardware may not be able to do it;
2. It will take time until a significant set of drivers can implement this.

> If that's the case, if you want to have something where DTR and RTS
> stay the same, and for some reason we can't assume that userspace
> can't just keep a process holding the tty device open, my suggestion is to use 
> 
> Given that DTR and RTS are secial port concepts, my suggesiton is to
> set a serial port flag, using setserial(8).  It may be the case that
> for certain types of serial device, the attempt to set the flag may be
> rejected, but that's something which the ioctl used by setserial
> already can do and which userspace applications such as setserial
> understand may be the case.

setserial (TIOCSSERIAL) and termios (TCSETS*) both require file descriptors,
so that is not suitable. The 8250 driver, but *not* other serial drivers,
allows the setserial information to be accessed via sysfs; however, this
functionality is local to the 8250 driver.

(Incidentally: the only way to find out the type of a tty driver in the
current Linux kernel is to parse /proc/tty/drivers. This information is
neither available in sysfs nor via ioctl.

Consider the case of a terminal program wanting to display a list of serial
ports. Right now, some serial drivers -- notably the generic UART driver --
will create device nodes for all available minors, exactly so you would be
able to manually attach a device with TIOCSSERIAL. There is no driver-generic
way to find out if there is a hardware device configured other than opening
the device and calling TCGETS or TIOCMGET (depending on exactly what you are
looking for) and see if you get EIO back.

The problem here really isn't the need for a file descriptor -- file
descriptors are The Unix Way[TM] to refer to almost any kind of entity after
all -- but that the act of obtaining the file descriptor -- open -- causes a
direct action as well as loss of existing state.

Now, we obviously can't disable the classical terminal behavior
unconditionally -- that would break a whole lot of perfectly valid code.

Using a sysfs attribute is reasonable on the surface of it (and is what the
patchset linked to above implements) I believe this is the wrong approach,
because it is modal on the device level, and that makes it racy: one program
comes in, flips the attribute, then another program comes in and tries to open
the same device for whatever reason. This opens up at least three possible
race conditions:

- Process 1 sets the nreset bit;
- Process 2 opens the device, not expecting the nreset bit.

- Process 1 reads the nreset bit, trying to be a good citizen;
- Process 1 sets the nreset bit;
- Process 2 reads the nreset bit, ditto;
- ... other stuff happens ...
- Process 1 restores the nreset bit
- Process 2 restores the nreset bit, incorrectly setting it to 1

- Process 1 sets the nreset bit;
- Process 2 sets the nreset bit;
- Process 1 does its work;
- Process 1 clears the nreset bit;
- Process 2 opens the device.


Oh, yes, let's not forget: you need a file descriptor to lock the device for
exclusive use.

This is why I believe this:

1. Needs to be atomic with open(), as opposed to tied to per-device state;

2. Likely can have applications beyond the serial port space, and it would
   be a good idea to have a uniform interface. We can discuss the proper
   behavior for a device which cannot comply; the best way probably would be
   to refuse the open and return an error, which would also happen on older
   kernels without this functionality.

Does this make more sense?

	-hpa

Powered by blists - more mailing lists