linux-kernel - RFC: Serial port DTR/RTS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <bb44f856-10a2-40c7-a3f7-be50c8e4b0a9@zytor.com>
Date: Thu, 6 Nov 2025 23:53:23 -0800
From: "H. Peter Anvin" <hpa@...or.com>
To: linux-serial@...r.kernel.org, linux-api@...r.kernel.org,
        LKML <linux-kernel@...r.kernel.org>
Subject: RFC: Serial port DTR/RTS - O_NRESETDEV

Hi,

I recently ran into a pretty serious issue due to the Unix/Linux (mis)behavior
of forcing DTR and RTS asserted when a serial port is set, losing the
pre-existing status in the process. Since it is impossible probe for the
current status or even if it is a functional serial port without a file
descriptor, this is very problematic. This came up in the context of probing
for serial ports from an application, so even if termios could be modified
without a file descriptor (which it can't) it would not be safe.

I noted there was a patchset for that on linux-serial from 2022 which
apparently got dropped and never merged, but I think it has a pretty serious
problem: it used a sysfs setting to control the behavior, which may be
reasonable for a default, but at the end of it this is really something that
is determined by the intent of the open() call, just like O_NONBLOCK replaced
the old callout devices we once had.

It seems to me that this may very well be a problem beyond ttys, in which case
a new open flag to request to a driver that the configuration and (observable)
state of the underlying hardware device -- whatever it may be -- should not be
disturbed by calling open(). This is of course already the case for many
devices, not to mention block and non-devices, in which case this flag is a
don't care.

The best name I came up with was O_NRESETDEV, but it's not something I'm
particularly attached to.

If the opinion is that this *doesn't* have a scope beyond ttys, then perhaps
abusing the O_DIRECT flag for this purpose would be an alternative.

Thoughts?

	-hpa