linux-kernel - Re: RFC: Serial port DTR/RTS

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <274BA19A-1432-481F-9BA1-6983DFABD542@zytor.com>
Date: Tue, 18 Nov 2025 09:31:33 -0800
From: "H. Peter Anvin" <hpa@...or.com>
To: Ned Ulbricht <nedu@...scape.net>, "Maciej W. Rozycki" <macro@...am.me.uk>
CC: Greg KH <gregkh@...uxfoundation.org>, "Theodore Ts'o" <tytso@....edu>,
        Maarten Brock <Maarten.Brock@...ls.nl>,
        "linux-serial@...r.kernel.org" <linux-serial@...r.kernel.org>,
        "linux-api@...r.kernel.org" <linux-api@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: RFC: Serial port DTR/RTS - O_<something>

On November 18, 2025 8:33:07 AM PST, Ned Ulbricht <nedu@...scape.net> wrote:
>On 11/15/25 16:47, H. Peter Anvin wrote:
>> On 2025-11-15 13:29, Ned Ulbricht wrote:
>>> |
>>> | O_TTY_INIT
>>> 
>>> https://pubs.opengroup.org/onlinepubs/9799919799/
>>> 
>>> That's what motivates my first-glance preference to name this new flag,
>>> which will have approximately opposite behavior, as O_TTY_NOINIT.
>>> 
>>> But as a generic abstraction, I more prefer O_KEEP.
>>> 
>> 
>> O_KEEP seems a little vague, but O_KEEPCONFIG seems like a decent name.
>> 
>> It seems like we don't have several new flags:
>> 
>> 	O_EXEC
>> 	O_SEARCH
>> 	O_CLOFORK
>> 	O_TTY_INIT
>> 	O_RSYNC
>> 	O_NOCLOBBER
>> 
>> Some of them *may* be possible to construct with existing Linux options, I'm
>> not 100% sure; in particular O_SEARCH might be the same as (O_DIRECTORY|O_PATH).
>> 
>> O_NOCLOBBER looks like an odd in-between between O_EXCL and
>> (O_EXCL|O_NOFOLLOW); stated to be specifically to implement the shell
>> "noclobber" semantic.
>
>"(O_EXCL|O_NOFOLLOW)" provokes a thought...
>
>As essential context, fs/open.c build_open_flags() has:
>
>if (flags & O_CREAT) {
>	op->intent |= LOOKUP_CREATE;
>	if (flags & O_EXCL) {
>		op->intent |= LOOKUP_EXCL;
>		flags |= O_NOFOLLOW;
>	}
>}
>
>if (!(flags & O_NOFOLLOW))
>	lookup_flags |= LOOKUP_FOLLOW;
>
>So with that context, just imagine hypothetically implementing both a
>non-zero O_TTY_INIT flag and an O_KEEPCONFIG flag. What would
>build_open_flags() look like to handle the case where userspace
>simultaneously asserts both flags?  Even if it's documented, specified
>as unspecified behavior, what would the code actually do?
>
>Or, alternatively, should an hypothetical standardization insist that in
>any implementation, one of O_TTY_INIT, O_KEEPCONFIG must be #define'd 0?
>
>
>Ned

It's not actually a contradiction: it means preserve all configuration *except* the minimal termios tweaks required to bring it inside the POSIX compliant envelope, notably setting winsize to "appropriate default parameters."

Linux doesn't have a lot of such settings, but I can see at least one *very useful* one: bringing B19200 and B38400 (EXTA and EXTB), which can be tweaked by setserial, back to their proper baud rates.

There are even two ways to do that: either keep the B19200/B38400 setting and change the baud rate, or keep the baud rate and change termios to match (using BOTHER if necessary; after my changes to glibc 2.42+ BOTHER is a private interface between glibc and the kernel and thus doesn't break POSIX compliance.)

A fairly reasonable implementation would be the first with O_TTY_INIT and the second with O_TTY_INIT | O_KEEPCONFIG.

Flags in termios that probably should be cleared by O_TTY_INIT are CMSPAR, OLCUC, IUCLC, IMAXBEL, ECHOPRT, ECHOKE, FLUSHO, PENDIN, IEXTEN and EXTPROC; I'm not sure about ADDRB, CRTSCTS, IUTF8 or the line discipline; at least with O_KEEPCONFIG at least CRTSCTS ought to be kept I would think, as changing it would have immediate effect visible to 

Obviously an application that wants to absolutely minimize changes must not pass O_TTY_INIT.

The other (and simpler!) option is to simply declare O_KEEPCONFIG | O_TTY_INIT as a reserved bit combination and return -EINVAL until we find a good reason to do anything different.