linux-kernel - Re: [regression] ENOTTY returned for tty fds

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <be1a3839-23a6-4726-9018-3d18a27163be@yandex.ru>
Date: Wed, 24 Jul 2024 12:55:22 +0300
From: stsp <stsp2@...dex.ru>
To: Greg KH <gregkh@...uxfoundation.org>
Cc: linux-serial@...r.kernel.org, Linux kernel
 <linux-kernel@...r.kernel.org>, Herbert Xu <herbert@...dor.apana.org.au>,
 Johan Hovold <johan@...nel.org>
Subject: Re: [regression] ENOTTY returned for tty fds

24.07.2024 12:08, Greg KH пишет:
> On Wed, Jul 24, 2024 at 11:07:32AM +0300, stsp wrote:
>> 24.07.2024 09:51, Greg KH пишет:
>>> What caused this change/regression?
>> I have absolutely no idea.
>> I've found it by debugging userspace,
>> and wrote a test-case to make sure the
>> problem is not in user-space.
> So this has always worked this way?  Or has it changed?  If changed,
> when did it work before?

OK, I did some extensive digging, and
now am pretty sure its this one:

commit 1b8b20868a6d64cfe8174a21b25b74367bdf0560
Author:     Johan Hovold <johan@...nel.org>
AuthorDate: Wed Apr 7 11:52:02 2021 +0200
Commit:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>
CommitDate: Sat Apr 10 10:36:34 2021 +0200

     tty: fix return value for unsupported ioctls

     Drivers should return -ENOTTY ("Inappropriate I/O control operation")
     when an ioctl isn't supported, while -EINVAL is used for invalid
     arguments.

>>>     And does any real-world programs
>>> rely on this?
>> dosemu
> It does this today or wants to do this in the future?

It does so since 2003/03/15, according
to a change-log (so for 21 year now).
Adding Herbert Xu to CC as an author of
that feature.

>>>     What exactly are you trying to determine with this ioctl
>>> test?
>> Whether it is a PTS (Pseudo-Tty-Slave), or
>> a real comport with MSR signalling.
> Why is that needed?

To fake modem status lines, mustly DCD.


>    And why not do it how other programs (like stty)
> does it?

I am not sure stty is interested in faking
modem status lines. It doesn't seem to be
using TIOCM ioctls at all.

>>>     Is there a different way to determine that?
>> I am not aware of any "canonical" way
>> of determining this. Maybe you tell me. :)
>> So far the only fix I know, is to stop checking
>> errno. But you return ENOTTY for a tty-associated
>> fd (isatty(fd)==1), so I believe this is a
>> bug in a kernel.
> isatty() is a libc provided function, not a kernel call.
It seems to be using TCGETS ioctl() and looks
for it to not return an error.
I still think returning ENOTTY where TCGETS
succeeds, is more than strange.
I looked at the tty code of current linux kernel,
and it returns EOPNOTSUPP in most such cases.
At least that makes sense, but ENOTTY?