linux-kernel - Advice sought: pty/tty gets into "gridlock" if it does too much input/output while stopped

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Wed, 24 Sep 2008 12:49:00 -0600
From:	Joe Peterson <joe@...rush.com>
To:	Linux Kernel <linux-kernel@...r.kernel.org>
CC:	Alan Cox <alan@...rguk.ukuu.org.uk>
Subject: Advice sought: pty/tty gets into "gridlock" if it does too much input/output
 while stopped

I've been investigating an issue I ran into recently, and I am not yet sure how
to fix it, so I am soliciting suggestions.

Basically, it is quite easy to "gridlock" (I avoided the term "deadlock", since
this is not a locking issue) a shell/xterm session by stopping the tty (e.g.
hitting ^S) and then causing lots of input and output.  Here is an easy way to
do it:

1) open an xterm
2) type "grep foo"
3) hit ^S
4) paste a large number of lines (more than the 4k size)
   containing the word "foo" (this causes output for every input)

There is now no way to break out of this condition in a normal way.  Signals
(e.g. ^C, ^Z) and ^Q cannot break out of this state, since the input character
cannot make it into the line discipline (I never see it in the buffer in
pty_write() after the condition takes hold).  The only thing that can be done is
to close the xterm window.

The problem is that pty_write() (when trying to process typed characters) finds
that "to->receive_room" is zero, so no data is passed to n_tty_receive_buf(),
and nothing can move.  Since both the reads and writes (both of which pass
through the pty_write() function) are waiting to have some space with which to
work, and the tty is stopped, things are stuck.  Since signal and ^Q chars do
not need buffer space, perhaps these could (perhaps) get a higher priority and
be sent through, but since I do not see them in the pty_write() buffer when
issued *during* the condition, it is not clear where this should be done, or if
it is the right solution.  I've verified this as far back as 2.6.24.

							-Joe

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/