linux-kernel - tty TTY_HUPPED anomaly

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Fri, 23 Dec 2011 09:51:37 -0500
From:	Mark Hounschell <markh@...pro.net>
To:	Linux-kernel <linux-kernel@...r.kernel.org>
CC:	Mark Hounschell <dmarkh@....rr.com>
Subject: tty  TTY_HUPPED anomaly

I have an application that has seemingly been stable for many years and 
even using the 3.1.1 kernel all is fine. I'm using the openSuSE 
distribution. All is good with the openSuSE-11.4 dist and the 3.1.1 
kernel. We recently started testing this application on openSuSE-12.1 
using the same  vanilla 3.1.1 kernel and now we have problems with just 
about all out tty/serial related devices that we use. I thought the 
problem maybe an openSuSE glibc problem so I built and ran the 
openSuSE-12.1 version of glibc on the openSuSE-11.4 dist but the 
application worked just fine. So I started looking at the kernel and one 
of our serial devices that fails. I choose the Synclink GT because its 
driver is "in kernel".

In our application with this particular card, the following TIOCSETD 
ioctl is this code snippet  fails most of the time.

     int32_t hdlc_disc = N_HDLC;

     Q->fd = open(Q->FileName, (O_RDWR | O_NONBLOCK), 0);

     if (Q->fd < 0)
            return (FALSE);

     Q->File = fdopen(Q->fd, "rw");

     fcntl(Q->fd, F_SETFL, fcntl(S->fd,F_GETFL) & ~O_NONBLOCK);

     if (ioctl(Q->fd, TIOCSETD, &hdlc_disc) < 0) {
         perror("SCM_open_port: TIOCSETD failed: ");
         return (FALSE);
     }

The kernel code that results in the above code snippet failure is the 
following section of the tty_set_ldisc function in tty_io.c.

         if (test_bit(TTY_HUPPED, &tty->flags)) {
                 /* We were raced by the hangup method. It will have stomped
                    the ldisc data and closed the ldisc down */
                 clear_bit(TTY_LDISC_CHANGING, &tty->flags);
                 mutex_unlock(&tty->ldisc_mutex);
                 tty_ldisc_put(new_ldisc);
                 tty_unlock();
                 return -EIO;
         }

I do not really understand what TTY_HUPPED is or how it could be getting 
set. I look at it when the tty_set_ldisc function is entered and it is 
not set. Some where between entrance of the tty_set_ldisc and getting to 
the above code TTY_HUPPED gets set causing the -EIO return.

I see nothing in my code that should cause this to happen. At start up 
time a thread for each port used does this then goes to sleep. Again, I 
can't understand why now with openSuSE-12.1 this starts happening. Can 
someone help me understand why this might happen. I don't know if it is 
me, the dist, or the kernel. Frequently it all works at startup and will 
eventually fail during operation.

Other serial cards that I also have similar problems with (different 
ioctls) are various Digi serial cards. I haven't look into the kernel to 
see what is happening with those as yet but am sure the problem is 
related. Sometimes they work, most of the time they do not.

Many thanks in advance
Mark
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/