linux-kernel - Re: Bisected regression in Linux 3.7.0-rc1, hang on login caused by replace

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 15 Oct 2012 19:11:05 -0400
From:	Pavel Roskin <proski@....org>
To:	Al Viro <viro@...IV.linux.org.uk>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Bisected regression in Linux 3.7.0-rc1, hang on login caused by
 replace_fd()

On Mon, 15 Oct 2012 22:51:27 +0100
Al Viro <viro@...IV.linux.org.uk> wrote:

> On Mon, Oct 15, 2012 at 05:14:37PM -0400, Pavel Roskin wrote:
> > Hello!
> > 
> > I tried the current mainline Linux on Fedora 16 x64_64 and found
> > that I cannot login in gdm.  I'm using gdm with LXDE.  After a few
> > minutes, kernel messages appear indicating a hung process
> > gnome-settings-daemon.
> 
> What is it hanging on?

Sorry, I have no idea.  I select my name, enter password and nothing
happens.  I can still move the mouse and select some widgets in gdm.  I
can use Ctrl-Alt-F2 to switch to the test console and login.

I attached gdb to the process and that's what I got:

(gdb) where
#0  0x0000003c19ae8803 in __GI___poll (fds=<optimized out>, nfds=<optimized out>, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:87
#1  0x0000003c1ba45448 in g_main_context_poll (n_fds=10, fds=0x2096dc0, priority=<optimized out>, timeout=41758, context=0x1e3e4a0) at gmain.c:3402
#2  g_main_context_iterate (context=0x1e3e4a0, block=<optimized out>, dispatch=1, self=<optimized out>) at gmain.c:3084
#3  0x0000003c1ba45c85 in g_main_loop_run (loop=0x1f35b70) at gmain.c:3297
#4  0x00000033cf75198d in gtk_main () at gtkmain.c:1362
#5  0x0000000000403d28 in main (argc=1, argv=0x7fffe57b3a78) at main.c:459
(gdb)

I'm not sure it's what you asked.

> Aha.  We have the first thing to test - is RLIMIT_NOFILE set to 0 for
> that process, by any chance?

No.  I made a copy of replace_fd() called by umh_pipe_setup() only, and
here's the output (debugging patch attached):

[   15.792895] replace_fd_debug: rlimit(RLIMIT_NOFILE) = 1024
[   15.792901] replace_fd_debug: file = ffff88011dd5db80
[   15.792903] replace_fd_debug: expand_files returned 0
[   15.792905] umh_pipe_setup: replace_fd_debug returned 0

>  We definitedly need to check
> replace_fd() return value, BTW.  OK, suppose it wasn't 0.

It's 0.

I made simplified functions replacing replace_fd() for umh_pipe_setup()
by substituting parameters and expanding do_dup2().  Here's the
combined function:

void replace_fd2(struct file *file)
{
	struct fdtable *fdt;
	struct files_struct *files = current->files;

	/* old code */
	sys_close(0);
	fd_install(0, file);

	spin_lock(&files->file_lock);
	fdt = files_fdtable(files);

	/* new code */
	get_file(file);
	rcu_assign_pointer(fdt->fd[0], file);

	__set_open_fd(0, fdt);
	__clear_close_on_exec(0, fdt);
	spin_unlock(&files->file_lock);
}

If the "new code" (two lines below the comment) is commented out and the
"old code" is not, everything works.  If the new code is used and the
old code is commented out, I have a hang.  Enabling all code leads
to a hang too.

> 	All right, so the next step in debugging is to print the damn
> return value of replace_fd().  That should narrow the things down.
> Another interesting question, of course, is what makes that gnome turd
> coredump on each boot?

I guess some GNOME programs are "uncomfortable" when launched by LXDE.
Considering that I'm running Fedora 16 with GNOME 3.2, I don't think
anyone would care about my bug reports.  But inability to login due to
kernel changes is bad and should not happen.

-- 
Regards,
Pavel Roskin

View attachment "replacefd-debug.patch" of type "text/x-patch" (1946 bytes)