linux-kernel - Stack size protection broken on ppc64

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Sat, 06 Feb 2010 11:43:13 +1100
From:	Michael Neuling <mikey@...ling.org>
To:	anton@...ba.org, linux-kernel@...r.kernel.org,
	linuxppc-dev@...abs.org, Serge Hallyn <serue@...ibm.com>,
	WANG Cong <xiyou.wangcong@...il.com>,
	Paul Mackerras <paulus@...ba.org>, benh@...nel.crashing.org
cc:	miltonm@....com
Subject: Stack size protection broken on ppc64

On recent ppc64 kernels, limiting the stack (using 'ulimit -s blah') is
now more restrictive than it was before.  On 2.6.31 with 4k pages I
could run 'ulimit -s 16; /usr/bin/test' without a problem.  Now with
mainline, even 'ulimit -s 64; /usr/bin/test' gets killed.

Using 64k pages is even worse.  I can't even run '/bin/ls' with a 1MB
stack (ulimit -s 1024; /bin/ls).  Hence, it seems new kernels are too
restrictive, rather than the old kernels being too liberal.

I've not tested with any other architectures.  

Bisecting, I found that this is the culprit (which is in 2.6.32)

  commit fc63cf237078c86214abcb2ee9926d8ad289da9b
  Author: Anton Blanchard <anton@...ba.org>
  exec:   setup_arg_pages() fails to return errors

Looking at the patch, it's probably just unmasking a preexisting issue.
The error path for expand_stack() (and others) was modified to:
---
	ret = expand_stack(vma, stack_base);
	if (ret)
		ret = -EFAULT;

out_unlock:
	up_write(&mm->mmap_sem);
-       return 0;
+       return ret;
 }
 EXPORT_SYMBOL(setup_arg_pages);
---

So previously expand_stack errors were not returned correctly by
setup_arg_pages, but now they are.

Any clues how to fix this?

Mikey
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/