linux-kernel - Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning, regression?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Wed, 27 Apr 2011 00:28:37 +0200 (CEST)
From:	Thomas Gleixner <tglx@...utronix.de>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
cc:	Bruno Prémont <bonbons@...ux-vserver.org>,
	Ingo Molnar <mingo@...e.hu>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	paulmck@...ux.vnet.ibm.com, Mike Frysinger <vapier.adi@...il.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org,
	linux-fsdevel@...r.kernel.org,
	"Paul E. McKenney" <paul.mckenney@...aro.org>,
	Pekka Enberg <penberg@...nel.org>
Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning,
 regression?

On Tue, 26 Apr 2011, Linus Torvalds wrote:

> On Tue, Apr 26, 2011 at 10:09 AM, Bruno Prémont
> <bonbons@...ux-vserver.org> wrote:
> >
> > Just in case, /proc/$(pidof rcu_kthread)/status shows ~20k voluntary
> > context switches and exactly one non-voluntary one.
> >
> > In addition when rcu_kthread has stopped doing its work
> > `swapoff $(swapdevice)` seems to block forever (at least normal shutdown
> > blocks on disabling swap device).
> > If I get to do it when I get back home I will manually try to swapoff
> > and take process traces with sysrq-t.
> 
> That "exactly one non-voluntary one" sounds like the smoking gun.
> 
> Normally SCHED_FIFO runs until it voluntarily gives up the CPU. That's
> kind of the point of SCHED_FIFO. Involuntary context switches happen
> when some higher-priority SCHED_FIFO process becomes runnable (irq
> handlers? You _do_ have CONFIG_IRQ_FORCED_THREADING=y in your config
> too), and maybe there is a bug in the runqueue handling for that case.

The forced irq threading is only effective when you add the command
line parameter "threadirqs". I don't see any irq threads in the ps
outputs, so that's not the problem.

Though the whole ps output is weird. There is only one thread/process
which accumulated CPU time

collectd  1605  0.6  0.7  49924  3748 ?        SNLsl 22:14   0:14

All others show 0:00 CPU time - not only kthread_rcu.

Bruno, are you running on real hardware or in a virtual machine?

Can you please enable CONFIG_SCHED_DEBUG and provide the output of
/proc/sched_stat when the problem surfaces and a minute after the
first snapshot?

Also please apply the patch below and check, whether the printk shows
up in your dmesg.

Thanks,

	tglx

---
 kernel/sched_rt.c |    1 +
 1 file changed, 1 insertion(+)

Index: linux-2.6-tip/kernel/sched_rt.c
===================================================================
--- linux-2.6-tip.orig/kernel/sched_rt.c
+++ linux-2.6-tip/kernel/sched_rt.c
@@ -609,6 +609,7 @@ static int sched_rt_runtime_exceeded(str
 
 	if (rt_rq->rt_time > runtime) {
 		rt_rq->rt_throttled = 1;
+		printk_once(KERN_WARNING "sched: RT throttling activated\n");
 		if (rt_rq_throttled(rt_rq)) {
 			sched_rt_rq_dequeue(rt_rq);
 			return 1;