Date:	Mon, 25 Apr 2011 18:04:50 +0200
From:	Bruno Prémont <bonbons@...ux-vserver.org>
To:	Linus Torvalds <torvalds@...ux-foundation.org>
Cc:	Mike Frysinger <vapier.adi@...il.com>,
	KOSAKI Motohiro <kosaki.motohiro@...fujitsu.com>,
	linux-kernel@...r.kernel.org, linux-mm@...ck.org,
	linux-fsdevel@...r.kernel.org,
	"Paul E. McKenney" <paul.mckenney@...aro.org>,
	Pekka Enberg <penberg@...nel.org>
Subject: Re: 2.6.39-rc4+: Kernel leaking memory during FS scanning,
 regression?

On Mon, 25 April 2011 Linus Torvalds wrote:
> On Mon, Apr 25, 2011 at 2:17 AM, Bruno Prémont wrote:
> >
> > Here it seems to happened when I run 2 intensive tasks in parallel, e.g.
> > (re)emerging gimp and running revdep-rebuild -pi in another terminal.
> > This produces a fork rate of about 100-300 per second.
> >
> > Suddenly kmalloc-128 slabs stop being freed and things degrade.
> 
> So everything seems to imply some kind of filesystem/vfs thing, but
> let's try to gather a bit more information about exactly what it is.
> 
> Some of it also points to RCU freeing, but that "kmalloc-128" doesn't
> really match my expectations. According to your slabinfo, it's not the
> dentries.
>
> One thing I'd ask you to do is to boot with the "slub_nomerge" kernel
> command  line switch. The SLUB "merge slab caches" thing may save some
> memory, but it has been a disaster from every other standpoint - every
> time there's a memory leak, it ends up making it very confusing to try
> to figure things out.
> 
> For example, your traces seem to imply that the kmalloc-128 allocation
> is actually the "filp" cache, but it has gotten merged with the
> kmalloc-128 cache, so slabinfo doesn't actually show the right user.

Redone with slub_nomerge cmdline switch.
Attached (for easy diffing):

slabinfo-2, meminfo-2: when memory use starts manifesting itself
                       (work triggering it being SIGSTOPped)

slabinfo-4, meminfo-4: info gathered again after sync && echo 2 > /proc/sys/vm/drop_caches

kmemleak reports 86681 new leaks between shortly after boot and -2 state.
(and 2348 additional ones between -2 and -4).

> (Pekka? This is a real _problem_. The whole "confused debugging" is
> wasting a lot of peoples time. Can we please try to get slabinfo
> statistics work right for the merged state. Or perhaps decide to just
> not merge at all?)
> 
> As to why it has started to happen now: with the whole RCU lookup
> thing, many more filesystem objects are RCU-free'd (dentries have been
> for a long time, but now we have inodes and filp's too), and that may
> end up delaying allocations sufficiently that you end up seeing
> something that used to be borderline become a major problem.
> 
> Also, what's your kernel config, in particular wrt RCU? The RCU
> freeing _should_ be self-limiting (if I recall correctly) and not let
> infinite amounts of RCU work (ie pending freeing) accumulate, but
> maybe something is broken. Do you have a UP kernel with TINY_RCU, for
> example?

Config was in first message of thread (but unfortunately not properly
labeled), attaching again (to include change for debugging features)

Yes, it's uni-processor system, so SMP=n.
TINY_RCU=y, PREEMPT_VOLUNTARY=y (whole /proc/config.gz attached keeping
compression)

Bruno


> Or maybe I'm just confused, and there's never any RCU throttling at
> all. Paul?
> 
>                                Linus

Download attachment "config.gz" of type "application/x-gzip" (15694 bytes)

View attachment "meminfo-2" of type "text/plain" (1008 bytes)

View attachment "meminfo-4" of type "text/plain" (1008 bytes)

View attachment "slabinfo-2" of type "text/plain" (16070 bytes)

View attachment "slabinfo-4" of type "text/plain" (16071 bytes)

Powered by blists - more mailing lists