linux-ext4 - Re: ext4_alloc_context occupies 150 GiB of memory and makes the system unusable

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <201011221637.58275.bartoschek@or.uni-bonn.de>
Date:	Mon, 22 Nov 2010 16:37:58 +0100
From:	Christoph Bartoschek <bartoschek@...uni-bonn.de>
To:	linux-ext4@...r.kernel.org
Subject: Re: ext4_alloc_context occupies 150 GiB of memory and makes the system unusable

Am Montag, 22. November 2010 schrieben Sie:

> > We see that Slab uses most of the memory. And within slab nearly
> > everything is
> > 
> > used for ext4_alloc_context. There is the output of slabtop:
> >  Active / Total Objects (% used)    : 364597 / 1070670469 (0.0%)
> >  Active / Total Slabs (% used)      : 52397 / 39688960 (0.1%)
> >  Active / Total Caches (% used)     : 107 / 193 (55.4%)
> >  Active / Total Size (% used)       : 159579.25K / 150697605.41K (0.1%)
> >  Minimum / Average / Maximum Object : 0.02K / 0.14K / 4096.00K
> >  
> >   OBJS     ACTIVE  USE OBJ SIZE    SLABS OBJ/SLAB CACHE SIZE NAME
> > 
> > 1070187012      0   0%    0.14K 39636556       27 158546224K
> > ext4_alloc_context
> 
> and it's all unused... (inactive)
> 
> To make matters worse drop_caches doesn't touch the slabs, IIRC, but you
> might try: echo 3 > /proc/sys/vm/drop_caches

I tried it and it did not improve anything.


> > I see no reason why ext4 should use so much memory. What is it used for?
> > And how can I release it to get it used for my processes.
> 
> You may need to reboot, or at best unmount ext4 filesystems and/or rmmod
> the ext4 module, if the drop_caches trick doesn't work.
> 
> The fact that this doesn't get reclaimed seems to point to a problem
> with the vm though, I think (aside from the craziness of ext4 using
> this slab so heavily without my patch...)

I see the problem for the first time and I do not know whether it is 
reproducable. We have several similar machines with similar workloads but none 
has shown such a problem till now.

I'm going to reboot the machine. If it shows the problem again I will try a 
newer kernel and then the patch.

Some workload will be lost, but the machine did not do anything useful for 
three days now :)

Thanks,
Christoph
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html