linux-ext4 - [Bug 108631] Stuck on mb_cache

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 07 Jan 2016 15:49:40 +0000
From:	bugzilla-daemon@...zilla.kernel.org
To:	linux-ext4@...r.kernel.org
Subject: [Bug 108631] Stuck on mb_cache_spinlock

https://bugzilla.kernel.org/show_bug.cgi?id=108631

--- Comment #6 from Theodore Tso <tytso@....edu> ---
You said that you had replicated this problem on 4.2, so it's probably not the
primary driver of the problem in your case, but be aware that in some of the
older kernels (and 3.14 is included in that list), the shrinker is not cgroup
aware.   That means that if a single container starts aggressively using all of
its memory, we will start kicking off *all* of the slab cache shrinkers.   I've
seen that in a very container-heavy environment, with many tasks that use right
up to their memory cgroup limit (and sometimes beyond), that this can cause the
ext4 extent status cache shrinker to end up getting run a lot, which leads to
symptoms similar to yours due to spinlock contention.  (Of course, for a
different spinlock since a different slab cache shrinker is involved).

I've been playing with some patches to use trylock and to abandon the ext4 es
shrinker if it's under too much lock contention, but I haven't come up with
something for which I was sufficiently happy to push upstream yet --- because
if the slab cache does need shrinker, just simply giving up isn't going to do
the right thing entirely, either.

One hacky solution would be to add knobs so we can cap the size of various
caches like the mbcache and extents status cache so we can afford to be more
aggressive at giving up if the trylock fails.  But assuming that system
administrators will *set* these knobs correctly is probably a bad choice.  So
this is a more general problem than just the mbcache, and I haven't found a
good general solution yet.  And there may not be; we can try to make systems
that are less likely to lead to spinlock contention, but under sufficiently
heavy memory pressure, this may only be forestalling the inevitable.

-- 
You are receiving this mail because:
You are watching the assignee of the bug.
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html