linux-kernel - [patch 3/3] slub: sort parital list when thrashing

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <alpine.DEB.2.00.0903292243200.15813@chino.kir.corp.google.com>
Date:	Sun, 29 Mar 2009 22:43:41 -0700 (PDT)
From:	David Rientjes <rientjes@...gle.com>
To:	Pekka Enberg <penberg@...helsinki.fi>
cc:	Christoph Lameter <cl@...ux-foundation.org>,
	Nick Piggin <nickpiggin@...oo.com.au>,
	Martin Bligh <mbligh@...gle.com>, linux-kernel@...r.kernel.org
Subject: [patch 3/3] slub: sort parital list when thrashing

Caches that are cpu slab thrashing will scan their entire partial list
until a slab is found that will satisfy at least the requisite number of
allocations so that it will not be considered thrashing (as defined by
/sys/kernel/slab/cache/slab_thrash_ratio).

The partial list can be extremely long and its scanning requires that
list_lock is held for that particular node.  This can be inefficient if
slabs at the head of the list are not appropriate cpu slab replacements.

When an object is freed, the number of free objects for its slab is
calculated if the cpu cache is currently thrashing.  If it can satisfy
the requisite number of allocations so that the slab thrash ratio is
exceeded, it is moved to the head of the partial list.  This minimizes
the time spent holding list_lock and can help cacheline optimizations
for recently freed objects.

Cc: Christoph Lameter <cl@...ux-foundation.org>
Cc: Nick Piggin <nickpiggin@...oo.com.au>
Signed-off-by: David Rientjes <rientjes@...gle.com>
---
 mm/slub.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

diff --git a/mm/slub.c b/mm/slub.c
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1258,6 +1258,13 @@ static void remove_partial(struct kmem_cache *s, struct page *page)
 	spin_unlock(&n->list_lock);
 }

+static void move_partial_to_head(struct kmem_cache_node *n, struct page *page)
+{
+	spin_lock(&n->list_lock);
+	list_move(&page->lru, &n->partial);
+	spin_unlock(&n->list_lock);
+}
+
 /*
  * Remove from the partial list.
  *
@@ -1720,6 +1727,15 @@ checks_ok:
 	if (unlikely(!prior)) {
 		add_partial(get_node(s, page_to_nid(page)), page, 1);
 		stat(c, FREE_ADD_PARTIAL);
+	} else if (c->slowpath_allocs >= SLAB_THRASHING_THRESHOLD) {
+		/*
+		 * If the cache is actively slab thrashing, it's necessary to
+		 * move partial slabs to the head of the list so there isn't
+		 * excessive partial list scanning while holding list_lock.
+		 */
+		if (!skip_partial(s, page))
+			move_partial_to_head(get_node(s, page_to_nid(page)),
+					     page);
 	}

 out_unlock:
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/