A cmpxchg allows us to avoid disabling and enabling interrupts. The cmpxchg is optimal to allow operations on per cpu freelist even if we may be moved to other processors while getting to the cmpxchg. So we do not need to be pinned to a cpu. This may be particularly useful for the RT kernel where we currently seem to have major SLAB issues with the per cpu structures. But the constant interrupt disable / enable of slab operations also increases the performance in general. The hard binding to per cpu structures only comes into play when we enter the slow path (__slab_alloc and __slab_free). At that point we have to disable interrupts like before. We have a problem of determining the page struct in slab_free due the issue that the freelist pointer is the only data value that we can reliably operate on. So we need to do a virt_to_page() on the freelist. This makes it impossible to use the fastpath for a full slab and increases overhead through a second virt_to_page for each slab_free(). We really need the virtual memmap patchset to get slab_free to good performance for this one. Pro: - Dirty single cacheline with a single instruction in slab_alloc to accomplish allocation. - Critical section is also a single instruction in slab_free. (but we need to write to the cacheline of the object too) Con: - Complex freelist management. __slab_alloc has to deal with results of race conditions. - Recalculation of per cpu structure address is necessary in __slab_alloc since process may be rescheduled while executing in slab_alloc. Signed-off-by: Christoph Lameter --- mm/slub.c | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) Index: linux-2.6.22-rc6-mm1/mm/slub.c =================================================================== --- linux-2.6.22-rc6-mm1.orig/mm/slub.c 2007-07-07 18:40:10.000000000 -0700 +++ linux-2.6.22-rc6-mm1/mm/slub.c 2007-07-07 18:46:04.000000000 -0700 @@ -1370,34 +1370,38 @@ static void unfreeze_slab(struct kmem_ca /* * Remove the cpu slab */ -static void deactivate_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) +static void deactivate_slab(struct kmem_cache *s, struct kmem_cache_cpu *c, + void **freelist) { struct page *page = c->page; + + c->page = NULL; /* * Merge cpu freelist into freelist. Typically we get here * because both freelists are empty. So this is unlikely * to occur. */ - while (unlikely(c->freelist)) { + while (unlikely(freelist)) { void **object; /* Retrieve object from cpu_freelist */ - object = c->freelist; - c->freelist = c->freelist[c->offset]; + object = freelist; + freelist = freelist[c->offset]; /* And put onto the regular freelist */ object[c->offset] = page->freelist; page->freelist = object; page->inuse--; } - c->page = NULL; unfreeze_slab(s, page); } static inline void flush_slab(struct kmem_cache *s, struct kmem_cache_cpu *c) { + void **freelist = xchg(&c->freelist, NULL); + slab_lock(c->page); - deactivate_slab(s, c); + deactivate_slab(s, c, freelist); } /* @@ -1467,10 +1471,13 @@ static void *__slab_alloc(struct kmem_ca { void **object; struct page *new; + void **freelist = NULL; if (!c->page) goto new_slab; + freelist = xchg(&c->freelist, NULL); + slab_lock(c->page); if (unlikely(!node_match(c, node))) goto another_slab; @@ -1490,7 +1497,7 @@ load_freelist: return object; another_slab: - deactivate_slab(s, c); + deactivate_slab(s, c, freelist); new_slab: new = get_partial(s, gfpflags, node); -- - To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/