| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAB=+i9T+aLEj6BHdXXJxqY914O1ZVCbmL8iL_5wiFB3dQN6yUg@mail.gmail.com>
Date: Sun, 3 Dec 2023 20:19:38 +0900
From: Hyeonggon Yoo <42.hyeyoo@...il.com>
To: Chengming Zhou <chengming.zhou@...ux.dev>
Cc: vbabka@...e.cz, cl@...ux.com, penberg@...nel.org,
rientjes@...gle.com, iamjoonsoo.kim@....com,
akpm@...ux-foundation.org, roman.gushchin@...ux.dev,
linux-mm@...ck.org, linux-kernel@...r.kernel.org,
Chengming Zhou <zhouchengming@...edance.com>
Subject: Re: [PATCH v5 7/9] slub: Optimize deactivate_slab()
On Sun, Dec 3, 2023 at 7:26 PM Chengming Zhou <chengming.zhou@...ux.dev> wrote:
>
> On 2023/12/3 17:23, Hyeonggon Yoo wrote:
> > On Thu, Nov 2, 2023 at 12:25 PM <chengming.zhou@...ux.dev> wrote:
> >>
> >> From: Chengming Zhou <zhouchengming@...edance.com>
> >>
> >> Since the introduce of unfrozen slabs on cpu partial list, we don't
> >> need to synchronize the slab frozen state under the node list_lock.
> >>
> >> The caller of deactivate_slab() and the caller of __slab_free() won't
> >> manipulate the slab list concurrently.
> >>
> >> So we can get node list_lock in the last stage if we really need to
> >> manipulate the slab list in this path.
> >>
> >> Signed-off-by: Chengming Zhou <zhouchengming@...edance.com>
> >> Reviewed-by: Vlastimil Babka <vbabka@...e.cz>
> >> Tested-by: Hyeonggon Yoo <42.hyeyoo@...il.com>
> >> ---
> >> mm/slub.c | 79 ++++++++++++++++++-------------------------------------
> >> 1 file changed, 26 insertions(+), 53 deletions(-)
> >>
> >> diff --git a/mm/slub.c b/mm/slub.c
> >> index bcb5b2c4e213..d137468fe4b9 100644
> >> --- a/mm/slub.c
> >> +++ b/mm/slub.c
> >> @@ -2468,10 +2468,8 @@ static void init_kmem_cache_cpus(struct kmem_cache *s)
> >> static void deactivate_slab(struct kmem_cache *s, struct slab *slab,
> >> void *freelist)
> >> {
> >> - enum slab_modes { M_NONE, M_PARTIAL, M_FREE, M_FULL_NOLIST };
> >> struct kmem_cache_node *n = get_node(s, slab_nid(slab));
> >> int free_delta = 0;
> >> - enum slab_modes mode = M_NONE;
> >> void *nextfree, *freelist_iter, *freelist_tail;
> >> int tail = DEACTIVATE_TO_HEAD;
> >> unsigned long flags = 0;
> >> @@ -2509,65 +2507,40 @@ static void deactivate_slab(struct kmem_cache *s, struct slab *slab,
> >> /*
> >> * Stage two: Unfreeze the slab while splicing the per-cpu
> >> * freelist to the head of slab's freelist.
> >> - *
> >> - * Ensure that the slab is unfrozen while the list presence
> >> - * reflects the actual number of objects during unfreeze.
> >> - *
> >> - * We first perform cmpxchg holding lock and insert to list
> >> - * when it succeed. If there is mismatch then the slab is not
> >> - * unfrozen and number of objects in the slab may have changed.
> >> - * Then release lock and retry cmpxchg again.
> >> */
> >> -redo:
> >> -
> >> - old.freelist = READ_ONCE(slab->freelist);
> >> - old.counters = READ_ONCE(slab->counters);
> >> - VM_BUG_ON(!old.frozen);
> >> -
> >> - /* Determine target state of the slab */
> >> - new.counters = old.counters;
> >> - if (freelist_tail) {
> >> - new.inuse -= free_delta;
> >> - set_freepointer(s, freelist_tail, old.freelist);
> >> - new.freelist = freelist;
> >> - } else
> >> - new.freelist = old.freelist;
> >> -
> >> - new.frozen = 0;
> >> + do {
> >> + old.freelist = READ_ONCE(slab->freelist);
> >> + old.counters = READ_ONCE(slab->counters);
> >> + VM_BUG_ON(!old.frozen);
> >> +
> >> + /* Determine target state of the slab */
> >> + new.counters = old.counters;
> >> + new.frozen = 0;
> >> + if (freelist_tail) {
> >> + new.inuse -= free_delta;
> >> + set_freepointer(s, freelist_tail, old.freelist);
> >> + new.freelist = freelist;
> >> + } else {
> >> + new.freelist = old.freelist;
> >> + }
> >> + } while (!slab_update_freelist(s, slab,
> >> + old.freelist, old.counters,
> >> + new.freelist, new.counters,
> >> + "unfreezing slab"));
> >>
> >> + /*
> >> + * Stage three: Manipulate the slab list based on the updated state.
> >> + */
> >
> > deactivate_slab() might unconsciously put empty slabs into partial list, like:
> >
> > deactivate_slab() __slab_free()
> > cmpxchg(), slab's not empty
> > cmpxchg(), slab's empty
> > and unfrozen
>
> Hi,
>
> Sorry, but I don't get it here how __slab_free() can see the slab empty,
> since the slab is not empty from deactivate_slab() path, and it can't be
> used by any CPU at that time?
The scenario is CPU B previously allocated an object from slab X, but
put it into node partial list and then CPU A have taken slab X into cpu slab.
While slab X is CPU A's cpu slab, when CPU B frees an object from slab X,
it puts the object into slab X's freelist using cmpxchg.
Let's say in CPU A the deactivation path performs cmpxchg and X.inuse was 1,
and then CPU B frees (__slab_free()) to slab X's freelist using cmpxchg,
_before_ slab X's put into partial list by CPU A.
Then CPU A thinks it's not empty so put it into partial list, but by CPU B
the slab has become empty.
Maybe I am confused, in that case please tell me I'm wrong :)
Thanks!
--
Hyeonggon
Powered by blists - more mailing lists