[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <d2e02a4e-2a73-7b07-1bfb-ddf3f2a53ff1@suse.cz>
Date: Wed, 24 May 2023 12:13:09 +0200
From: Vlastimil Babka <vbabka@...e.cz>
To: Peter Zijlstra <peterz@...radead.org>,
torvalds@...ux-foundation.org
Cc: corbet@....net, will@...nel.org, boqun.feng@...il.com,
mark.rutland@....com, catalin.marinas@....com, dennis@...nel.org,
tj@...nel.org, cl@...ux.com, hca@...ux.ibm.com, gor@...ux.ibm.com,
agordeev@...ux.ibm.com, borntraeger@...ux.ibm.com,
svens@...ux.ibm.com, tglx@...utronix.de, mingo@...hat.com,
bp@...en8.de, dave.hansen@...ux.intel.com, x86@...nel.org,
hpa@...or.com, joro@...tes.org, suravee.suthikulpanit@....com,
robin.murphy@....com, dwmw2@...radead.org,
baolu.lu@...ux.intel.com, Arnd Bergmann <arnd@...db.de>,
Herbert Xu <herbert@...dor.apana.org.au>, davem@...emloft.net,
penberg@...nel.org, rientjes@...gle.com, iamjoonsoo.kim@....com,
Andrew Morton <akpm@...ux-foundation.org>,
roman.gushchin@...ux.dev, 42.hyeyoo@...il.com,
linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
linux-mm@...ck.org, linux-s390@...r.kernel.org,
iommu@...ts.linux.dev, linux-arch@...r.kernel.org,
linux-crypto@...r.kernel.org, sfr@...b.auug.org.au,
mpe@...erman.id.au
Subject: Re: [PATCH v3 08/11] slub: Replace cmpxchg_double()
On 5/24/23 11:32, Peter Zijlstra wrote:
> On Mon, May 15, 2023 at 09:57:07AM +0200, Peter Zijlstra wrote:
>
>> @@ -3008,6 +3029,22 @@ static inline bool pfmemalloc_match(stru
>> }
>>
>> #ifndef CONFIG_SLUB_TINY
>> +static inline bool
>> +__update_cpu_freelist_fast(struct kmem_cache *s,
>> + void *freelist_old, void *freelist_new,
>> + unsigned long tid)
>> +{
>> +#ifdef system_has_freelist_aba
>> + freelist_aba_t old = { .freelist = freelist_old, .counter = tid };
>> + freelist_aba_t new = { .freelist = freelist_new, .counter = next_tid(tid) };
>> +
>> + return this_cpu_cmpxchg_freelist(s->cpu_slab->freelist_tid.full,
>> + old.full, new.full) == old.full;
>> +#else
>> + return false;
>> +#endif
>> +}
>> +
>> /*
>> * Check the slab->freelist and either transfer the freelist to the
>> * per cpu freelist or deactivate the slab.
>> @@ -3359,11 +3396,7 @@ static __always_inline void *__slab_allo
>> * against code executing on this cpu *not* from access by
>> * other cpus.
>> */
>> - if (unlikely(!this_cpu_cmpxchg_double(
>> - s->cpu_slab->freelist, s->cpu_slab->tid,
>> - object, tid,
>> - next_object, next_tid(tid)))) {
>> -
>> + if (unlikely(!__update_cpu_freelist_fast(s, object, next_object, tid))) {
>> note_cmpxchg_failure("slab_alloc", s, tid);
>> goto redo;
>> }
>> @@ -3736,11 +3769,7 @@ static __always_inline void do_slab_free
>>
>> set_freepointer(s, tail_obj, freelist);
>>
>> - if (unlikely(!this_cpu_cmpxchg_double(
>> - s->cpu_slab->freelist, s->cpu_slab->tid,
>> - freelist, tid,
>> - head, next_tid(tid)))) {
>> -
>> + if (unlikely(!__update_cpu_freelist_fast(s, freelist, head, tid))) {
>> note_cmpxchg_failure("slab_free", s, tid);
>> goto redo;
>> }
>
> This isn't right; the this_cpu_cmpxchg_double() was unconditional and
> relied on the local_irq_save() fallback when no native cmpxchg128 is
> present.
>
> The below delta makes things boot again when system_has_cmpxchg128 is
> not defined.
Right, that should do.
> I'm going to zap these patches from tip/locking/core for a few days and
> fold the below back into the series and let it run through the robots
> again.
I noticed some comments in mm/slub.c still mention "cmpxchg_double", dunno
how much you want to clean it right now or can be postponed. Also some sysfs
stats files for CONFIG_SLUB_STATS (not widely used) which we probably might
try renaming without breaking anyone, but it's not guaranteed.
Powered by blists - more mailing lists