| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAJuCfpFMhoCmqGJMU2uc4JHmk9zh88JzhZAeSz3DgvXEh+u+_g@mail.gmail.com>
Date: Wed, 23 Oct 2024 15:17:01 -0700
From: Suren Baghdasaryan <surenb@...gle.com>
To: Peter Zijlstra <peterz@...radead.org>
Cc: Andrii Nakryiko <andrii@...nel.org>, linux-trace-kernel@...r.kernel.org,
linux-mm@...ck.org, oleg@...hat.com, rostedt@...dmis.org, mhiramat@...nel.org,
bpf@...r.kernel.org, linux-kernel@...r.kernel.org, jolsa@...nel.org,
paulmck@...nel.org, willy@...radead.org, akpm@...ux-foundation.org,
mjguzik@...il.com, brauner@...nel.org, jannh@...gle.com, mhocko@...nel.org,
vbabka@...e.cz, shakeel.butt@...ux.dev, hannes@...xchg.org,
Liam.Howlett@...cle.com, lorenzo.stoakes@...cle.com
Subject: Re: [PATCH v3 tip/perf/core 1/4] mm: introduce mmap_lock_speculation_{start|end}
On Wed, Oct 23, 2024 at 1:10 PM Peter Zijlstra <peterz@...radead.org> wrote:
>
> On Thu, Oct 10, 2024 at 01:56:41PM -0700, Andrii Nakryiko wrote:
> > From: Suren Baghdasaryan <surenb@...gle.com>
> >
> > Add helper functions to speculatively perform operations without
> > read-locking mmap_lock, expecting that mmap_lock will not be
> > write-locked and mm is not modified from under us.
> >
> > Suggested-by: Peter Zijlstra <peterz@...radead.org>
> > Signed-off-by: Suren Baghdasaryan <surenb@...gle.com>
> > Signed-off-by: Andrii Nakryiko <andrii@...nel.org>
> > Link: https://lore.kernel.org/bpf/20240912210222.186542-1-surenb@google.com
> > ---
> > include/linux/mm_types.h | 3 ++
> > include/linux/mmap_lock.h | 72 ++++++++++++++++++++++++++++++++-------
> > kernel/fork.c | 3 --
> > 3 files changed, 63 insertions(+), 15 deletions(-)
> >
> > diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
> > index 6e3bdf8e38bc..5d8cdebd42bc 100644
> > --- a/include/linux/mm_types.h
> > +++ b/include/linux/mm_types.h
> > @@ -887,6 +887,9 @@ struct mm_struct {
> > * Roughly speaking, incrementing the sequence number is
> > * equivalent to releasing locks on VMAs; reading the sequence
> > * number can be part of taking a read lock on a VMA.
> > + * Incremented every time mmap_lock is write-locked/unlocked.
> > + * Initialized to 0, therefore odd values indicate mmap_lock
> > + * is write-locked and even values that it's released.
> > *
> > * Can be modified under write mmap_lock using RELEASE
> > * semantics.
> > diff --git a/include/linux/mmap_lock.h b/include/linux/mmap_lock.h
> > index de9dc20b01ba..9d23635bc701 100644
> > --- a/include/linux/mmap_lock.h
> > +++ b/include/linux/mmap_lock.h
> > @@ -71,39 +71,84 @@ static inline void mmap_assert_write_locked(const struct mm_struct *mm)
> > }
> >
> > #ifdef CONFIG_PER_VMA_LOCK
> > +static inline void init_mm_lock_seq(struct mm_struct *mm)
> > +{
> > + mm->mm_lock_seq = 0;
> > +}
> > +
> > /*
> > - * Drop all currently-held per-VMA locks.
> > - * This is called from the mmap_lock implementation directly before releasing
> > - * a write-locked mmap_lock (or downgrading it to read-locked).
> > - * This should normally NOT be called manually from other places.
> > - * If you want to call this manually anyway, keep in mind that this will release
> > - * *all* VMA write locks, including ones from further up the stack.
> > + * Increment mm->mm_lock_seq when mmap_lock is write-locked (ACQUIRE semantics)
> > + * or write-unlocked (RELEASE semantics).
> > */
> > -static inline void vma_end_write_all(struct mm_struct *mm)
> > +static inline void inc_mm_lock_seq(struct mm_struct *mm, bool acquire)
> > {
> > mmap_assert_write_locked(mm);
> > /*
> > * Nobody can concurrently modify mm->mm_lock_seq due to exclusive
> > * mmap_lock being held.
> > - * We need RELEASE semantics here to ensure that preceding stores into
> > - * the VMA take effect before we unlock it with this store.
> > - * Pairs with ACQUIRE semantics in vma_start_read().
> > */
> > - smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1);
> > +
> > + if (acquire) {
> > + WRITE_ONCE(mm->mm_lock_seq, mm->mm_lock_seq + 1);
> > + /*
> > + * For ACQUIRE semantics we should ensure no following stores are
> > + * reordered to appear before the mm->mm_lock_seq modification.
> > + */
> > + smp_wmb();
>
> Strictly speaking this isn't ACQUIRE, nor do we care about ACQUIRE here.
> This really is about subsequent stores, loads are irrelevant.
>
> > + } else {
> > + /*
> > + * We need RELEASE semantics here to ensure that preceding stores
> > + * into the VMA take effect before we unlock it with this store.
> > + * Pairs with ACQUIRE semantics in vma_start_read().
> > + */
>
> Again, not strictly true. We don't care about loads. Using RELEASE here
> is fine and probably cheaper on a few platforms, but we don't strictly
> need/care about RELEASE.
>
> > + smp_store_release(&mm->mm_lock_seq, mm->mm_lock_seq + 1);
> > + }
> > +}
>
> Also, it might be saner to stick closer to the seqcount naming of
> things and use two different functions for these two different things.
>
> /* straight up copy of do_raw_write_seqcount_begin() */
> static inline void mm_write_seqlock_begin(struct mm_struct *mm)
> {
> kcsan_nestable_atomic_begin();
> mm->mm_lock_seq++;
> smp_wmb();
> }
>
> /* straigjt up copy of do_raw_write_seqcount_end() */
> static inline void mm_write_seqcount_end(struct mm_struct *mm)
> {
> smp_wmb();
> mm->mm_lock_seq++;
> kcsan_nestable_atomic_end();
> }
>
> Or better yet, just use seqcount...
Yeah, with these changes it does look a lot like seqcount now...
I can take another stab at rewriting this using seqcount_t but one
issue that Jann was concerned about is the counter being int vs long.
seqcount_t uses unsigned, so I'm not sure how to address that if I
were to use seqcount_t. Any suggestions how to address that before I
move forward with a rewrite?
>
> > +
> > +static inline bool mmap_lock_speculation_start(struct mm_struct *mm, int *seq)
> > +{
> > + /* Pairs with RELEASE semantics in inc_mm_lock_seq(). */
> > + *seq = smp_load_acquire(&mm->mm_lock_seq);
> > + /* Allow speculation if mmap_lock is not write-locked */
> > + return (*seq & 1) == 0;
> > +}
> > +
> > +static inline bool mmap_lock_speculation_end(struct mm_struct *mm, int seq)
> > +{
> > + /* Pairs with ACQUIRE semantics in inc_mm_lock_seq(). */
> > + smp_rmb();
> > + return seq == READ_ONCE(mm->mm_lock_seq);
> > }
>
> Because there's nothing better than well known functions with a randomly
> different name and interface I suppose...
>
>
> Anyway, all the actual code proposed is not wrong. I'm just a bit
> annoyed its a random NIH of seqcount.
Ack. Let's decide what we do about u32 vs u64 issue and I'll rewrite this.
Powered by blists - more mailing lists