Message-ID: <CAC2o3DJjQvSvoYSOXv15hiu1YQS6TpHdg-j0gY4o90w2ti9bkw@mail.gmail.com>
Date:   Tue, 18 May 2021 16:26:40 +0800
From:   Fox Chen <foxhlchen@...il.com>
To:     Ian Kent <raven@...maw.net>
Cc:     Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Tejun Heo <tj@...nel.org>, Al Viro <viro@...iv.linux.org.uk>,
        Eric Sandeen <sandeen@...deen.net>,
        Brice Goglin <brice.goglin@...il.com>,
        Rick Lindsley <ricklind@...ux.vnet.ibm.com>,
        David Howells <dhowells@...hat.com>,
        Miklos Szeredi <miklos@...redi.hu>,
        Marcelo Tosatti <mtosatti@...hat.com>,
        linux-fsdevel <linux-fsdevel@...r.kernel.org>,
        Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v4 0/5] kernfs: proposed locking and concurrency improvement

On Mon, May 17, 2021 at 9:32 AM Ian Kent <raven@...maw.net> wrote:
>
> On Fri, 2021-05-14 at 10:34 +0800, Fox Chen wrote:
> > On Fri, May 14, 2021 at 9:34 AM Ian Kent <raven@...maw.net> wrote:
> > >
> > > On Thu, 2021-05-13 at 23:37 +0800, Fox Chen wrote:
> > > > Hi Ian
> > > >
> > > > On Thu, May 13, 2021 at 10:10 PM Ian Kent <raven@...maw.net>
> > > > wrote:
> > > > >
> > > > > On Wed, 2021-05-12 at 16:54 +0800, Fox Chen wrote:
> > > > > > On Wed, May 12, 2021 at 4:47 PM Fox Chen
> > > > > > <foxhlchen@...il.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I ran it on my benchmark (
> > > > > > > https://github.com/foxhlchen/sysfs_benchmark).
> > > > > > >
> > > > > > > machine: aws c5 (Intel Xeon with 96 logical cores)
> > > > > > > kernel: v5.12
> > > > > > > benchmark: create 96 threads and bind them to each core
> > > > > > > then
> > > > > > > run
> > > > > > > open+read+close on a sysfs file simultaneously for 1000
> > > > > > > times.
> > > > > > > result:
> > > > > > > Without the patchset, an open+read+close operation takes
> > > > > > > 550-
> > > > > > > 570
> > > > > > > us,
> > > > > > > perf shows significant time(>40%) spending on mutex_lock.
> > > > > > > After applying it, it takes 410-440 us for that operation
> > > > > > > and
> > > > > > > perf
> > > > > > > shows only ~4% time on mutex_lock.
> > > > > > >
> > > > > > > It's weird, I don't see a huge performance boost compared
> > > > > > > to
> > > > > > > v2,
> > > > > > > even
> > > > > >
> > > > > > I meant I don't see a huge performance boost here and it's
> > > > > > way
> > > > > > worse
> > > > > > than v2.
> > > > > > IIRC, for v2 fastest one only takes 40us
> > > > >
> > > > > Thanks Fox,
> > > > >
> > > > > I'll have a look at those reports but this is puzzling.
> > > > >
> > > > > Perhaps the added overhead of the check if an update is
> > > > > needed is taking more than expected and more than just
> > > > > taking the lock and being done with it. Then there's
> > > > > the v2 series ... I'll see if I can dig out your reports
> > > > > on those too.
> > > >
> > > > Apologies, I was mistaken, it's compared to V3, not V2.  The
> > > > previous
> > > > benchmark report is here.
> > > > https://lore.kernel.org/linux-fsdevel/CAC2o3DKNc=sL2n8291Dpiyb0bRHaX=nd33ogvO_LkJqpBj-YmA@mail.gmail.com/
> > >
> > > Are all these tests using a single file name in the open/read/close
> > > loop?
> >
> > Yes,  because It's easy to implement yet enough to trigger the
> > mutex_lock.
> >
> > And you are right It's not a real-life pattern, but on the bright
> > side, it proves there is no original mutex_lock problem anymore. :)
>
> I've been looking at your reports and they are quite interesting.
>
> >
> > > That being the case the per-object inode lock will behave like a
> > > mutex and once contention occurs any speed benefits of a spinlock
> > > over a mutex (or rwsem) will disappear.
> > >
> > > In this case changing from a write lock to a read lock in those
> > > functions and adding the inode mutex will do nothing but add the
> > > overhead of taking the read lock. And similarly adding the update
> > > check function also just adds overhead and, as we see, once
> > > contention starts it has a cumulative effect that's often not
> > > linear.
> > >
> > > The whole idea of a read lock/per-object spin lock was to reduce
> > > the possibility of contention for paths other than the same path
> > > while not impacting same path accesses too much for an overall
> > > gain. Based on this I'm thinking the update check function is
> > > probably not worth keeping, it just adds unnecessary churn and
> > > has a negative impact for same file contention access patterns.
>
> The reports indicate (to me anyway) that the slowdown isn't
> due to kernfs. It looks more like kernfs is now putting pressure
> on the VFS, mostly on the file table lock but it looks like
> there's a mild amount of contention on a few other locks as well
> now.

That's correct, I ran my benchmark on ext4 the result was similarly
slow. But It shouldn't be that as I remember I tested it before it was
very fast and you can see the result of V3 was much faster. So I ran
this benchmark again on AWS c5a which also has 96 cores but with AMD
CPUs. The result was amazing the fastest one had a 10x boost (~40us)
very similar to the V3 one (see attachment) I guess my previous
benchmark of V3 was run on c5a.

I can't figure why it is so slow on Intel's CPUs, I also tried
C5.metal which is running on the physical machine, the result is still
slow (~200us). But anyway, at least it shows this patchset solves the
mutex_lock problem and can bring even 10x boosts on some occasions.

> That's a whole different problem and those file table handling
> functions don't appear to have any obvious problems so they are
> doing what they have to do and that can't be avoided.
>
> That's definitely out of scope for these changes.
>
> And, as you'd expect, once any appreciable amount of contention
> happens our measurements go out the window, certainly with
> respect to kernfs.
>
> It also doesn't change my option that checking if an inode
> attribute update is needed in kernfs isn't useful since, IIUC
> that file table lock contention would result even if you were
> using different paths.
>
> So I'll drop that patch from the series.
>
> Ian
> > >
> > > I think that using multiple paths, at least one per test process
> > > (so if you are running 16 processes use at least 16 different
> > > files, the same in each process), and selecting one at random
> > > for each loop of the open would better simulate real world
> > > access patterns.
> > >
> > >
> > > Ian
> > >
> >
> >
> > thanks,
> > fox
>
>


thanks,
fox

Download attachment "result.after" of type "application/octet-stream" (4895 bytes)

Download attachment "report.after" of type "application/octet-stream" (236534 bytes)

Download attachment "result.baremetal.after" of type "application/octet-stream" (5055 bytes)

Download attachment "report.baremetal.after" of type "application/octet-stream" (278355 bytes)

Powered by blists - more mailing lists