linux-kernel - Re: ext4 performance falloff

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 8 Apr 2014 20:30:27 +1000
From:	Dave Chinner <david@...morbit.com>
To:	Andi Kleen <andi@...stfloor.org>
Cc:	Jan Kara <jack@...e.cz>, Daniel J Blueman <daniel@...ascale.com>,
	Theodore Ts'o <tytso@....edu>, linux-ext4@...r.kernel.org,
	LKML <linux-kernel@...r.kernel.org>,
	Steffen Persvold <sp@...ascale.com>,
	Andreas Dilger <adilger.kernel@...ger.ca>
Subject: Re: ext4 performance falloff

On Mon, Apr 07, 2014 at 09:40:28AM -0700, Andi Kleen wrote:
> Jan Kara <jack@...e.cz> writes:
> >
> > What we really need is a counter where we can better estimate counts
> > accumulated in the percpu part of it. As the counter approaches zero, it's
> > CPU overhead will have to become that of a single locked variable but when
> > the value of counter is relatively high, we want it to be fast as the
> > percpu one.  Possibly, each CPU could "reserve" part of the value in the
> > counter (by just decrementing the total value; how large that part should
> > be really needs to depend to the total value of the counter and number of
> > CPUs - in this regard we really differ from classical percpu couters) and
> > allocate/free using that part. If CPU cannot reserve what it is asked for
> > anymore, it would go and steal from parts other CPUs have accumulated,
> > returning them to global pool until it can satisfy the allocation.

Yup, that's pretty much what the slow path/fast path breakdown of
the xfs_icsb_* (XFS In-Core Super Block) code in fs/xfs/xfs_mount.c
does. :)

It distributes free space across all the CPUs and
rebalances them when a per-CPu counter runs out. And to avoid lots
of rebalances when ENOSPC approaches (512 blocks per CPU, IIRC),
it disables the per-CPU counters completely and falls back to a
global counter protected by a mutex to avoid wasting hundreds of
CPUs spinning on a contended global lock. When the free space goes
back above that threshold, it returns to per-cpu mode (the fast
path code).

> That's a percpu_counter() isn't it? (or cookie jar)

No. percpu_counters do not guarantee accuracy nor can the counters
be externally serialised for things like concurrent ENOSPC detection
that require a guarantee that the counter never, ever goes below
zero.

> The MM uses similar techniques.

I haven't seen anything else that uses similar techniques to the XFS
code - I wrote it back in 2005 before there was generic per-cpu
counter infrastructure, and I've been keeping an eye out as
to whether it could be replaced with generic code ever since....

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/