linux-ext4 - Re: ext4 dbench performance with CONFIG_PREEMPT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1271130777.3469.18.camel@localhost.localdomain>
Date:	Mon, 12 Apr 2010 20:52:57 -0700
From:	john stultz <johnstul@...ibm.com>
To:	tytso@....edu
Cc:	linux-ext4@...r.kernel.org, Mingming Cao <cmm@...ibm.com>,
	keith maanthey <kmannth@...ibm.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...e.hu>, Darren Hart <dvhltc@...ibm.com>
Subject: Re: ext4 dbench performance with CONFIG_PREEMPT_RT

On Thu, 2010-04-08 at 17:10 -0400, tytso@....edu wrote:
> > Any thoughts for ways to rework the state_lock in start_this_handle?
> > (Now that its at the top of the contention logs? :)
> 
> That's going to be much harder.  We're going to have to take
> j_state_lock at some point inside start_this_handle.  We might be able
> to decrease the amount of code which is run while the spinlock is
> taken, but I very much doubt it's possible to eliminate that spinlock
> entirely.
> 
> Do you have detailed lockstat information showing the hold-time and
> wait-time of j_lock_stat (especially in start_this_handle)?

Hey Ted,
	Sorry this took so long. I've been using a fairly large pile of patches
in my testing on top of -rt, and since with -rt lockstat is less useful
(you don't get any of the contention data for mutexes, and the contended
spinlocks are always the internal rtmutex locks), I tried to regenerate
the data on something closer to plain vanilla.

So I ran dbench with 2.6.33, 2.6.33 + Nick Piggin's VFS scalability
patches, and 2.6.33 + Nick's patches + your state-lock patch on an 8 cpu
system. 

Here's the chart of the performance difference:
http://sr71.net/~jstultz/dbench-scalability/graphs/2.6.33-ext4-state-lock/2.6.33_ext4-state-lock.png

Here's the perf log output:
http://sr71.net/~jstultz/dbench-scalability/perflogs/2.6.33-ext4-state-lock/

And finally, as requested, here's the lockstat data:
http://sr71.net/~jstultz/dbench-scalability/lockstat/2.6.33-ext4-state-lock/


Now, again, because the -rt kernel amplifies the contention cost, the
data above doesn't show as much pain at only 8 cpus as we see with -rt.
However, the contention does show up, and your patch helps. 

In fact, with your patch, I'm not seeing any major contention in the
perf logs at 8 cpus. Although the lockstat logs still show:

t_handle_lock contention in start_This_handle/jbd2_journal_stop
	- Likely the j_stat_lock was previously serializing this

j_state_lock contention in start_this_handle
	- Expected

j_revoke_lock contention in find_revoke_record
	- Also observed by Tim Chen



Let me know if there's any other data you'd like to see.

thanks
-john





--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html