linux-ext4 - Re: ext4 dbench performance with CONFIG_PREEMPT

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100409233325.GF1849@thunk.org>
Date:	Fri, 9 Apr 2010 19:33:26 -0400
From:	tytso@....edu
To:	Andi Kleen <andi@...stfloor.org>
Cc:	john stultz <johnstul@...ibm.com>, linux-ext4@...r.kernel.org,
	Mingming Cao <cmm@...ibm.com>,
	keith maanthey <kmannth@...ibm.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	Ingo Molnar <mingo@...e.hu>, Darren Hart <dvhltc@...ibm.com>,
	tim.c.chen@...el.com
Subject: Re: ext4 dbench performance with CONFIG_PREEMPT_RT

On Fri, Apr 09, 2010 at 05:49:54PM +0200, Andi Kleen wrote:
> john stultz <johnstul@...ibm.com> writes:
> >
> > Further using lockstat I was able to isolate it the contention down to
> > the journal j_state_lock, and then adding some lock owner tracking, I
> > was able to see that the lock owners were almost always in
> > start_this_handle, and jbd2_journal_stop when we saw contention (with
> > the freq breakdown being about 55% in jbd2_journal_stop and 45% in
> > start_this_handle).
> 
> FWIW we've been also seeing this on larger systems without RT.
> The journal locks are the number one contention in some workloads.
> So it's not just a RT problem.

Yeah, I'm very much aware of that.  What worries me is that locking
problems in the jbd2 layer could be very hard to debug, so we need to
make sure we have some really good testing as we make any changes.

Not taking the j_state_lock spinlock in jbd2_stop_lock() was relatively
easy to prove to be safe, but I'm really worried about
start_this_handle() the locking around that is going to be subtle, and
it's not just the specific fields in the transaction and journal
handle.

And even with the jbd2_stop_lock() change, I'd really prefer some
pretty exhaustive testing, including power fail testing, just to make
sure we're in practice when/if we make more subtle or more invasive
changes to the jbd2 layer...

So I'm mot waving the red flag, but the yellow flag (as they would say
in auto racing circles).

Regards,

						- Ted
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html