linux-kernel - Re: [RFC][PATCH RT] rwsem_rt: Another (more sane) approach to mulit reader rt locks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <1337701801.13348.56.camel@gandalf.stny.rr.com>
Date:	Tue, 22 May 2012 11:50:01 -0400
From:	Steven Rostedt <rostedt@...dmis.org>
To:	Thomas Gleixner <tglx@...utronix.de>
Cc:	LKML <linux-kernel@...r.kernel.org>,
	RT <linux-rt-users@...r.kernel.org>,
	Clark Williams <williams@...hat.com>,
	Peter Zijlstra <peterz@...radead.org>
Subject: Re: [RFC][PATCH RT] rwsem_rt: Another (more sane) approach to mulit
 reader rt locks

On Tue, 2012-05-22 at 17:26 +0200, Thomas Gleixner wrote:
> On Tue, 15 May 2012, Steven Rostedt wrote:
> > +struct rw_semaphore {
> > +	int			initialized;
> > +	struct __rw_semaphore	lock[NR_CPUS];
> 
> So that will blow up every rw_semaphore user by
> 
>    NR_CPUS * sizeof(struct __rw_semaphore)
> 
> With lockdep off thats: NR_CPUS * 48
> 
> With lockdep on thats:  NR_CPUS * 128 + NR_CPUS * 8 (__key)
> 
> So for NR_CPUS=64 that's 3072 or 8704 Bytes.

For a box that has 64 CPUS, 8k should be nothing (even for every task).
But then again, NR_CPUS is compile time option. It would be nice if we
could make NR_CPUS just what was actually available :-/

> 
> That'll make e.g. XFS happy. xfs_inode has two rw_sems.
> 
> sizeof(xfs_inode) in mainline is:  856 bytes
> 
> sizeof(xfs_inode) on RT is:       1028 bytes
> 
> But with your change it would goto (NR_CPUS = 64):
> 
>     1028 - 96 + 2 * 3072 =        7076 bytes
> 
> That's almost an order of magnitude!
> 
> NFS has an rwsem in the inode as well, and ext4 has two.
> 
> So we trade massive memory waste for how much performance? 

We could always make this an option. I may be able to also do linker
tricks to make it a boot time option where the memory is allocated in
sections that can be freed if the option is not enabled. Just a thought,
I know this is making it more complex than necessary.

> 
> We really need numbers for various scenarios. There are applications
> which are pretty mmap heavy and it would really surprise me when
> taking NR_CPUS locks in one go is not going to cause a massive
> overhead.

Well, it doesn't take NR_CPUS locks, it takes possible_cpus() locks,
which may be much smaller. As a compiled time NR_CPUS=64 running on a
box with just 4 cpus will do a loop of 4 and not 64.

I'm all for benchmarks. But right now, making all readers pass through a
single mutex is a huge bottle neck for a lot of loads. Yes, they are
mostly Java loads, but for some strange reason, our customers seems to
like to run Java on our RT kernel :-p

-- Steve

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/