linux-kernel - Re: Latest vfs scalability patch

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20091015100854.GA19948@kryten>
Date:	Thu, 15 Oct 2009 21:08:54 +1100
From:	Anton Blanchard <anton@...ba.org>
To:	Nick Piggin <npiggin@...e.de>
Cc:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	linux-fsdevel@...r.kernel.org,
	Ravikiran G Thirumalai <kiran@...lex86.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Jens Axboe <axboe@...nel.dk>
Subject: Re: Latest vfs scalability patch


Hi Nick,

> Several people have been interested to test my vfs patches, so rather
> than resend patches I have uploaded a rollup against Linus's current
> head.
> 
> ftp://ftp.kernel.org/pub/linux/kernel/people/npiggin/patches/fs-scale/
> 
> I have used ext2,ext3,autofs4,nfs as well as in-memory filesystems
> OK (although this doesn't mean there are no bugs!). Otherwise, if your
> filesystem compiles, then there is a reasonable chance of it working,
> or ask me and I can try updating it for the new locking.
> 
> I would be interested in seeing any numbers people might come up with,
> including single-threaded performance.

Thanks for doing a rollup patch, it made it easy to test. I gave it a spin on
a 64 core (128 thread) POWER5+ box. I started simple by looking at open/close
performance, eg:

void testcase(void)
{
	char tmpfile[] = "/tmp/testfile.XXXXXX";

	mkstemp(tmpfile);

	while (1) {
		int fd = open(tmpfile, O_RDWR);
		close(fd);
	}
}

At first the results were 10x slower. I took a look and it appears the
MNT_MOUNTED flag is getting cleared by a remount (I'm testing on the root
filesystem). This fixed it:

--- fs/namespace.c~	2009-10-15 04:34:02.000000000 -0500
+++ fs/namespace.c	2009-10-15 04:35:00.000000000 -0500
@@ -1711,7 +1711,8 @@ static int do_remount(struct path *path,
 	else
 		err = do_remount_sb(sb, flags, data, 0);
 	if (!err)
-		path->mnt->mnt_flags = mnt_flags;
+		path->mnt->mnt_flags = mnt_flags |
+			(path->mnt->mnt_flags & MNT_MOUNTED);
 	up_write(&sb->s_umount);
 	if (!err) {
 		security_sb_post_remount(path->mnt, flags, data);

Attached is a before and after graph. Single thread performance is 20%
faster, and we go from hitting a wall at 2 cores to scaling all the way
to 64 cores. Nice work!!!

Anton

Download attachment "open_close_performance.png" of type "image/png" (26403 bytes)