lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Tue, 30 Jun 2015 15:04:56 -0700 From: Dave Hansen <dave@...1.net> To: dave@...1.net Cc: dave.hansen@...ux.intel.com, viro@...iv.linux.org.uk, linux-fsdevel@...r.kernel.org, linux-kernel@...r.kernel.org, paulmck@...ux.vnet.ibm.com, ak@...ux.intel.com Subject: [PATCH 3/3] fs: replace memory barrier in __sb_end_write() with RCU From: Dave Hansen <dave.hansen@...ux.intel.com> If I sit in a loop and do write()s to small tmpfs files, __sb_end_write() is third-hottest kernel function due to its smp_mb(). __sb_end_write() uses the barrier to avoid races with freeze_super() and its calls to sb_wait_write(). But, now that freeze_super() is calling synchronize_rcu() before each sb_wait_write() call, we can use that to our advantage. The synchronize_rcu() ensures that all __sb_end_write() will see freeze_super()'s updates to s_writers.counter. That, in turn, guarantees that __sb_end_write() will try to wake up any subsequent call by freeze_super() to sb_wait_write(). This improves the number of writes/second I can do by 6.1% on top of the previous patch. The total improvement is 27.1% over a completely unpatched kernel. Signed-off-by: Dave Hansen <dave.hansen@...ux.intel.com> Reviewed-by: Jan Kara <jack@...e.cz> Cc: Alexander Viro <viro@...iv.linux.org.uk> Cc: linux-fsdevel@...r.kernel.org Cc: linux-kernel@...r.kernel.org Cc: Paul E. McKenney <paulmck@...ux.vnet.ibm.com> Cc: Andi Kleen <ak@...ux.intel.com> --- b/fs/super.c | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff -puN fs/super.c~selectively-do-barriers-in-__sb_end_write fs/super.c --- a/fs/super.c~selectively-do-barriers-in-__sb_end_write 2015-06-30 15:03:57.565433061 -0700 +++ b/fs/super.c 2015-06-30 15:03:57.568433196 -0700 @@ -1146,14 +1146,23 @@ out: */ void __sb_end_write(struct super_block *sb, int level) { + rcu_read_lock(); percpu_counter_dec(&sb->s_writers.counter[level-1]); /* - * Make sure s_writers are updated before we wake up waiters in - * freeze_super(). + * We are racing here with freeze_super()'s calls to + * sb_wait_write(). We want to ensure that we call + * wake_up() whenever one of those calls _might_ be + * in sb_wait_write(). + * + * Since freeze_super() does a synchronize_rcu() before + * each of its sb_wait_write() calls, it can guarantee + * that it sees our update to s_writers.counter as well + * as that we see its update to s_writers.frozen. */ - smp_mb(); - if (waitqueue_active(&sb->s_writers.wait)) + if (unlikely(sb->s_writers.frozen >= level)) wake_up(&sb->s_writers.wait); + rcu_read_unlock(); + rwsem_release(&sb->s_writers.lock_map[level-1], 1, _RET_IP_); } EXPORT_SYMBOL(__sb_end_write); _ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists