linux-kernel - Re: [PATCH] locking/rwsem: Optimize write lock slowpath

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 9 May 2016 22:25:54 -0400
From:	Waiman Long <waiman.long@....com>
To:	Jason Low <jason.low2@...com>
CC:	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...hat.com>, <linux-kernel@...r.kernel.org>,
	Davidlohr Bueso <dave@...olabs.net>,
	Scott J Norton <scott.norton@....com>,
	<peter@...leysoftware.com>, Jason Low <jason.low2@....com>
Subject: Re: [PATCH] locking/rwsem: Optimize write lock slowpath

On 05/09/2016 03:16 PM, Jason Low wrote:
> When acquiring the rwsem write lock in the slowpath, we first try
> to set count to RWSEM_WAITING_BIAS. When that is successful,
> we then atomically add the RWSEM_WAITING_BIAS in cases where
> there are other tasks on the wait list. This causes write lock
> operations to often issue multiple atomic operations.
>
> We can instead make the list_is_singular() check first, and then
> set the count accordingly, so that we issue at most 1 atomic
> operation when acquiring the write lock and reduce unnecessary
> cacheline contention.
>
> Signed-off-by: Jason Low<jason.low2@...com>
> ---
>   kernel/locking/rwsem-xadd.c | 20 +++++++++++++-------
>   1 file changed, 13 insertions(+), 7 deletions(-)
>
> diff --git a/kernel/locking/rwsem-xadd.c b/kernel/locking/rwsem-xadd.c
> index df4dcb8..23c33e6 100644
> --- a/kernel/locking/rwsem-xadd.c
> +++ b/kernel/locking/rwsem-xadd.c
> @@ -258,14 +258,20 @@ EXPORT_SYMBOL(rwsem_down_read_failed);
>   static inline bool rwsem_try_write_lock(long count, struct rw_semaphore *sem)
>   {
>   	/*
> -	 * Try acquiring the write lock. Check count first in order
> -	 * to reduce unnecessary expensive cmpxchg() operations.
> +	 * Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS.
>   	 */
> -	if (count == RWSEM_WAITING_BIAS&&
> -	    cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS,
> -		    RWSEM_ACTIVE_WRITE_BIAS) == RWSEM_WAITING_BIAS) {
> -		if (!list_is_singular(&sem->wait_list))
> -			rwsem_atomic_update(RWSEM_WAITING_BIAS, sem);
> +	if (count != RWSEM_WAITING_BIAS)
> +		return false;
> +
> +	/*
> +	 * Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there
> +	 * are other tasks on the wait list, we need to add on WAITING_BIAS.
> +	 */
> +	count = list_is_singular(&sem->wait_list) ?
> +			RWSEM_ACTIVE_WRITE_BIAS :
> +			RWSEM_ACTIVE_WRITE_BIAS + RWSEM_WAITING_BIAS;
> +
> +	if (cmpxchg_acquire(&sem->count, RWSEM_WAITING_BIAS, count) == RWSEM_WAITING_BIAS) {
>   		rwsem_set_owner(sem);
>   		return true;
>   	}

Acked-by: Waiman Long<Waiman.Long@....com>