netdev - Re: [PATCH net-next] Revert "RDS: IB: split the mr registration and invalidation path"

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <956db880-8bb0-442f-4db4-c51c538a618f@oracle.com>
Date:   Wed, 20 Mar 2019 11:39:41 -0500
From:   "santosh.shilimkar@...cle.com" <santosh.shilimkar@...cle.com>
To:     Gerd Rausch <gerd.rausch@...cle.com>, netdev@...r.kernel.org
Cc:     David Miller <davem@...emloft.net>
Subject: Re: [PATCH net-next] Revert "RDS: IB: split the mr registration and
 invalidation path"

On 11/6/18 4:52 PM, Gerd Rausch wrote:
> This reverts commit 56012459310a1dbcc55c2dbf5500a9f7571402cb.
> 
> Subject: rds_ib_post_reg_frmr spins and spins and spins
> 
> RDS kept spinning inside function "rds_ib_post_reg_frmr", waiting for
> "i_fastreg_wrs" to become incremented:
>          while (atomic_dec_return(&ibmr->ic->i_fastreg_wrs) <= 0) {
>                  atomic_inc(&ibmr->ic->i_fastreg_wrs);
>                  cpu_relax();
>          }
> 
> Looking at the original commit:
> 
> commit 56012459310a ("RDS: IB: split the mr registration and invalidation path")
> 
> In there, the "rds_ib_mr_cqe_handler" was changed in the following way:
> 
>> @@ -289,9 +289,10 @@ void rds_ib_mr_cqe_handler(struct rds_ib_connection *ic,
>> struct ib_wc *wc)
>>          if (frmr->fr_inv) {
>>                  frmr->fr_state = FRMR_IS_FREE;
>>                  frmr->fr_inv = false;
>> +               atomic_inc(&ic->i_fastreg_wrs);
>> +       } else {
>> +               atomic_inc(&ic->i_fastunreg_wrs);
>>          }
>> -
>> -       atomic_inc(&ic->i_fastreg_wrs);
>>   }
> 
> It looks like it's got it exactly backwards:
> 
> Function "rds_ib_post_reg_frmr" keeps track of the outstanding requests
> via "i_fastreg_wrs".
> 
> Function "rds_ib_post_inv" keeps track of the outstanding requests
> via "i_fastunreg_wrs" (post original commit). It also sets:
>          frmr->fr_inv = true;
> 
> However the completion handler "rds_ib_mr_cqe_handler" adjusts "i_fastreg_wrs"
> when "fr_inv" had been true, and adjusts "i_fastunreg_wrs" otherwise.
> 
> The original commit was done in the name of performance:
>> to remove the performance bottleneck
> 
> No performance benefit could be observed with a fixed-up version
> of the original commit measured between two Oracle X7 servers,
> both equipped with Mellanox Connect-X5 HCAs.
> 
> The prudent course of action is to revert this commit.
> 
> Signed-off-by: Gerd Rausch <gerd.rausch@...cle.com>
> ---
As mentioned in offline discussion, when we added this change, the test
server was equipped with Connect-X3 NIC running with FRWR mode and it
did show step improvements. Unfortunately I don't have that data
stored, so it is just from memory.

But looking at Connext-X5 numbers(thanks), am fine to back this out.

Acked-by: Santosh Shilimkar <santosh.shilimkar@...cle.com>