netdev - Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow control support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <a1c8957b-8ca8-fa84-105f-17619b2c8371@linux.ibm.com>
Date:   Mon, 31 Jan 2022 13:56:15 +0100
From:   Karsten Graul <kgraul@...ux.ibm.com>
To:     Guangguan Wang <guangguan.wang@...ux.alibaba.com>,
        Stefan Raspl <raspl@...ux.ibm.com>
Cc:     linux-s390@...r.kernel.org, netdev@...r.kernel.org,
        linux-kernel@...r.kernel.org, kuba@...nel.org, davem@...emloft.net
Subject: Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow
 control support

On 29/01/2022 04:43, Guangguan Wang wrote:
> 
> On 2022/1/25 17:42, Stefan Raspl wrote:
>>
>> That's some truly substantial improvements!
>> But we need to be careful with protocol-level changes: There are other operating systems like z/OS and AIX which have compatible implementations of SMC, too. Changes like a reduction of connections per link group or usage of reserved fields would need to be coordinated, and likely would have unwanted side-effects even when used with older Linux kernel versions.
>> Changing the protocol is "expensive" insofar as it requires time to thoroughly discuss the changes, perform compatibility tests, and so on.
>> So I would like to urge you to investigate alternative ways that do not require protocol-level changes to address this scenario, e.g. by modifying the number of completion queue elements, to see if this could yield similar results.
>>
>> Thx!
>>
> 
> Yes, there are alternative ways, as RNR caused by the missmatch of send rate and receive rate, which means sending too fast
> or receiving too slow. What I have done in this patch is to backpressure the sending side when sending too fast.
> 
> Another solution is to process and refill the receive queue as quickly as posibble, which requires no protocol-level change. 
> The fllowing modifications are needed:
> - Enqueue cdc msgs to backlog queues instead of processing in rx tasklet. llc msgs remain unchanged.
> - A mempool is needed as cdc msgs are processed asynchronously. Allocate new receive buffers from mempool when refill receive queue.
> - Schedule backlog queues to other cpus, which are calculated by 4-tuple or 5-tuple hash of the connections, to process the cdc msgs,
>   in order to reduce the usage of the cpu where rx tasklet runs on.
> 
> the pseudocode shows below:
> rx_tasklet
>     if cdc_msgs
>         enqueue to backlog;
> 	maybe smp_call_function_single_async is needed to wakeup the corresponding cpu to process backlog;
>         allocate new buffer and modify the sge in rq_wr;
>     else
>         process remains unchanged;
>     endif
> 
>     post_recv rq_wr;
> end rx_tasklet
> 
> smp_backlog_process in corresponding cpu, called by smp_call_function_single_async
>     for connections hashed to this cpu
>         for cdc_msgs in backlog
>             process cdc msgs;
>         end cdc_msgs
>     end connections
> end smp_backlog_process
> 
> I‘d like to hear your suggestions of this solution.
> Thank you.

I like this idea, this should improve the RX handling a lot!