netdev - Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow control support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YfTBj6o8fpcWAfpL@TonyMac-Alibaba>
Date:   Sat, 29 Jan 2022 12:24:47 +0800
From:   Tony Lu <tonylu@...ux.alibaba.com>
To:     Guangguan Wang <guangguan.wang@...ux.alibaba.com>
Cc:     Stefan Raspl <raspl@...ux.ibm.com>, kgraul@...ux.ibm.com,
        linux-s390@...r.kernel.org, netdev@...r.kernel.org,
        linux-kernel@...r.kernel.org, kuba@...nel.org, davem@...emloft.net
Subject: Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow
 control support

On Sat, Jan 29, 2022 at 11:43:22AM +0800, Guangguan Wang wrote:
> 
> Yes, there are alternative ways, as RNR caused by the missmatch of send rate and receive rate, which means sending too fast
> or receiving too slow. What I have done in this patch is to backpressure the sending side when sending too fast.
> 
> Another solution is to process and refill the receive queue as quickly as posibble, which requires no protocol-level change. 
> The fllowing modifications are needed:
> - Enqueue cdc msgs to backlog queues instead of processing in rx tasklet. llc msgs remain unchanged.

It's a good idea to use backlog to free the work in softirq. Rx backlog
can help move the heavy logic out of softirq, and let extra kthread or
workqueue to handle it, then let kernel scheduler to deal with the
balance between userspace process and kthread.

There are two things to be careful, one for introducing more latency,
this should trade off latency and throughput, the other for backlog full.

> - A mempool is needed as cdc msgs are processed asynchronously. Allocate new receive buffers from mempool when refill receive queue.

Yes, we need a elastically expanding RX buffer, also elastically
shrinking. This looks like tcp_mem with tree elements to limit the
memory usage. We also need to free memory automatically, based on memcg
pressure is a good idea.

> - Schedule backlog queues to other cpus, which are calculated by 4-tuple or 5-tuple hash of the connections, to process the cdc msgs,
>   in order to reduce the usage of the cpu where rx tasklet runs on.

I am wondering if it is need for now. In general, it should spread the
CPU usage to different cores. The memory usage or CPU usage which one
will reach its limitation before trigger RNR. Maybe there should some
data to support it?

Thank you,
Tony Lu