netdev - Re: [PATCH net-next v2 0/6] net/smc: Introduce virtually contiguous buffers for SMC-R

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <345053d6-5ecb-066d-8eeb-7637da1d7370@linux.ibm.com>
Date:   Thu, 14 Jul 2022 17:16:47 +0200
From:   Wenjia Zhang <wenjia@...ux.ibm.com>
To:     Wen Gu <guwen@...ux.alibaba.com>, kgraul@...ux.ibm.com,
        davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org,
        pabeni@...hat.com
Cc:     linux-s390@...r.kernel.org, netdev@...r.kernel.org,
        linux-rdma@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH net-next v2 0/6] net/smc: Introduce virtually contiguous
 buffers for SMC-R



On 14.07.22 11:43, Wen Gu wrote:
> On long-running enterprise production servers, high-order contiguous
> memory pages are usually very rare and in most cases we can only get
> fragmented pages.
> 
> When replacing TCP with SMC-R in such production scenarios, attempting
> to allocate high-order physically contiguous sndbufs and RMBs may result
> in frequent memory compaction, which will cause unexpected hung issue
> and further stability risks.
> 
> So this patch set is aimed to allow SMC-R link group to use virtually
> contiguous sndbufs and RMBs to avoid potential issues mentioned above.
> Whether to use physically or virtually contiguous buffers can be set
> by sysctl smcr_buf_type.
> 
> Note that using virtually contiguous buffers will bring an acceptable
> performance regression, which can be mainly divided into two parts:
> 
> 1) regression in data path, which is brought by additional address
>     translation of sndbuf by RNIC in Tx. But in general, translating
>     address through MTT is fast. According to qperf test, this part
>     regression is basically less than 10% in latency and bandwidth.
>     (see patch 5/6 for details)
> 
> 2) regression in buffer initialization and destruction path, which is
>     brought by additional MR operations of sndbufs. But thanks to link
>     group buffer reuse mechanism, the impact of this kind of regression
>     decreases as times of buffer reuse increases.
> 
> Patch set overview:
> - Patch 1/6 and 2/6 mainly about simplifying and optimizing DMA sync
>    operation, which will reduce overhead on the data path, especially
>    when using virtually contiguous buffers;
> - Patch 3/6 and 4/6 introduce a sysctl smcr_buf_type to set the type
>    of buffers in new created link group;
> - Patch 5/6 allows SMC-R to use virtually contiguous sndbufs and RMBs,
>    including buffer creation, destruction, MR operation and access;
> - patch 6/6 extends netlink attribute for buffer type of SMC-R link group;
> 
> v1->v2:
> - Patch 5/6 fixes build issue on 32bit;
> - Patch 3/6 adds description of new sysctl in smc-sysctl.rst;
> 
> Guangguan Wang (2):
>    net/smc: remove redundant dma sync ops
>    net/smc: optimize for smc_sndbuf_sync_sg_for_device and
>      smc_rmb_sync_sg_for_cpu
> 
> Wen Gu (4):
>    net/smc: Introduce a sysctl for setting SMC-R buffer type
>    net/smc: Use sysctl-specified types of buffers in new link group
>    net/smc: Allow virtually contiguous sndbufs or RMBs for SMC-R
>    net/smc: Extend SMC-R link group netlink attribute
> 
>   Documentation/networking/smc-sysctl.rst |  13 ++
>   include/net/netns/smc.h                 |   1 +
>   include/uapi/linux/smc.h                |   1 +
>   net/smc/af_smc.c                        |  68 +++++++--
>   net/smc/smc_clc.c                       |   8 +-
>   net/smc/smc_clc.h                       |   2 +-
>   net/smc/smc_core.c                      | 246 +++++++++++++++++++++-----------
>   net/smc/smc_core.h                      |  20 ++-
>   net/smc/smc_ib.c                        |  44 +++++-
>   net/smc/smc_ib.h                        |   2 +
>   net/smc/smc_llc.c                       |  33 +++--
>   net/smc/smc_rx.c                        |  92 +++++++++---
>   net/smc/smc_sysctl.c                    |  11 ++
>   net/smc/smc_tx.c                        |  10 +-
>   14 files changed, 404 insertions(+), 147 deletions(-)
> 
This idea is very cool! Thank you for your effort! But we still need to 
verify if this solution can run well on our system. I'll come to you soon.