lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Wed, 28 Dec 2022 18:26:09 +0800 From: Wen Gu <guwen@...ux.alibaba.com> To: kgraul@...ux.ibm.com, wenjia@...ux.ibm.com, jaka@...ux.ibm.com, davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org, pabeni@...hat.com Cc: linux-s390@...r.kernel.org, netdev@...r.kernel.org, linux-kernel@...r.kernel.org Subject: Re: [RFC PATCH net-next v2 0/5] net/smc:Introduce SMC-D based loopback acceleration On 2022/12/20 11:21, Wen Gu wrote: > Hi, all > > # Background > > As previously mentioned in [1], we (Alibaba Cloud) are trying to use SMC > to accelerate TCP applications in cloud environment, improving inter-host > or inter-VM communication. > > In addition of these, we also found the value of SMC-D in scenario of local > inter-process communication, such as accelerate communication between containers > within the same host. So this RFC tries to provide a SMC-D loopback solution > in such scenario, to bring a significant improvement in latency and throughput > compared to TCP loopback. > > # Design > > This patch set provides a kind of SMC-D loopback solution. > > Patch #1/5 and #2/5 provide an SMC-D based dummy device, preparing for the > inter-process communication acceleration. Except for loopback acceleration, > the dummy device can also meet the requirements mentioned in [2], which is > providing a way to test SMC-D logic for broad community without ISM device. > > +------------------------------------------+ > | +-----------+ +-----------+ | > | | process A | | process B | | > | +-----------+ +-----------+ | > | ^ ^ | > | | +---------------+ | | > | | | SMC stack | | | > | +--->| +-----------+ |<--| | > | | | dummy | | | > | | | device | | | > | +-+-----------+-+ | > | VM | > +------------------------------------------+ > > Patch #3/5, #4/5, #5/5 provides a way to avoid data copy from sndbuf to RMB > and improve SMC-D loopback performance. Through extending smcd_ops with two > new semantic: attach_dmb and detach_dmb, sender's sndbuf shares the same > physical memory region with receiver's RMB. The data copied from userspace > to sender's sndbuf directly reaches the receiver's RMB without unnecessary > memory copy in the same kernel. > > +----------+ +----------+ > | socket A | | socket B | > +----------+ +----------+ > | ^ > | +---------+ | > regard as | | ----------| > local sndbuf | B's | regard as > | | RMB | local RMB > |-------> | | > +---------+ > > # Benchmark Test > > * Test environments: > - VM with Intel Xeon Platinum 8 core 2.50GHz, 16 GiB mem. > - SMC sndbuf/RMB size 1MB. > > * Test object: > - TCP: run on TCP loopback. > - domain: run on UNIX domain. > - SMC lo: run on SMC loopback device with patch #1/5 ~ #2/5. > - SMC lo-nocpy: run on SMC loopback device with patch #1/5 ~ #5/5. > > 1. ipc-benchmark (see [3]) > > - ./<foo> -c 1000000 -s 100 > > TCP domain SMC-lo SMC-lo-nocpy > Message > rate (msg/s) 75140 129548(+72.41) 152266(+102.64%) 151914(+102.17%) > > 2. sockperf > > - serv: <smc_run> taskset -c <cpu> sockperf sr --tcp > - clnt: <smc_run> taskset -c <cpu> sockperf { tp | pp } --tcp --msg-size={ 64000 for tp | 14 for pp } -i 127.0.0.1 -t 30 > > TCP SMC-lo SMC-lo-nocpy > Bandwidth(MBps) 4943.359 4936.096(-0.15%) 8239.624(+66.68%) > Latency(us) 6.372 3.359(-47.28%) 3.25(-49.00%) > > 3. iperf3 > > - serv: <smc_run> taskset -c <cpu> iperf3 -s > - clnt: <smc_run> taskset -c <cpu> iperf3 -c 127.0.0.1 -t 15 > > TCP SMC-lo SMC-lo-nocpy > Bitrate(Gb/s) 40.5 41.4(+2.22%) 76.4(+88.64%) > > 4. nginx/wrk > > - serv: <smc_run> nginx > - clnt: <smc_run> wrk -t 8 -c 500 -d 30 http://127.0.0.1:80 > > TCP SMC-lo SMC-lo-nocpy > Requests/s 154643.22 220894.03(+42.84%) 226754.3(+46.63%) > > > # Discussion > > 1. API between SMC-D and ISM device > > As Jan mentioned in [2], IBM are working on placing an API between SMC-D > and the ISM device for easier use of different "devices" for SMC-D. > > So, considering that the introduction of attach_dmb or detach_dmb can > effectively avoid data copying from sndbuf to RMB and brings obvious > throughput advantages in inter-VM or inter-process scenarios, can the > attach/detach semantics be taken into consideration when designing the > API to make it a standard ISM device behavior? > > Maybe our RFC of SMC-D based inter-process acceleration (this one) and > inter-VM acceleration (will coming soon, which is the update of [1]) The patch of SMC-D + virtio-ism device is now discussed in virtio community: https://lists.oasis-open.org/archives/virtio-comment/202212/msg00030.html > can provide some examples for new API design. And we are very glad to > discuss this on the mail list. > > 2. Way to select different ISM-like devices > > With the proposal of SMC-D loopback 'device' (this RFC) and incoming > device used for inter-VM acceleration as update of [1], SMC-D has more > options to choose from. So we need to consider that how to indicate > supported devices, how to determine which one to use, and their priority... > > IMHO, this may require an update of CLC message and negotiation mechanism. > Again, we are very glad to discuss this with you on the mailing list. > > [1] https://lore.kernel.org/netdev/20220720170048.20806-1-tonylu@linux.alibaba.com/ > [2] https://lore.kernel.org/netdev/35d14144-28f7-6129-d6d3-ba16dae7a646@linux.ibm.com/ > [3] https://github.com/goldsborough/ipc-bench > > v1->v2 > 1. Fix some build WARNINGs complained by kernel test rebot > Reported-by: kernel test robot <lkp@...el.com> > 2. Add iperf3 test data. > > Wen Gu (5): > net/smc: introduce SMC-D loopback device > net/smc: choose loopback device in SMC-D communication > net/smc: add dmb attach and detach interface > net/smc: avoid data copy from sndbuf to peer RMB in SMC-D loopback > net/smc: logic of cursors update in SMC-D loopback connections > > include/net/smc.h | 3 + > net/smc/Makefile | 2 +- > net/smc/af_smc.c | 88 +++++++++++- > net/smc/smc_cdc.c | 59 ++++++-- > net/smc/smc_cdc.h | 1 + > net/smc/smc_clc.c | 4 +- > net/smc/smc_core.c | 62 +++++++++ > net/smc/smc_core.h | 2 + > net/smc/smc_ism.c | 39 +++++- > net/smc/smc_ism.h | 2 + > net/smc/smc_loopback.c | 358 +++++++++++++++++++++++++++++++++++++++++++++++++ > net/smc/smc_loopback.h | 63 +++++++++ > 12 files changed, 662 insertions(+), 21 deletions(-) > create mode 100644 net/smc/smc_loopback.c > create mode 100644 net/smc/smc_loopback.h >
Powered by blists - more mailing lists