Date:   Thu, 17 Feb 2022 12:33:06 +0100
From:   Niklas Schnelle <schnelle@...ux.ibm.com>
To:     Tony Lu <tonylu@...ux.alibaba.com>
Cc:     netdev@...r.kernel.org, linux-s390@...r.kernel.org,
        linux-rdma@...r.kernel.org, kgraul@...ux.ibm.com,
        Wenjia Zhang <wenjia@...ux.ibm.com>,
        Stefan Raspl <raspl@...ux.ibm.com>
Subject: Re: [PATCH 0/4] RDMA device net namespace support for SMC

On Tue, 2021-12-28 at 21:06 +0800, Tony Lu wrote:
> This patch set introduces net namespace support for linkgroups.
> 
> Path 1 is the main approach to implement net ns support.
> 
> Path 2 - 4 are the additional modifications to let us know the netns.
> Also, I will submit changes of smc-tools to github later.
> 
> Currently, smc doesn't support net namespace isolation. The ibdevs
> registered to smc are shared for all linkgroups and connections. When
> running applications in different net namespaces, such as container
> environment, applications should only use the ibdevs that belongs to the
> same net namespace.
> 
> This adds a new field, net, in smc linkgroup struct. During first
> contact, it checks and find the linkgroup has same net namespace, if
> not, it is going to create and initialized the net field with first
> link's ibdev net namespace. When finding the rdma devices, it also checks
> the sk net device's and ibdev's net namespaces. After net namespace
> destroyed, the net device and ibdev move to root net namespace,
> linkgroups won't be matched, and wait for lgr free.
> 
> If rdma net namespace exclusive mode is not enabled, it behaves as
> before.
> 
> Steps to enable and test net namespaces:
> 
> 1. enable RDMA device net namespace exclusive support
> 	rdma system set netns exclusive # default is shared
> 
> 2. create new net namespace, move and initialize them
> 	ip netns add test1 
> 	rdma dev set mlx5_1 netns test1
> 	ip link set dev eth2 netns test1
> 	ip netns exec test1 ip link set eth2 up
> 	ip netns exec test1 ip addr add ${HOST_IP}/26 dev eth2
> 
> 3. setup server and client, connect N <-> M
> 	ip netns exec test1 smc_run sockperf server --tcp # server
> 	ip netns exec test1 smc_run sockperf pp --tcp -i ${SERVER_IP} # client
> 
> 4. netns isolated linkgroups (2 * 2 mesh) with their own linkgroups
>   - server

Hi Tony,

I'm having a bit of trouble getting this to work for me and was
wondering if you could test my scenario or help me figure out what's
wrong.

I'm using network namespacing to be able to test traffic between two
VFs of the same card/port with a single Linux system. By having one VF
in each of a client and server namespace, traffic doesn't shortcut via
loopback. This works great for TCP and with "rdma system set netns
exclusive" I can also verify that RDMA with "qperf -cm1 ... rc_bw" only
works once the respective RDMA device is also added to each namespace.

When I try the same with SMC-R I tried:

  ip netns exec server smc_run qperf &
  ip netns exec client smc_run qperf <ip_server> tcp_bw

With that however I only see fallback TCP connections in "ip netns exec
client watch smc_dbg". It doesn't seem to be an "smc_dbg" problem
either since the performance with and without smc_run is the same. I
also do have the same PNET_ID set on the interfaces.

As an aside do you know how to gracefully put the RDMA devices back
into the default namespace? For network interfaces I can use "ip -n
<ns> link set dev <iface> netns 1" but the equivalent "ip netns exec
<ns> rdma dev set <rdmadev> netns 1" doesn't work because there is no
PID variant. Deleting the namespace and killing processes using the
RDMA device does seem to get it back but with some delay.

Thanks,
Niklas

Powered by blists - more mailing lists