Message-ID: <20250806154122.3413330-1-wintera@linux.ibm.com>
Date: Wed,  6 Aug 2025 17:41:05 +0200
From: Alexandra Winter <wintera@...ux.ibm.com>
To: David Miller <davem@...emloft.net>, Jakub Kicinski <kuba@...nel.org>,
        Paolo Abeni <pabeni@...hat.com>, Eric Dumazet <edumazet@...gle.com>,
        Andrew Lunn <andrew+netdev@...n.ch>,
        "D. Wythe" <alibuda@...ux.alibaba.com>,
        Dust Li <dust.li@...ux.alibaba.com>,
        Sidraya Jayagond <sidraya@...ux.ibm.com>,
        Wenjia Zhang <wenjia@...ux.ibm.com>,
        Julian Ruess <julianr@...ux.ibm.com>
Cc: netdev@...r.kernel.org, linux-s390@...r.kernel.org,
        Heiko Carstens <hca@...ux.ibm.com>, Vasily Gorbik <gor@...ux.ibm.com>,
        Alexander Gordeev <agordeev@...ux.ibm.com>,
        Christian Borntraeger <borntraeger@...ux.ibm.com>,
        Sven Schnelle <svens@...ux.ibm.com>,
        Thorsten Winkler <twinkler@...ux.ibm.com>,
        Simon Horman <horms@...nel.org>,
        Mahanta Jambigi <mjambigi@...ux.ibm.com>,
        Tony Lu <tonylu@...ux.alibaba.com>, Wen Gu <guwen@...ux.alibaba.com>,
        Halil Pasic <pasic@...ux.ibm.com>, linux-rdma@...r.kernel.org
Subject: [RFC net-next 00/17] dibs - Direct Internal Buffer Sharing

This series introduces a generic abstraction of existing components like:
- the s390 specific ISM device (Internal Shared Memory),
- the SMC-D loopback mechanism (Shared Memory Communication - Direct)
- the client interface of the SMC-D module to the transport devices
This generic shim layer can be extended with more devices, more clients and
more features in the future.

This layer is called 'dibs' for Direct Internal Buffer Sharing based on the
common scheme that these mechanisms enable controlled sharing of memory
buffers within some containing entity such as a hypervisor or a Linux
instance.

Benefits:
- Cleaner separation of ISM and SMC-D functionality
- simpler and less module dependencies
- Clear interface definition.
- Extendable for future devices and clients.

An overview was given at the Netdev 0x19 conference, recordings and slides
are available [1].

Background / Status quo:
------------------------
Currently s390 hardware provides virtual PCI ISM devices (Internal Shared
Memory). Their driver is in drivers/s390/net/ism_drv.c. The main user is
SMC-D (net/smc). The ism driver offers a client interface so other
users/protocols can also use them, but it is still heavily intermingled
with the smc code. Namely, the ism module cannot be used without the smc
module, which feels artificial.

There is ongoing work to extend the ISM concept of shared buffers that can
be accessed directly by another instance on the same hardware: [2] proposed
a loopback interface (ism_lo), that can be used on non-s390 architectures
(e.g. between containers or to test SMC-D). A minimal implementation went
upstream with [3]: ism_lo currently is a part of the smc protocol and
rather hidden.

[4] proposed a virtio definition of ism (ism_virtio) that can be used
between kvm guests.

We will shortly send an RFC for an dibs client that uses dibs as transport
for TTY.

Concept:
--------
Create a shim layer in net/dibs that contains common definitions and code
for all dibs devices and all dibs clients. Any device or client module only
needs to depend on this dibs layer module and any device or client code
only needs to include the definitions in include/linux/dibs.h.

The name dibs was chosen to clearly distinguish it from the existing s390
ism devices. And to emphasize that it is not about sharing whole memory
regions with anybody, but dedicating single buffers for another system.

Implementation:
---------------
The end result of this series is: A dibs shim layer with
One dibs client: smc-d
Two dibs device drivers: ism and dibs-loopback
Everything prepared to add more clients and more device drivers.

As net-next is still closed, I am sending this series as RFC to give
reviewers an early start (vacation season). This series is based on
net-next tag 'bpf-next-6.17' and contains everything I want to include for
the initial implementation.

Patches 1-5 contain some issues that were found along the way. They make
sense on their own, but also enable a better structured dibs series.

There are three components that exist today:
a) smc module (especially SMC-D functionality, which is an ism client today)
b) ism device driver (supports multiple ism clients today)
c) smc-loopback (integrated with smc today)
In order to preserve existing functionality at each step, these are not
moved to dibs layer by component, instead:
- the dibs layer is established in parallel to existing code [patches 6-9]
- then some service functions are moved to the dibs layer [patches 10-15]
- the actual data movement is moved to the dibs layer [patch 16]
- and last event handling is moved to the dibs layer [patch 17]

Future:
-------
Items that are not part of this patchset but could be added later:
- dynamically add or remove dibs_loopback. That will be allow for simple
  testing of add_dev()/del_dev()
- handle_irq(): Call clients without interrupt context. e.g using
  threaded interrupts. I left this for a follow-on, because it includes
  conceptual changes for the smcd receive code.
- Any improvements of locking scopes. I mainly moved some of the the
  existing locks to dibs layer. I have the feeling there is room for
  improvements.

Link: [1] https://netdevconf.info/0x19/sessions/talk/communication-via-internal-shared-memory-ism-time-to-open-up.html
Link: [2] https://lore.kernel.org/netdev/1695568613-125057-1-git-send-email-guwen@linux.alibaba.com/
Link: [3] https://lore.kernel.org/linux-kernel//20240428060738.60843-1-guwen@linux.alibaba.com/
Link: [4] https://groups.oasis-open.org/communities/community-home/digestviewer/viewthread?GroupId=3973&MessageKey=c060ecf9-ea1a-49a2-9827-c92f0e6447b2&CommunityKey=2f26be99-3aa1-48f6-93a5-018dce262226&hlmlt=VT

Alexandra Winter (14):
  net/smc: Remove __init marker from smc_core_init()
  s390/ism: Log module load/unload
  net/smc: Remove error handling of unregister_dmb()
  net/smc: Decouple sf and attached send_buf in smc_loopback
  net/smc: Improve log message for devices w/o pnetid
  net/dibs: Create net/dibs
  net/dibs: Register smc as dibs_client
  net/dibs: Register ism as dibs device
  net/dibs: Define dibs loopback
  net/dibs: Define dibs_client_ops and dibs_dev_ops
  net/dibs: Local gid for dibs devices
  net/dibs: Move vlan support to dibs_dev_ops
  net/dibs: Move query_remote_gid() to dibs_dev_ops
  net/dibs: Move data path to dibs layer

Julian Ruess (3):
  net/dibs: Move struct device to dibs_dev
  net/dibs: Create class dibs
  net/dibs: Move event handling to dibs layer

 MAINTAINERS                |   9 +-
 drivers/s390/net/Kconfig   |   3 +-
 drivers/s390/net/ism.h     |  53 +++-
 drivers/s390/net/ism_drv.c | 580 ++++++++++++++-----------------------
 include/linux/dibs.h       | 479 ++++++++++++++++++++++++++++++
 include/linux/ism.h        |  93 ------
 include/net/smc.h          |  51 +---
 net/Kconfig                |   1 +
 net/Makefile               |   1 +
 net/dibs/Kconfig           |  27 ++
 net/dibs/Makefile          |   8 +
 net/dibs/dibs_loopback.c   | 355 +++++++++++++++++++++++
 net/dibs/dibs_loopback.h   |  57 ++++
 net/dibs/dibs_main.c       | 280 ++++++++++++++++++
 net/smc/Kconfig            |  16 +-
 net/smc/Makefile           |   1 -
 net/smc/af_smc.c           |  12 +-
 net/smc/smc_clc.c          |   6 +-
 net/smc/smc_core.c         |   8 +-
 net/smc/smc_core.h         |   5 +
 net/smc/smc_diag.c         |   2 +-
 net/smc/smc_ib.c           |  18 +-
 net/smc/smc_ism.c          | 233 ++++++++-------
 net/smc/smc_ism.h          |  36 ++-
 net/smc/smc_loopback.c     | 421 ---------------------------
 net/smc/smc_loopback.h     |  60 ----
 net/smc/smc_pnet.c         |   8 +-
 net/smc/smc_tx.c           |   3 +
 28 files changed, 1672 insertions(+), 1154 deletions(-)
 create mode 100644 include/linux/dibs.h
 delete mode 100644 include/linux/ism.h
 create mode 100644 net/dibs/Kconfig
 create mode 100644 net/dibs/Makefile
 create mode 100644 net/dibs/dibs_loopback.c
 create mode 100644 net/dibs/dibs_loopback.h
 create mode 100644 net/dibs/dibs_main.c
 delete mode 100644 net/smc/smc_loopback.c
 delete mode 100644 net/smc/smc_loopback.h

-- 
2.48.1

Powered by blists - more mailing lists