lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Thu, 16 Jun 2011 09:10:44 -0700 From: Randy Dunlap <rdunlap@...otime.net> To: Bernard Metzler <bmt@...ich.ibm.com> Cc: netdev@...r.kernel.org, linux-rdma@...r.kernel.org Subject: Re: [PATCH 14/14] SIWv2: Documentation: siw.txt On Thu, 16 Jun 2011 14:42:54 +0200 Bernard Metzler wrote: > --- > Documentation/networking/siw.txt | 156 ++++++++++++++++++++++++++++++++++++++ > 1 files changed, 156 insertions(+), 0 deletions(-) > create mode 100644 Documentation/networking/siw.txt > > diff --git a/Documentation/networking/siw.txt b/Documentation/networking/siw.txt > new file mode 100644 > index 0000000..805e21b > --- /dev/null > +++ b/Documentation/networking/siw.txt > @@ -0,0 +1,156 @@ > +SoftiWARP: Software iWARP kernel driver module. > + > +General > +------- > +SoftiWARP (siw) implements the iWARP protocol suite (MPA/DDP/RDMAP, > +IETF-RFC 5044/5041/5040) completely in software as a Linux kernel module. > +siw runs on top of TCP kernel sockets and exports the Linux kernel ibverbs > +RDMA interface. siw interfaces with the iwcm connection manager. > + > + > +Transmit Path > +------------- > +If a send queue (SQ) work queue element gets posted, siw tries to send > +it directly out of the application context. If the SQ was non-empty, > +SQ processing is done asynchronously by a kernel worker thread. This > +thread gets scheduled if the TCP socket signals new write space to s/gets/is/ > +be available. If during send operation the socket send space becomes > +exhausted, SQ processing is abandoned until new socket write space > +becomes available. > + > + > +Receive Path > +------------ > +All application data is placed into target buffers within softirq > +socket callback. Application notification is asynchronous. > + > + > +User Interface > +-------------- > +All user space fast path operations such as posting of work requests and > +reaping of work completions currently involve a isynchronous call into If you really mean "isynchronous", then it should be: an isynchronous call but what is isynchronous? > +the siw kernel module via ib_uverbs interface. Kernel/user-mapped send > +and receive as well as completion queues are not part of the current code. > +In particular, mapped completion queues may improve performance, > +since reaping completion queue entries as well as re-arming > +the completion queue could be done more efficiently. > + > + > +Kernel Client Support > +--------------------- > +To guarantee non-blocking fast path operations, for kernel clients > +all work queue elements (send/receive/shared-receive queue) are > +pre-allocated during connection resource setup. > + > + > +Memory Management > +----------------- > +siw currently uses the ib_umem_get() function of the ib_core module > +to pin memory for later use in data transfer operations. Transmit > +and receive memory are checked against correct access permissions only > +in the moment of access by the network input path or before pushing it at the moment > +to the TCP socket for transmission. > +ib_umem_get() provides DMA mappings for the requested address space which > +are not used by siw. > + > + > +Module Parameters > +----------------- > +The following siw module parameters are recognized. > + > +loopback_enabled: > + If set, siw attaches also to the looback device. Checked only > + during module insertion. > + > +mpa_crc_required: > + If set, the MPA CRC gets generated and checked both in tx and rx s/gets/is/ > + path. Without hardware support, setting this flag will severely > + hurt throughput. Default setting is 0 (off). > + > +mpa_crc_strict: > + If set, MPA CRC will not be enabled, even if peer requests > + it. If the peer requests CRC generation, the connection setup > + will be aborted. Default setting is 1 (on). > + > +zcopy_tx: > + If set, payload of non-signalled work requests payloads ... are transferred > + (such as non-signalled WRITE or SEND as well as all READ > + responses) are transferred using the TCP sockets > + sendpage interface. This parameter can be switched on and > + off dynamically (echo 1 >> /sys/module/siw/parameters/zcopy_tx > + for enablement, 0 for disabling). System load may benefits from may benefit > + using 0copy data transmission. 0copy is not enabled if "0copy" is fugly (IMO). > + mpa_crc_enabled is set. Default setting is 1 (on). > + > +tcp_nodelay: > + If set, on the TCP socket the TCP_NODELAY option is set. > + Default setting is 1 (on). > + > +iface_list: > + Comma separated list of interfaces siw should attach to. Comma-separated > + If no list is given, siw attaches to all available devices. > + If a list is given, siw skips those devices not listed. > + Currently, the list is restricted to 12 entries. If needed, > + the 'SIW_MAX_IF' #define in siw_main.c can be adaped. adapted. ? (or modified) > + This parameter might be usefull to skip devices which are useful > + attached to a real RNIC device. Default setting is an empty list. > + > + > +Compile Time Flags: > +------------------- > +-DCHECK_DMA_CAPABILITIES > + Checks if the device siw wants to attach to provides > + DMA capabilities. While DMA capabilities are currently not > + needed (siw works on top of a kernel TCP socket), siw > + uses ib_umem_get() which performs a (not used) DMA address > + translation. Writing a siw private memory reservation and > + pinning routine would solve the issue. > + > +-DSIW_TX_FULLSEGS > + Experimental, not enabled by default. If set, > + siw tries not to overrun the socket (not sending until > + -EAGAIN return), but stops sending if the current segment > + would not fit into the socket's estimated tx buffer. With that, > + wire FPDUs may get truncated by the TCP stack far less often. > + Since this feature manipulates the sock's SOCK_NOSPACE > + bit, it violates strict layering and is therefore considered > + proprietary. > + Since TCP is a byte stream protocol, no guarantee can be given > + if FPDU's are not fragmented. or FPDUs > + > + > +Debugging SIW: > +-------------- > +The siw_debug.h file defines a 'dprint' macro which is used to debug > +siw at runtime. Verbosity of debugging is controlled at compile time > +via setting the 'DPRINT_MASK' to a or'd list of know value as defined to an or'd list of known value > +in siw_debug.h, e.g. '#define DPRINT_MASK (DBG_ON|DBG_CM)' to debug > +errors and connection management. Defining DPRINT_MASK to '0' avoids > +to compile any runtime debugging code. compiling any > + > +To track siw's useage of its objects (connection endpoints, tcp sockets, usage > +protection domains, queue pairs, shared receive queues, completion queues, > +memory registrations, work queue elements), the /sys/class/infiniband/siw* > +directory contains siw interface specific objects, which can be read to > +gather simple statistics: > + > +/sys/class/infiniband/siw*/stats: > + Summary of allocated WQE's, PD's, QP's, CQ's, SRQ's, MR's, CEP's. All of those single quote/apostrophe marks are not needed. > + WQE statistics are not gathered if 'DPRINT_MASK' is set to '0' > + (see above). > + > +/sys/class/infiniband/siw*/qp: > + Summary of allocated queue pairs. If queue pairs are allocated, > + after reading 'qp' a more detailed status of all queue pairs has > + been printed to the kernel syslog and can be retrieved via > + 'dmesg' command. > + > +/sys/class/infiniband/siw*/cep: > + Summary of allocated connection end points. If connection endpoints > + are allocated, after reading 'cep' a more detailed status of all > + CEP's is printed to the kernel syslog and can be retrieved via ditto > + 'dmesg' command. > + > +Using the sysfs to gather siw's object allocations is considered a > +tentative aid during further driver development and should disappear > +in a stable version of siw. > -- HTH. --- ~Randy *** Remember to use Documentation/SubmitChecklist when testing your code *** -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists