lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Sun, 3 Apr 2022 21:37:55 +0900 From: Dominique Martinet <asmadeus@...ewreck.org> To: Christian Schoenebeck <linux_oss@...debyte.com> Cc: v9fs-developer@...ts.sourceforge.net, netdev@...r.kernel.org, Eric Van Hensbergen <ericvh@...il.com>, Latchesar Ionkov <lucho@...kov.net>, Greg Kurz <groug@...d.org>, Vivek Goyal <vgoyal@...hat.com>, Nikolay Kichukov <nikolay@...um.net> Subject: Re: [PATCH v4 12/12] net/9p: allocate appropriate reduced message buffers Christian Schoenebeck wrote on Sun, Apr 03, 2022 at 01:29:53PM +0200: > So maybe I should just exclude the 9p RDMA transport from this 9p message size > reduction change in v5 until somebody had a chance to test this change with > RDMA. Yes, I'm pretty certain it won't work so we'll want to exclude it unless we can extend the RDMA protocol to address buffers. >From my understanding, RDMA comes with two type of primitives: - recv/send that 9p exlusively uses, which is just a pool of buffers registered to the NIC and get filled on a first-come-first-serve basis (I'm not sure if it's a first-fit, or if message will be truncated, or if it'll error out if the message doesn't fit... But basically given that's what we use for 9p we have no way of guaranteeing that a read reply will be filled in the big buffer allocated for it and not something else) If we're lucky the algorithm used is smallest-fit first, but it doesn't look like it: --- The order of the Receive Request consumptions in a Receive Queue is by the order that they were posted to it. When you have a SRQ, you cannot predict which Receive Request will be consumed by which QP, so all the Receive Requests in that SRQ should be able to contain the incoming message (in terms of length). --- https://www.rdmamojo.com/2013/02/02/ibv_post_recv/ (in a comment) - read/write, which can be addressed e.g. the remote end can specify a cookie along with address/size and directly operate on remote memory (hence the "remote direct memory access" name). There are also some cool stuff that can be done like atomic compare and swap or arithmetic operations on remote memory which doesn't really concern us. Using read/writes like NFS over RDMA does would resolve the problem and allow what they call "direct data placement", which is reading or writing directly from the page cache or user buffer as a real zero copy operation, but it requires the cookie/address to be sent and client to act on it so it's a real transport-specific protocol change, but given the low number of users I think that's something that could be considered if someone wants to work on it. Until then we'll be safer with that bit disabled... > Which makes me wonder, what is that exact hardware, hypervisor, OS that > supports 9p & RDMA? I've used it with mellanox infiniband cards in the past. These support SRIOV virtual functions so are quite practical for VMs, could let it do the work with a single machine and no cable. I'm pretty sure it'd work with any recent server hardware that supports RoCE (I -think- it's getting more common?), or with emulation ~10 years ago I got it to run with softiwarp which has been merged in the kernel (siw) since so that might be the easiest way to run it now. Server-side, both diod and nfs-ganesha support 9p over RDMA, I haven't used diod recently but ganesha ought to work. > On the long-term I can imagine to add RDMA transport support on QEMU 9p side. What would you expect it to be used for? > There is already RDMA code in QEMU, however it is only used for migration by > QEMU so far I think. Yes, looking at it a bit there's live migration over RDMA (I tested it at my previous job), some handling for gluster+rdma, and a paravirtualized RDMA device (pvrdma). the docs for it says it works with soft-roce so it would also probably work for tests (I'm not sure what difference there is between rxe and siw), but at this point you've just setup virtualized rdma on the host anyway... I'll try to get something setup for tests on my end as well, it's definitely something I had on my todo... -- Dominique
Powered by blists - more mailing lists