Date: Tue, 25 Jun 2024 23:44:30 +0600
From: Dorjoy Chowdhury <dorjoychy111@...il.com>
To: Stefano Garzarella <sgarzare@...hat.com>
Cc: Alexander Graf <graf@...zon.com>, Paolo Bonzini <pbonzini@...hat.com>, 
	Alexander Graf <agraf@...raf.de>, virtualization@...ts.linux.dev, kvm@...r.kernel.org, 
	netdev@...r.kernel.org, stefanha@...hat.com
Subject: Re: How to implement message forwarding from one CID to another in
 vhost driver

Hey Stefano,

On Wed, May 29, 2024 at 4:56 PM Stefano Garzarella <sgarzare@...hat.com> wrote:
>
> On Wed, May 29, 2024 at 12:43:57PM GMT, Alexander Graf wrote:
> >
> >On 29.05.24 10:04, Stefano Garzarella wrote:
> >>
> >>On Tue, May 28, 2024 at 06:38:24PM GMT, Paolo Bonzini wrote:
> >>>On Tue, May 28, 2024 at 5:53 PM Stefano Garzarella
> >>><sgarzare@...hat.com> wrote:
> >>>>
> >>>>On Tue, May 28, 2024 at 05:49:32PM GMT, Paolo Bonzini wrote:
> >>>>>On Tue, May 28, 2024 at 5:41 PM Stefano Garzarella
> >>>><sgarzare@...hat.com> wrote:
> >>>>>> >I think it's either that or implementing virtio-vsock in userspace
> >>>>>> >(https://lore.kernel.org/qemu-devel/30baeb56-64d2-4ea3-8e53-6a5c50999979@redhat.com/,
> >>>>>> >search for "To connect host<->guest").
> >>>>>>
> >>>>>> For in this case AF_VSOCK can't be used in the host, right?
> >>>>>> So it's similar to vhost-user-vsock.
> >>>>>
> >>>>>Not sure if I understand but in this case QEMU knows which CIDs are
> >>>>>forwarded to the host (either listen on vsock and connect to the host,
> >>>>>or vice versa), so there is no kernel and no VMADDR_FLAG_TO_HOST
> >>>>>involved.
> >>>>
> >>>>I meant that the application in the host that wants to connect to the
> >>>>guest cannot use AF_VSOCK in the host, but must use the one where QEMU
> >>>>is listening (e.g. AF_INET, AF_UNIX), right?
> >>>>
> >>>>I think one of Alex's requirements was that the application in the host
> >>>>continue to use AF_VSOCK as in their environment.
> >>>
> >>>Can the host use VMADDR_CID_LOCAL for host-to-host communication?
> >>
> >>Yep!
> >>
> >>>If
> >>>so, the proposed "-object vsock-forward" syntax can connect to it and
> >>>it should work as long as the application on the host does not assume
> >>>that it is on CID 3.
> >>
> >>Right, good point!
> >>We can also support something similar in vhost-user-vsock, where instead
> >>of using AF_UNIX and firecracker's hybrid vsock, we can redirect
> >>everything to VMADDR_CID_LOCAL.
> >>
> >>Alex what do you think? That would simplify things a lot to do.
> >>The only difference is that the application in the host has to talk to
> >>VMADDR_CID_LOCAL (1).
> >
> >
> >The application in the host would see an incoming connection from CID
> >1 (which is probably fine) and would still be able to establish
> >outgoing connections to the actual VM's CID as long as the Enclave
> >doesn't check for the peer CID (I haven't seen anyone check yet). So
> >yes, indeed, this should work.
> >
> >The only case where I can see it breaking is when you run multiple
> >Enclave VMs in parallel. In that case, each would try to listen to CID
> >3 and the second that does would fail. But it's a well solvable
> >problem: We could (in addition to the simple in-QEMU case) build an
> >external daemon that does the proxying and hence owns CID3.
>
> Well, we can modify vhost-user-vsock for that. It's already a daemon,
> already supports different VMs per single daemon but as of now they have
> to have different CIDs.
>
> >
> >So the immediate plan would be to:
> >
> >  1) Build a new vhost-vsock-forward object model that connects to
> >vhost as CID 3 and then forwards every packet from CID 1 to the
> >Enclave-CID and every packet that arrives on to CID 3 to CID 2.
>
> This though requires writing completely from scratch the virtio-vsock
> emulation in QEMU. If you have time that would be great, otherwise if
> you want to do a PoC, my advice is to start with vhost-user-vsock which
> is already there.
>

Can you give me some more details about how I can implement the
daemon? I would appreciate some pointers to code too.

Right now, the "nitro-enclave" machine type (wip) in QEMU
automatically spawns a VHOST_VSOCK device with the CID equal to the
"guest-cid" machine option. I think this is equivalent to using the
"-device vhost-vsock-device,guest-cid=N" option explicitly. Does that
need any change? I guess instead of "vhost-vsock-device", the
vhost-vsock device needs to be equivalent to "-device
vhost-user-vsock-device,guest-cid=N"?

The applications inside the nitro-enclave VM will still connect and
talk to CID 3. So on the daemon side, do we need to spawn a device
that has CID 3 and then forward everything this device receives to CID
1 (VMADDR_CID_LOCAL) same port and everything it receives from CID 1
to the "guest-cid"? The applications that will be running in the host
need to be changed so that instead of connecting to the "guest-cid" of
the nitro-enclave VM, they will instead connect to VMADDR_CID_LOCAL.
Is my understanding correct?

BTW is there anything related to the "VMADDR_FLAG_TO_HOST" flag that
needs to be checked? I remember some discussion about it.

It would be great if you could give me some details about how I can
achieve the CID 3 <-> CID 2 communication using the vhost-user-vsock.
Is this https://github.com/stefano-garzarella/vhost-user-vsock where I
would need to add support for forwarding everything to
VMADDR_CID_LOCAL via an option maybe?

Thanks and Regards,
Dorjoy

Powered by blists - more mailing lists