Message-ID: <CADvbK_fkbOnhKL+Rb+pp+NF+VzppOQ68c=nk_6MSNjM_dxpCoQ@mail.gmail.com>
Date: Thu, 14 Mar 2024 12:21:46 -0400
From: Xin Long <lucien.xin@...il.com>
To: Stefan Metzmacher <metze@...ba.org>
Cc: network dev <netdev@...r.kernel.org>, davem@...emloft.net, kuba@...nel.org, 
	Eric Dumazet <edumazet@...gle.com>, Paolo Abeni <pabeni@...hat.com>, 
	Steve French <smfrench@...il.com>, Namjae Jeon <linkinjeon@...nel.org>, 
	Chuck Lever III <chuck.lever@...cle.com>, Jeff Layton <jlayton@...nel.org>, 
	Sabrina Dubroca <sd@...asysnail.net>, Tyler Fanelli <tfanelli@...hat.com>, 
	Pengtao He <hepengtao@...omi.com>, 
	"linux-cifs@...r.kernel.org" <linux-cifs@...r.kernel.org>, 
	Samba Technical <samba-technical@...ts.samba.org>
Subject: Re: [RFC PATCH net-next 0/5] net: In-kernel QUIC implementation with
 Userspace handshake

On Thu, Mar 14, 2024 at 5:21 AM Stefan Metzmacher <metze@...ba.org> wrote:
>
> Am 13.03.24 um 20:39 schrieb Xin Long:
> > On Wed, Mar 13, 2024 at 1:28 PM Stefan Metzmacher <metze@...ba.org> wrote:
> >>
> >> Am 13.03.24 um 17:03 schrieb Xin Long:
> >>> On Wed, Mar 13, 2024 at 4:56 AM Stefan Metzmacher <metze@...ba.org> wrote:
> >>>>
> >>>> Hi Xin Long,
> >>>>
> >>>> first many thanks for working on this topic!
> >>>>
> >>> Hi, Stefan
> >>>
> >>> Thanks for the comment!
> >>>
> >>>>> Usage
> >>>>> =====
> >>>>>
> >>>>> This implementation supports a mapping of QUIC into sockets APIs. Similar
> >>>>> to TCP and SCTP, a typical Server and Client use the following system call
> >>>>> sequence to communicate:
> >>>>>
> >>>>>           Client                    Server
> >>>>>        ------------------------------------------------------------------
> >>>>>        sockfd = socket(IPPROTO_QUIC)      listenfd = socket(IPPROTO_QUIC)
> >>>>>        bind(sockfd)                       bind(listenfd)
> >>>>>                                           listen(listenfd)
> >>>>>        connect(sockfd)
> >>>>>        quic_client_handshake(sockfd)
> >>>>>                                           sockfd = accecpt(listenfd)
> >>>>>                                           quic_server_handshake(sockfd, cert)
> >>>>>
> >>>>>        sendmsg(sockfd)                    recvmsg(sockfd)
> >>>>>        close(sockfd)                      close(sockfd)
> >>>>>                                           close(listenfd)
> >>>>>
> >>>>> Please note that quic_client_handshake() and quic_server_handshake() functions
> >>>>> are currently sourced from libquic in the github lxin/quic repository, and might
> >>>>> be integrated into ktls-utils in the future. These functions are responsible for
> >>>>> receiving and processing the raw TLS handshake messages until the completion of
> >>>>> the handshake process.
> >>>>
> >>>> I see a problem with this design for the server, as one reason to
> >>>> have SMB over QUIC is to use udp port 443 in order to get through
> >>>> firewalls. As QUIC has the concept of ALPN it should be possible
> >>>> let a conumer only listen on a specif ALPN, so that the smb server
> >>>> and web server on "h3" could both accept connections.
> >>> We do provide a sockopt to set ALPN before bind or handshaking:
> >>>
> >>>     https://github.com/lxin/quic/wiki/man#quic_sockopt_alpn
> >>>
> >>> But it's used more like to verify if the ALPN set on the server
> >>> matches the one received from the client, instead of to find
> >>> the correct server.
> >>
> >> Ah, ok.
> > Just note that, with a bit change in the current libquic, it still
> > allows users to use ALPN to find the correct function or thread in
> > the *same* process, usage be like:
> >
> > listenfd = socket(IPPROTO_QUIC);
> > /* match all during handshake with wildcard ALPN */
> > setsockopt(listenfd, QUIC_SOCKOPT_ALPN, "*");
> > bind(listenfd)
> > listen(listenfd)
> >
> > while (1) {
> >    sockfd = accept(listenfd);
> >    /* the alpn from client will be set to sockfd during handshake */
> >    quic_server_handshake(sockfd, cert);
> >
> >    getsockopt(sockfd, QUIC_SOCKOPT_ALPN, alpn);
>
> Would quic_server_handshake() call setsockopt()?
Yes, I just made a bit change in the userspace libquic:

  https://github.com/lxin/quic/commit/9c75bd42769a8cbc1652e2f4c8d77780f23afde6

So you can set up multple ALPNs on listen sock:

  setsockopt(listenfd, QUIC_SOCKOPT_ALPN, "smbd, h3, ksmbd");

Then during handshake, the matched ALPN from client will be set into
the accept socket, then users can get it later after handshake.

Note that userspace libquic is a very light lib (a couple of hundred lines
of code), you can add more TLS related support without touching Kernel code,
including the SNI support you mentioned.

>
> >    switch (alpn) {
> >      case "smbd": smbd_thread(sockfd);
> >      case "h3": h3_thread(sockfd);
> >      case "ksmbd": ksmbd_thread(sockfd);
> >    }
> > }
>
> Ok, but that would mean all application need to be aware of each other,
> but it would be possible and socket fds could be passed to other
> processes.
It doesn't sound common to me, but yes, I think Unix Domain Sockets
can pass it to another process.

>
> >>
> >>> So you expect (k)smbd server and web server both to listen on UDP
> >>> port 443 on the same host, and which APP server accepts the request
> >>> from a client depends on ALPN, right?
> >>
> >> yes.
> > Got you. This can be done by also moving TLS 1.3 message exchange to
> > kernel where we can get the ALPN before looking up the listening socket.
> > However, In-kernel TLS 1.3 Handshake had been NACKed by both kernel
> > netdev maintainers and userland ssl lib developers with good reasons.
> >
> >>
> >>> Currently, in Kernel, this implementation doesn't process any raw TLS
> >>> MSG/EXTs but deliver them to userspace after decryption, and the accept
> >>> socket is created before processing handshake.
> >>>
> >>> I'm actually curious how userland QUIC handles this, considering
> >>> that the UDP sockets('listening' on the same IP:PORT) are used in
> >>> two different servers' processes. I think socket lookup with ALPN
> >>> has to be done in Kernel Space. Do you know any userland QUIC
> >>> implementation for this?
> >>
> >> I don't now, but I guess QUIC is only used for http so
> >> far and maybe dns, but that seems to use port 853.
> >>
> >> So there's no strict need for it and the web server
> >> would handle all relevant ALPNs.
> > Honestly, I don't think any userland QUIC can use ALPN to lookup for
> > different sockets used by different servers/processes. As such thing
> > can be only done in Kernel Space.
> >
> >>
> >>>>
> >>>> So the server application should have a way to specify the desired
> >>>> ALPN before or during the bind() call. I'm not sure if the
> >>>> ALPN is available in cleartext before any crypto is needed,
> >>>> so if the ALPN is encrypted it might be needed to also register
> >>>> a server certificate and key together with the ALPN.
> >>>> Because multiple application may not want to share the same key.
> >>> On send side, ALPN extension is in raw TLS messages created in userspace
> >>> and passed into the kernel and encoded into QUIC crypto frame and then
> >>> *encrypted* before sending out.
> >>
> >> Ok.
> >>
> >>> On recv side, after decryption, the raw TLS messages are decoded from
> >>> the QUIC crypto frame and then delivered to userspace, so in userspace
> >>> it processes certificate validation and also see cleartext ALPN.
> >>>
> >>> Let me know if I don't make it clear.
> >>
> >> But the first "new" QUIC pdu from will trigger the accept() to
> >> return and userspace (or the kernel helper function) will to
> >> all crypto? Or does the first decryption happen in kernel (before accept returns)?
> > Good question!
> >
> > The first "new" QUIC pdu will cause to create a 'request sock' (contains
> > 4-tuple and connection IDs only) and queue up to reqsk list of the listen
> > sock (if validate_peer_address param is not set), and this pdu is enqueued
> > in the inq->backlog_list of the listen sock.
> >
> > When accept() is called, in Kernel, it dequeues the "request sock" from the
> > reqsk list of the listen sock, and creates the accept socket based on this
> > reqsk. Then it processes the pdu for this new accept socket from the
> > inq->backlog_list of the listen sock, including *decrypting* QUIC packet
> > and decoding CRYPTO frame, then deliver the raw/cleartext TLS message to
> > the Userspace libquic.
>
> Ok, when the kernel already decrypts it could already
> look find the ALPN. It doesn't mean it should do the full
> handshake, but parse enough to find the ALPN.
Correct, in-kernel QUIC should only do the QUIC related things,
and all TLS handshake msgs must be handled in Userspace.
This won't cause "layering violation", as Nick Banks said.

>
> But I don't yet understand how the kernel gets the key to
> do the initlal decryption, I'd assume some call before listen()
> need to tell the kernel about the keys.
For initlal decryption, the keys can be derived with the initial packet.
basically, it only needs the dst_connection_id from the client initial
packet. see:

  https://datatracker.ietf.org/doc/html/rfc9001#name-initial-secrets

so we don't need to set up anything to kernel for initial's keys.

But for the handshake, application or early_data keys, they will be set
up into kernel during handshake via:

  setsockopt(QUIC_SOCKOPT_CRYPTO_SECRET)

Thanks.
>
> > Then in Userspace libquic, it handles the received TLS message and creates
> > a new raw/cleartext TLS message for response via libgnutls, and delivers to
> > kernel. In kernel, it will encode this message to a CRYPTO frame in a QUIC
> > packet and then *encrypt* this QUIC packet and send it out.
> >
> > So as you can see, there's no en/decryption happening in Userspace. In
> > Userspace libquic, it only does raw/cleartext TLS message exchange. ALL
> > en/decryption happens in Kernel Space, as these en/decryption are done
> > against QUIC packets, not directly against the TLS messages.
> >
> >>
> >> Maybe it would be possible to optionally have socket option to
> >> register ALPNs with certificates so that tls_server_hello_x509()
> >> could be called automatically before accept returns (even for
> >> userspace consumers).
> >>
> >> It may mean the tlshd protocol needs to be extended...
> >>
> > so that userspace consumers don't need quic_client/server_handshake(), and
> > accept() returns a socket that already has the handshake done, right?
> >
> > We didn't do that, as:
> >
> > 1. It's not a good idea for Userspace consumers' applications to reply on
> >     a daemon like tlshd, not convenient for users, also a bit weird for
> >     userspace app to ask another userspace app to help do the handshake.
> > 2. It's too complex to implement, especially if we also want to call
> >     tls_client_hello_x509() before connect() returns on client side.
> > 3. For Kernel usage, I prefer leaving this to the kernel consumers for
> >     more flexibility for handshake requests.
> >
> > As for the ALPNs with certificates, not sure if I understand correctly.
> > But if you want the server to select certificates according to the ALPN
> > received from the client during handshake. I think it could be done in
> > userspace libquic. But yes, tlshd service may also need to extend.
>
> I was just brainstorming for ideas...
>
> metze

Powered by blists - more mailing lists