Message-ID: <668dac0bd73ba_1ce27f2945b@willemb.c.googlers.com.notmuch>
Date: Tue, 09 Jul 2024 17:30:51 -0400
From: Willem de Bruijn <willemdebruijn.kernel@...il.com>
To: Zijian Zhang <zijianzhang@...edance.com>, 
 Willem de Bruijn <willemdebruijn.kernel@...il.com>, 
 netdev@...r.kernel.org
Cc: edumazet@...gle.com, 
 cong.wang@...edance.com, 
 xiaochun.lu@...edance.com
Subject: Re: [PATCH net-next v7 1/3] sock: support copying cmsgs to the user
 space in sendmsg

Zijian Zhang wrote:
> On 7/9/24 9:40 AM, Willem de Bruijn wrote:
> > zijianzhang@ wrote:
> >> From: Zijian Zhang <zijianzhang@...edance.com>
> >>
> >> Users can pass msg_control as a placeholder to recvmsg, and get some info
> >> from the kernel upon returning of it, but it's not available for sendmsg.
> >> Recvmsg uses put_cmsg to copy info back to the user, while ____sys_sendmsg
> >> creates a kernel copy of msg_control and passes that to the callees,
> >> put_cmsg in sendmsg path will write into this kernel buffer.
> >>
> >> If users want to get info after returning of sendmsg, they typically have
> >> to call recvmsg on the ERRMSG_QUEUE of the socket, incurring extra system
> > 
> > nit: error queue or MSG_ERRQUEUE
> > 
> >> call overhead. This commit supports copying cmsg from the kernel space to
> >> the user space upon returning of sendmsg to mitigate this overhead.
> >>
> >> Signed-off-by: Zijian Zhang <zijianzhang@...edance.com>
> >> Signed-off-by: Xiaochun Lu <xiaochun.lu@...edance.com>
> > 
> > Overall this approach follows what I had in mind, thanks.
> > 
> > Looking forward to the discussion with a wider audience at netdevconf
> > next week.
> > 
> 
> No problem, see you next week ;)
> 
> >> ---
> >>   include/linux/socket.h |  6 +++++
> >>   include/net/sock.h     |  2 +-
> >>   net/core/sock.c        |  6 +++--
> >>   net/ipv4/ip_sockglue.c |  2 +-
> >>   net/ipv6/datagram.c    |  2 +-
> >>   net/socket.c           | 54 ++++++++++++++++++++++++++++++++++++++----
> >>   6 files changed, 62 insertions(+), 10 deletions(-)
> >>
> >> diff --git a/include/linux/socket.h b/include/linux/socket.h
> >> index 2a1ff91d1914..75461812a7a3 100644
> >> --- a/include/linux/socket.h
> >> +++ b/include/linux/socket.h
> >> @@ -71,6 +71,7 @@ struct msghdr {
> >>   		void __user	*msg_control_user;
> >>   	};
> >>   	bool		msg_control_is_user : 1;
> >> +	bool		msg_control_copy_to_user : 1;
> >>   	bool		msg_get_inq : 1;/* return INQ after receive */
> >>   	unsigned int	msg_flags;	/* flags on received message */
> >>   	__kernel_size_t	msg_controllen;	/* ancillary data buffer length */
> >> @@ -168,6 +169,11 @@ static inline struct cmsghdr * cmsg_nxthdr (struct msghdr *__msg, struct cmsghdr
> >>   	return __cmsg_nxthdr(__msg->msg_control, __msg->msg_controllen, __cmsg);
> >>   }
> >>   
> >> +static inline bool cmsg_copy_to_user(struct cmsghdr *__cmsg)
> >> +{
> >> +	return 0;
> >> +}
> >> +
> >>   static inline size_t msg_data_left(struct msghdr *msg)
> >>   {
> >>   	return iov_iter_count(&msg->msg_iter);
> >> diff --git a/include/net/sock.h b/include/net/sock.h
> >> index cce23ac4d514..9c728287d21d 100644
> >> --- a/include/net/sock.h
> >> +++ b/include/net/sock.h
> >> @@ -1804,7 +1804,7 @@ static inline void sockcm_init(struct sockcm_cookie *sockc,
> >>   	};
> >>   }
> >>   
> >> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
> >>   		     struct sockcm_cookie *sockc);
> >>   int sock_cmsg_send(struct sock *sk, struct msghdr *msg,
> >>   		   struct sockcm_cookie *sockc);
> >> diff --git a/net/core/sock.c b/net/core/sock.c
> >> index 9abc4fe25953..efb30668dac3 100644
> >> --- a/net/core/sock.c
> >> +++ b/net/core/sock.c
> >> @@ -2826,7 +2826,7 @@ struct sk_buff *sock_alloc_send_pskb(struct sock *sk, unsigned long header_len,
> >>   }
> >>   EXPORT_SYMBOL(sock_alloc_send_pskb);
> >>   
> >> -int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >> +int __sock_cmsg_send(struct sock *sk, struct msghdr *msg, struct cmsghdr *cmsg,
> >>   		     struct sockcm_cookie *sockc)
> >>   {
> >>   	u32 tsflags;
> >> @@ -2866,6 +2866,8 @@ int __sock_cmsg_send(struct sock *sk, struct cmsghdr *cmsg,
> >>   	default:
> >>   		return -EINVAL;
> >>   	}
> >> +	if (cmsg_copy_to_user(cmsg))
> >> +		msg->msg_control_copy_to_user = true;
> > 
> > This seems a bit roundabout.
> > 
> > Just have case SCM_ZC_NOTIFICATION set this bit directly?
> 
> If I directly set this bit in SCM_ZC_... and delete this if code block,
> I may have to add "msg" argument to __sock_cmsg_send in the second
> commit, because if I still keep it in this commit, there will be an
> "unused argument" warning.
> 
> However, I think the change to __sock_cmsg_send function declaration is
> generic, so I would like to make it in the first commit, but it is truly
> a bit roundabout. Not sure which way is better?

A temporary __attribute__((unused))
 
> >>   	return 0;
> >>   }
> > 
> >> -static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >> -			   unsigned int flags, struct used_address *used_address,
> >> +static int sendmsg_copy_cmsg_to_user(struct msghdr *msg_sys,
> >> +				     struct user_msghdr __user *umsg)
> >> +{
> >> +	struct compat_msghdr __user *umsg_compat =
> >> +				(struct compat_msghdr __user *)umsg;
> >> +	unsigned int flags = msg_sys->msg_flags;
> >> +	struct msghdr msg_user = *msg_sys;
> >> +	unsigned long cmsg_ptr;
> >> +	struct cmsghdr *cmsg;
> >> +	int err;
> >> +
> >> +	msg_user.msg_control_is_user = true;
> >> +	msg_user.msg_control_user = umsg->msg_control;
> >> +	cmsg_ptr = (unsigned long)msg_user.msg_control;
> >> +	for_each_cmsghdr(cmsg, msg_sys) {
> >> +		if (!CMSG_OK(msg_sys, cmsg))
> >> +			break;
> >> +		if (cmsg_copy_to_user(cmsg))
> >> +			put_cmsg(&msg_user, cmsg->cmsg_level, cmsg->cmsg_type,
> >> +				 cmsg->cmsg_len - sizeof(*cmsg), CMSG_DATA(cmsg));
> >> +	}
> >> +
> >> +	err = __put_user((msg_sys->msg_flags & ~MSG_CMSG_COMPAT), COMPAT_FLAGS(umsg));
> >> +	if (err)
> >> +		return err;
> >> +	if (MSG_CMSG_COMPAT & flags)
> >> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> >> +				 &umsg_compat->msg_controllen);
> >> +	else
> >> +		err = __put_user((unsigned long)msg_user.msg_control - cmsg_ptr,
> >> +				 &umsg->msg_controllen);
> >> +	return err;
> >> +}
> >> +
> >> +static int ____sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> >> +			   struct msghdr *msg_sys, unsigned int flags,
> >> +			   struct used_address *used_address,
> >>   			   unsigned int allowed_msghdr_flags)
> >>   {
> >>   	unsigned char ctl[sizeof(struct cmsghdr) + 20]
> >> @@ -2537,6 +2572,7 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >>   	ssize_t err;
> >>   
> >>   	err = -ENOBUFS;
> >> +	msg_sys->msg_control_copy_to_user = false;
> >>   
> >>   	if (msg_sys->msg_controllen > INT_MAX)
> >>   		goto out;
> >> @@ -2594,6 +2630,14 @@ static int ____sys_sendmsg(struct socket *sock, struct msghdr *msg_sys,
> >>   			       used_address->name_len);
> >>   	}
> >>   
> >> +	if (msg && msg_sys->msg_control_copy_to_user && err >= 0) {
> >> +		ssize_t len = err;
> >> +
> >> +		err = sendmsg_copy_cmsg_to_user(msg_sys, msg);
> >> +		if (!err)
> >> +			err = len;
> >> +	}
> >> +
> > 
> > The main issue is adding the above initialization and this branch in
> > the hot path, adding a minor cost to every other send call only for
> > this use case (and potentially tx timestamps eventually).
> > 
> >>   out_freectl:
> >>   	if (ctl_buf != ctl)
> >>   		sock_kfree_s(sock->sk, ctl_buf, ctl_len);
> >> @@ -2636,8 +2680,8 @@ static int ___sys_sendmsg(struct socket *sock, struct user_msghdr __user *msg,
> >>   	if (err < 0)
> >>   		return err;
> >>   
> >> -	err = ____sys_sendmsg(sock, msg_sys, flags, used_address,
> >> -				allowed_msghdr_flags);
> >> +	err = ____sys_sendmsg(sock, msg, msg_sys, flags, used_address,
> >> +			      allowed_msghdr_flags);
> > 
> > Does it make more sense to do the copy_to_user here, so as not to have to plumb
> > msg down to the callee?
> 
> I did this in the previous patchset. The problem is that the msg_control
> of msg_sys is either a stack pointer or kmalloc-ed pointer (in
> ____sys_sendmsg), after returning of it, the msg_control of msg_sys is
> either invalid or freed. I may have to do the copy_to_user at the end of
> ____sys_sendmsg.

I see. Ack.

Powered by blists - more mailing lists