Message-ID: <ef67fdfc-d5f1-79af-064d-997c13adbea7@akamai.com>
Date:   Wed, 18 Sep 2019 19:59:02 -0700
From:   Josh Hunt <johunt@...mai.com>
To:     Willem de Bruijn <willemdebruijn.kernel@...il.com>
Cc:     netdev <netdev@...r.kernel.org>,
        Eric Dumazet <edumazet@...gle.com>,
        David Miller <davem@...emloft.net>
Subject: Re: udp sendmsg ENOBUFS clarification

On 9/18/19 8:35 AM, Willem de Bruijn wrote:
> On Tue, Sep 17, 2019 at 4:20 PM Josh Hunt <johunt@...mai.com> wrote:
>>
>> I was running some tests recently with the udpgso_bench_tx benchmark in
>> selftests and noticed that in some configurations it reported sending
>> more than line rate! Looking into it more I found that I was overflowing
>> the qdisc queue and so it was sending back NET_XMIT_DROP however this
>> error did not propagate back up to the application and so it assumed
>> whatever it sent was done successfully. That's when I learned about
>> IP_RECVERR and saw that the benchmark isn't using that socket option.
>>
>> That's all fairly straightforward, but what I was hoping to get
>> clarification on is where is the line drawn on when or when not to send
>> ENOBUFS back to the application if IP_RECVERR is *not* set? My guess
>> based on going through the code is that as long as the packet leaves the
>> stack (in this case sent to the qdisc) that's where we stop reporting
>> ENOBUFS back to the application, but can someone confirm?
> 
> Once a packet is queued the system call may return, so any subsequent
> drops after dequeue are not propagated back. The relevant rc is set in
> __dev_xmit_skb on q->enqueue. On setups with multiple devices, such as
> a tunnel or bonding path, enqueue on the lower device is similar not
> propagated.

Yeah that makes total sense. Once it's enqueued you'd expect it to not 
be able to return an error, but in this particular case we get an error 
on enqueue so was surprised when it did not get back to the application.

> 
>> For example, we sanitize the error in udp_send_skb():
>> send:
>>           err = ip_send_skb(sock_net(sk), skb);
>>           if (err) {
>>                   if (err == -ENOBUFS && !inet->recverr) {
>>                           UDP_INC_STATS(sock_net(sk),
>>                                         UDP_MIB_SNDBUFERRORS, is_udplite);
>>                           err = 0;
>>                   }
>>           } else
>>
>>
>> but in udp_sendmsg() we don't:
>>
>>           if (err == -ENOBUFS || test_bit(SOCK_NOSPACE,
>> &sk->sk_socket->flags)) {
>>                   UDP_INC_STATS(sock_net(sk),
>>                                 UDP_MIB_SNDBUFERRORS, is_udplite);
>>           }
>>           return err;
> 
> That's interesting. My --incorrect-- understanding until now had been
> that IP_RECVERR does nothing but enable optional extra detailed error
> reporting on top of system call error codes.
> 
> But indeed it enables backpressure being reported as a system call
> error that is suppressed otherwise. I don't know why. The behavior
> precedes git history.

Yeah it's interesting. I wasn't able to find any documentation or 
discussion on it either which is why I figured I'd ask the question on 
netdev in case others know.

> 
>> In the case above it looks like we may only get ENOBUFS for allocation
>> failures inside of the stack in udp_sendmsg() and so that's why we
>> propagate the error back up to the application?
> 
> Both the udp lockless fast path and the slow corked path go through
> udp_send_skb, so the backpressure is suppressed consistently across
> both cases.
> 
> Indeed the error handling in udp_sendmsg then is not related to
> backpressure, but to other causes of ENOBUF, i.e., allocation failure.
> 

Yep. Thanks for going through this. We'll see if others have any 
comments. I will likely send a patch for the man page adding that you 
can get ENOBUFS on Linux but need to set IP_RECVERR as Eric pointed out 
in the patch I linked to previously.

Josh

Powered by blists - more mailing lists