Message-ID: <579fee5c-fdfb-4305-9f64-231c0d8ebabd@antgroup.com>
Date: Fri, 23 Aug 2024 11:28:28 +0800
From: "沈安琪(凛玥)" <amy.saq@...group.com>
To: Paolo Abeni <pabeni@...hat.com>, netdev@...r.kernel.org
Cc: "David S. Miller" <davem@...emloft.net>,
 Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 linux-kernel@...r.kernel.org
Subject: Re: [PATCH] net: fix csum calculation for encapsulated packets


在 8/22/24 下午6:05, Paolo Abeni 写道:
>
>
> On 8/19/24 13:17, 沈安琪(凛玥) wrote:
>> This commit fixes the issue that when a packet is encapsulated, such as
>> sending through a UDP tunnel, the outer TCP/UDP checksum is not
>> correctly recalculated if (1) checksum has been offloaded to hardware
>> and (2) encapsulated packet has been NAT-ed again, which causes the
>> packet being dropped due to the invalid outer checksum.
>>
>> Previously, when an encapsulated packet met some NAT rules and its
>> src/dst ip and/or src/dst port has been modified,
>> inet_proto_csum_replace4 will be invoked to recalculated the outer
>> checksum. However, if the packet is under the following condition: (1)
>> checksum offloaded to hardware and (2) NAT rule has changed the src/dst
>> port, its outer checksum will not be recalculated, since (1)
>> skb->ip_summed is set to CHECKSUM_PARTIAL due to csum offload and (2)
>> pseudohdr is set to false since port number is not part of pseudo
>> header. 
>
> I don't see where nat is calling inet_proto_csum_replace4() with 
> pseudohdr == false: please include more detailed description of the 
> relevant setup (ideally a self-test) or at least a backtrace leading 
> to the issue.


The relevant setup we found this issue was:

1. we setup a VXLAN tunnel and set a MASQUERADE rule with --random-fully 
enabled on client side (tx side).

2. we enabled. NIC checksum offload.


We used the following bpftrace script to get the call trace:

bpftrace -o trace.log -e '#include <linux/skbuff.h>
#include <linux/net.h>
#include <linux/ip.h>
#include <linux/udp.h>
#include <linux/if_ether.h>
#include <linux/types.h>
#include <net/sock.h>
#include <net/route.h>
#include <net/dst.h>
kprobe:inet_proto_csum_replace4
{
     $skb = (struct sk_buff *)arg1;
     $ip_hdr = (struct iphdr *)($skb->head + $skb->network_header);
     $proto = $ip_hdr->protocol;
     if ($proto == 17) {
         $udp_hdr = (struct udphdr *)($skb->head + $skb->transport_header);
         printf("udp check: 0x%04x; \n", $udp_hdr->check);
         printf("skb ip_summed: %d, skb encapsulation: %d, skb 
encap_csum_hdr: %d; \n", $skb->ip_summed, $skb->encapsulation, 
$skb->encap_hdr_csum);
         printf("from: 0x%08x, to: 0x%08x; \n", arg2, arg3);
         printf("pseudohdr: %d; \n", arg4);
         printf("enter kprobe:inet_proto_csum_replace4 : %s\n\n", kstack);
     }
}
'


And get the following call trace:

udp check: [...];
skb ip_summed: 3, skb encapsulation: 1, skb encap_csum_hdr: 0;
from: ...[old src ip], to: ...[NAT-ed src ip];
pseudohdr: 1;
enter kprobe:inet_proto_csum_replace4 :
     inet_proto_csum_replace4+1
     l4proto_manip_pkt+1166
     nf_nat_ipv4_manip_pkt+90
     nf_nat_manip_pkt+141
     nf_nat_ipv4_out+76
     nf_hook_slow+57
     ip_output+225
     iptunnel_xmit+356
     vxlan_xmit_one+3184
     vxlan_xmit+823
     xmit_one.constprop.0+149
     dev_hard_start_xmit+80
     __dev_queue_xmit+962
     ip_finish_output2+417
     ip_send_skb+56
     udp_send_skb+337
     udp_sendmsg+2404
     sock_sendmsg+51
     ____sys_sendmsg+487
     ___sys_sendmsg+117
     __sys_sendmsg+89
     do_syscall_64+45
     entry_SYSCALL_64_after_hwframe+68


udp check: [...];
skb ip_summed: 3, skb encapsulation: 1, skb encap_csum_hdr: 0;
from: ...[old src port], to: ...[new src port];
pseudohdr: 0;
enter kprobe:inet_proto_csum_replace4 :
     inet_proto_csum_replace4+1
     l4proto_manip_pkt+1195
     nf_nat_ipv4_manip_pkt+90
     nf_nat_manip_pkt+141
     nf_nat_ipv4_out+76
     nf_hook_slow+57
     ip_output+225
     iptunnel_xmit+356
     vxlan_xmit_one+3184
     vxlan_xmit+823
     xmit_one.constprop.0+149
     dev_hard_start_xmit+80
     __dev_queue_xmit+962
     ip_finish_output2+417
     ip_send_skb+56
     udp_send_skb+337
     udp_sendmsg+2404
     sock_sendmsg+51
     ____sys_sendmsg+487
     ___sys_sendmsg+117
     __sys_sendmsg+89
     do_syscall_64+45
     entry_SYSCALL_64_after_hwframe+68



We trace the current implementation and found in __udp_manip_pkt

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/net/netfilter/nf_nat_proto.c#n57
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/tree/include/net/checksum.h#n164

will invoke inet_proto_csum_replace4 with pseudohdr as false.


>
>> This leads to the outer TCP/UDP checksum invalid since it does
>> not change along with the port number change.
>>
>> In this commit, another condition has been added to recalculate outer
>> checksum: if (1) the packet is encapsulated, (2) checksum has been
>> offloaded, (3) the encapsulated packet has been NAT-ed to change port
>> number and (4) outer checksum is needed, the outer checksum for
>> encapsulated packet will be recalculated to make sure it is valid.
>
> Please add a suitable fix tag.


Thanks for notifying this. We will add it in the next version of this patch.


>
>> Signed-off-by: Anqi Shen <amy.saq@...group.com>
>> ---
>>   net/core/utils.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/net/core/utils.c b/net/core/utils.c
>> index c994e95172ac..d9de60e9b347 100644
>> --- a/net/core/utils.c
>> +++ b/net/core/utils.c
>> @@ -435,6 +435,8 @@ void inet_proto_csum_replace4(__sum16 *sum, 
>> struct sk_buff *skb,
>>           *sum = ~csum_fold(csum_add(csum_sub(csum_unfold(*sum),
>>                               (__force __wsum)from),
>>                          (__force __wsum)to));
>> +    else if (skb->encapsulation && !!(*sum))
>> +        csum_replace4(sum, from, to);
>
> This looks incorrect for a csum partial value, and AFAICS the nat 
> caller has already checked for !!(*sum).
>
> Thanks,
>
> Paolo

Powered by blists - more mailing lists