netdev - [BUG] Thunderbolt network ip forward package drop problem.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:   Mon, 12 Apr 2021 22:55:22 +0800
From:   Jax Jiang <jax.jiang.007@...il.com>
To:     netdev@...r.kernel.org
Subject: [BUG] Thunderbolt network ip forward package drop problem.

Hi.
I am geek and try to build 10G NIC and thunderbolt network software router.
And I found there have some performance problem on Linux thunderbolt network.

10G NIC -> Linux Kernel -> Thunderbolt Network: downstream speed about 8-9Gbps
Thunderbolt Network -> Linux Kernel -> 10G NIC: upsteam speed only just 1Mbps.

OS: OpenWRT 19.07 (I also tested ubuntu got same result)
Kernel Version: 5.10.27 (I also tested Linux 5.12-rc4 got same result)
These days I try to learn and understand Linux network stack process.
And I found there have two MTU check codes on ip forward process.
1. In net/ipv4/ip_forward.c :
    if (ip_exceeds_mtu(skb, mtu)) {
        IP_INC_STATS(net, IPSTATS_MIB_FRAGFAILS);
        icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED,
              htonl(mtu));
        goto drop;
    }
2. In net/ipv4/ip_output.c:
      if (unlikely(!skb->ignore_df ||
             (IPCB(skb)->frag_max_size &&
              IPCB(skb)->frag_max_size > mtu))) {
        IP_INC_STATS(net, IPSTATS_MIB_FRAGFAILS);
        icmp_send(skb, ICMP_DEST_UNREACH, ICMP_FRAG_NEEDED,
              htonl(mtu));
        kfree_skb(skb);
        return -EMSGSIZE;
    }
Both two codes cause if found package bigger than MTU it will send
icmp "ICMP_FRAG_NEEDED" msg to client.
And I capture package by use tcpdump. And I confirmed there have
"unreachable - need to frag (mtu 1416)" icmp
package back.
I try to force remove both two codes(only remove one of them are not
work). And upsteam speed looks slightly normal. (UP TO 3Gpbs not just
1Mbps).
I capture package on normal 10GNIC: Because of TSO, Kernel received
bigger than mtu package. But on transmit side, it auto split to mtu
size package. On thunderbolt network, looks received TSO large package
not automatic split before ip forward.

I found where package drop on thunderbolt network. But I still don't
know why cause that. And how to solve it.

Could anyone help me?


Jax Jiang