Message-Id: <C0407638-FD77-4D21-A262-A05AD7428012@bejarano.io>
Date: Thu, 22 May 2025 19:19:52 +0200
From: Ricard Bejarano <ricard@...arano.io>
To: netdev@...r.kernel.org
Cc: michael.jamet@...el.com,
 mika.westerberg@...ux.intel.com,
 YehezkelShB@...il.com,
 andrew+netdev@...n.ch,
 davem@...emloft.net,
 edumazet@...gle.com,
 kuba@...nel.org,
 pabeni@...hat.com
Subject: Poor thunderbolt-net interface performance when bridged

Hi all,

Please excuse me if this is not the right place or way to report this, in which case I'd appreciate a pointer to the proper forum. I've CC'd every one who showed up in get_maintainer.pl.

I'm investigating a performance issue in the bridging of traffic coming in via a Thunderbolt 3/4 (thunderbolt-net driver) network interface. I don't think this is tracked from what I could find online.

Summary:
When a thunderbolt-net interface is slave to a bridge, traffic in the "other slave interfaces -> bridge -> thunderbolt-net interface" direction approximates maximum line bandwidth (~9Gbps in Thunderbolt 3, ~16Gbps in Thunderbolt 4); but traffic in the opposite "thunderbolt -> bridge -> other" direction drops to ~5Mbps (in my testing). More details below.

I need some pointers on how to proceed.

Thanks,
RB

-- 

## 1. Setup

Three hosts:
- `red`:
  - Board: Intel NUC8i5BEH2
  - CPU: 1x 4-core x86-64 (Intel Core i5-8259U)
  - RAM: 2x 8GB DDR4-2666 SODIMM CL19 (Crucial)
  - Disk: 1x 120GB SATA SSD (Crucial BX500)
  - Relevant interfaces:
    - `br0` (`bridge` driver, `10.0.0.1/24` address)
    - `tb0` (`thunderbolt-net` driver): maps to the board's Thunderbolt port, slave of `br0`
- `blue`:
  - Board: Intel NUC8i5BEH2
  - CPU: 1x 4-core x86-64 (Intel Core i5-8259U)
  - RAM: 2x 8GB DDR4-2666 SODIMM CL19 (Crucial)
  - Disk: 1x 120GB SATA SSD (Crucial BX500)
  - Relevant interfaces:
    - `br0` (`bridge` driver, `10.0.0.2/24` address)
    - `tb0` (`thunderbolt-net` driver): maps to the board's Thunderbolt port, slave of `br0`
    - `eno1` (`e1000e` driver): maps to the board's Ethernet port, slave of `br0`
- `purple`:
  - Board: Intel NUC8i5BEHS
  - CPU: 1x 4-core x86-64 (Intel Core i5-8260U)
  - RAM: 2x 8GB DDR4-2666 SODIMM CL19 (Crucial)
  - Disk: 1x 240GB M.2 SATA SSD (WD Green)
  - Relevant interfaces:
    - `br0` (`bridge` driver, `10.0.0.3/24` address)
    - `eno1` (`e1000e` driver): maps to the board's Ethernet port, slave of `br0`

Connected with two cables:
- Amazon Basics Thunderbolt 3 & 4 cable, connecting `red` (`tb0`) to `blue` (`tb0`).
- Monoprice SlimRun Cat6A Ethernet cable, connecting `blue` (`eno1`) to `purple` (`eno1`).

All three running Linux 6.14.7 (built from source) on Ubuntu Server 24.04.2 LTS, running iperf 2.1.9 servers.
See "4. References" section for details.

## 2. The problem

As seen in [4.6.3b], traffic going in the `purple:br0 -> purple:eno1 -> blue:eno1 -> blue:br0 -> blue:tb0 -> red:tb0 -> red:br0` direction approaches line speed (~1Gbps).
However, per [4.6.3a], traffic going in the opposite `red:br0 -> red:tb0 -> blue:tb0 -> blue:br0 -> blue:eno1 -> purple:eno1 -> purple:br0` direction is several orders of magnitude slower (~5Mbps).

This is abnormal, given [4.6.1] sets the bidirectional Thunderbolt line speed at ~9Gbps and [4.6.2] sets the bidirectional Ethernet line speed at ~1Gbps.

Per the above, we can safely assume that the problem is localized at `blue`, specifically in how `blue` bridges traffic out of `tb0` and into `eno1`.

From prior undocumented anecdata, we know this also happens in Thunderbolt-to-Thunderbolt bridged traffic, which hints at a problem in how traffic goes out of `tb0` and into `br0`, not with how traffic goes out of `br0` and into `eno1`.
This is further consolidated by the fact that Ethernet-to-Ethernet bridging is known to approach line speed in both directions (or otherwise the Internet would be way slower, I suppose).

And finally, hosts are only assuming an IP address at their respective `br0` interfaces, and [4.6.1] shows line speed performance in the `red:br0 -> red:tb0 -> blue:tb0 -> blue:br0` direction (and reverse).
Meaning, we can reduce the scope further to how traffic goes out of `tb0` and into some other slave of `br0`, but not `br0` itself.

## 3. The solution

    ¯\_(;.;)_/¯

## 4. References

### 4.1. `uname -a`
#### 4.1.1. `red`
```shell
# red
$ uname -a
Linux red 6.14.7 #1 SMP PREEMPT_DYNAMIC Mon May 19 13:38:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
```
#### 4.1.2. `blue`
```shell
# blue
$ uname -a
Linux blue 6.14.7 #1 SMP PREEMPT_DYNAMIC Mon May 19 15:01:20 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
```
#### 4.1.3. `purple`
```shell
# purple
$ uname -a
Linux purple 6.14.7 #1 SMP PREEMPT_DYNAMIC Tue May 20 09:04:42 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
```

### 4.2. `ip a`
#### 4.2.1. `red`
```shell
# red
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 94:c6:91:a3:f5:1a brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 04:d3:b0:0f:e6:cd brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.201/23 metric 600 brd 192.168.11.255 scope global dynamic wlp0s20f3
       valid_lft 163sec preferred_lft 163sec
    inet6 fd9f:7271:415f:d845:6d3:b0ff:fe0f:e6cd/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 1724sec preferred_lft 1724sec
    inet6 fe80::6d3:b0ff:fe0f:e6cd/64 scope link
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether ce:42:52:00:a0:5b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.1/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
7: tb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 02:5f:d6:57:71:93 brd ff:ff:ff:ff:ff:ff
```
#### 4.2.2. `blue`
```shell
# blue
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 1c:69:7a:00:22:99 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
5: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d0:c6:37:09:01:5a brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.200/23 metric 600 brd 192.168.11.255 scope global dynamic wlp0s20f3
       valid_lft 247sec preferred_lft 247sec
    inet6 fd9f:7271:415f:d845:d2c6:37ff:fe09:15a/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 1651sec preferred_lft 1651sec
    inet6 fe80::d2c6:37ff:fe09:15a/64 scope link
       valid_lft forever preferred_lft forever
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 3a:4d:83:e0:ab:3b brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.2/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
7: tb0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 02:70:19:dc:92:96 brd ff:ff:ff:ff:ff:ff
```
#### 4.2.3. `purple`
```shell
# purple
$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 1c:69:7a:60:d8:69 brd ff:ff:ff:ff:ff:ff
    altname enp0s31f6
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 94:e6:f7:7c:2d:fb brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.199/23 metric 600 brd 192.168.11.255 scope global dynamic wlp0s20f3
       valid_lft 165sec preferred_lft 165sec
    inet6 fd9f:7271:415f:d845:96e6:f7ff:fe7c:2dfb/64 scope global dynamic mngtmpaddr noprefixroute
       valid_lft 1640sec preferred_lft 1640sec
    inet6 fe80::96e6:f7ff:fe7c:2dfb/64 scope link
       valid_lft forever preferred_lft forever
4: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 1a:45:1d:c0:46:02 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 brd 10.0.0.255 scope global br0
       valid_lft forever preferred_lft forever
```

### 4.3. `ethtool -i br0`
#### 4.3.1. `red`
```shell
# red
$ ethtool -i br0
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
```
#### 4.3.2. `blue`
```shell
# blue
$ ethtool -i br0
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
```
#### 4.3.3. `purple`
```shell
# purple
$ ethtool -i br0
driver: bridge
version: 2.3
firmware-version: N/A
expansion-rom-version:
bus-info: N/A
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
```

### 4.4. `ethtool -i tb0`
#### 4.4.1. `red`
```shell
# red
$ ethtool -i tb0
driver: thunderbolt-net
version: 6.14.7
firmware-version:
expansion-rom-version:
bus-info: 0-1.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
```
#### 4.4.2. `blue`
```shell
# blue
$ ethtool -i tb0
driver: thunderbolt-net
version: 6.14.7
firmware-version:
expansion-rom-version:
bus-info: 0-1.0
supports-statistics: no
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no
```

### 4.5. `ethtool -i eno1`
#### 4.4.1. `blue`
```shell
# blue
$ ethtool -i eno1
driver: e1000e
version: 6.14.7
firmware-version: 0.4-4
expansion-rom-version:
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
```
#### 4.4.2. `purple`
```shell
# purple
$ ethtool -i eno1
driver: e1000e
version: 6.14.7
firmware-version: 0.4-4
expansion-rom-version:
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes
```

### 4.6. `iperf` tests
#### 4.6.1a. `red` to `blue`
```shell
# red
$ iperf -c 10.0.0.2
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 38902 connected with 10.0.0.2 port 5001 (icwnd/mss/irtt=14/1448/538)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0076 sec  11.0 GBytes  9.40 Gbits/sec
```
#### 4.6.1b. `blue` to `red`
```shell
# blue
$ iperf -c 10.0.0.1
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.2 port 49660 connected with 10.0.0.1 port 5001 (icwnd/mss/irtt=14/1448/464)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0079 sec  10.8 GBytes  9.26 Gbits/sec
```
#### 4.6.2a. `purple` to `blue`
```shell
# purple
$ iperf -c 10.0.0.2
------------------------------------------------------------
Client connecting to 10.0.0.2, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.3 port 56150 connected with 10.0.0.2 port 5001 (icwnd/mss/irtt=14/1448/580)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0358 sec  1.09 GBytes   933 Mbits/sec
```
#### 4.6.2b. `blue` to `purple`
```shell
# blue
$ iperf -c 10.0.0.3
------------------------------------------------------------
Client connecting to 10.0.0.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.2 port 37106 connected with 10.0.0.3 port 5001 (icwnd/mss/irtt=14/1448/958)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0239 sec  1.09 GBytes   934 Mbits/sec
```
#### 4.6.3a. `red` to `purple`
```shell
# red
$ iperf -c 10.0.0.3
------------------------------------------------------------
Client connecting to 10.0.0.3, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.1 port 38260 connected with 10.0.0.3 port 5001 (icwnd/mss/irtt=14/1448/1578)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.2234 sec  5.88 MBytes  4.82 Mbits/sec
```
#### 4.6.3b. `purple` to `red`
```shell
# purple
$ iperf -c 10.0.0.1
------------------------------------------------------------
Client connecting to 10.0.0.1, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  1] local 10.0.0.3 port 48392 connected with 10.0.0.1 port 5001 (icwnd/mss/irtt=14/1448/1243)
[ ID] Interval       Transfer     Bandwidth
[  1] 0.0000-10.0233 sec  1.09 GBytes   932 Mbits/sec
```

Powered by blists - more mailing lists