netdev - Empty return from bond_eth

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CABWYdi1a7MKxM8XX9_1zRkp_h8AHGWT_GQTwAbJdz0iKEfrsEA@mail.gmail.com>
Date:   Tue, 18 Jan 2022 16:03:15 -0800
From:   Ivan Babrou <ivan@...udflare.com>
To:     Jussi Maki <joamaki@...il.com>
Cc:     Daniel Borkmann <daniel@...earbox.net>,
        Jay Vosburgh <j.vosburgh@...il.com>,
        Veaceslav Falico <vfalico@...il.com>,
        Andy Gospodarek <andy@...yhouse.net>,
        kernel-team <kernel-team@...udflare.com>,
        Linux Kernel Network Developers <netdev@...r.kernel.org>
Subject: Empty return from bond_eth_hash in 5.15

Hello,

We noticed an issue on Linux 5.15 where it sends packets from a single
connection via different bond members. Some of our machines are
connected to multiple TORs, which means that BGP can attract the same
connection to different servers, depending on which cable you
traverse.

On Linux 5.10 I can see bond_xmit_hash always return the same hash for
the same connection:

$ sudo bpftrace --include linux/ip.h -e 'kprobe:bond_xmit_hash {
@skbs[pid] = arg1 } kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid];
if ($skb_ptr) { $skb = (struct sk_buff *) $skb_ptr; $ipheader =
((struct iphdr *) ($skb->head + $skb->network_header)); printf("%s
%x\n", ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
x.y.z.205
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
x.y.z.205 9f24591
... many more of these

On Linux 5.10 I get fewer lines, mostly zeros for hash and one actual hash:

$ sudo bpftrace -e 'kprobe:bond_xmit_hash { @skbs[pid] = arg1 }
kretprobe:bond_xmit_hash { $skb_ptr = @skbs[pid]; if ($skb_ptr) { $skb
= (struct sk_buff *) $skb_ptr; $ipheader = ((struct iphdr *)
($skb->head + $skb->network_header)); printf("%s %x\n",
ntop($ipheader->daddr), retval); } }' | fgrep --line-buffered
x.y.z.205
x.y.z.205 0
x.y.z.205 0
x.y.z.205 215fec1b

As I mentioned above, this ends up breaking connections for us, which
is unfortunate.

I suspect that "net, bonding: Refactor bond_xmit_hash for use with
xdp_buff" commit a815bde56b1 has something to do with this. I don't
think we use XDP on the machines I tested.