linux-kernel - Performance regression in ip_set

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-Id: <C0829B10-EAA6-4809-874E-E1E9C05A8D84@automattic.com>
Date: Fri, 5 Jan 2024 23:45:27 +0100
From: Ale Crismani <ale.crismani@...omattic.com>
To: linux-kernel@...r.kernel.org
Subject: Performance regression in ip_set_swap on 6.1.69

Dear all,

When upgrading some of our Debian hosts that compose a Kubernetes cluster we found a regression in ip_set_swap on 6.1.69. Calls to ip_set_swap now take roughly 15ms, while they used to take just tens of microseconds before.

The issue is very visible for use, since we use kube-router as our Kubernetes networking interface, and it uses ipset swap all the time to populate sets that enforce firewall policies between containers.

We tracked the issue down with strace, and then took stats with bpftrace running:
---
kfunc:ip_set:ip_set_swap {
        @start[tid] = nsecs;
}

kretfunc:ip_set:ip_set_swap {
        if (@start[tid]) {
                @srlat = hist((nsecs - @start[tid])/1000);
                delete(@start[tid]);
        }
}

interval:s:20 {
        printf("ip_set_swap() latency, milliseconds:\n");
---

On 6.1.69 results look like:

ip_set_swap() latency, milliseconds:
[8K, 16K)           1848 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[16K, 32K)         1017 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32K, 64K)         38     |@

while on 6.1.67:
ip_set_swap() latency, milliseconds:

[0]                  166   |@
[1]                  378   |@@
[2, 4)              762   |@@@@@
[4, 8)              1624 |@@@@@@@@@@@
[8, 16)            3493 |@@@@@@@@@@@@@@@@@@@@@@@@
[16, 32)          7308 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[32, 64)          6412 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
[64, 128)              1 |

We tried compiling commits between 6.1.67 and 6.1.69 and it seems the performance regression was introduced by 875ee3a, ip_set_swap is fast on 602505 that precedes it, and slow on it.

First time I post here, hope the format is appropriate, and thanks for any help with this! Also, if possible, I'd appreciate if any reply could CC me, as I am not subscribed.

Alessandro Crismani