netdev - Re: This is the fourth time I've tried to find what led to the regression of outgoing network speed and each time I find the merge commit 8c94ccc7cd691472461448f98e2372c75849406c

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <3ea25443-1275-4c67-90e0-b637212d32b5@leemhuis.info>
Date: Mon, 26 Feb 2024 06:45:08 +0100
From: "Linux regression tracking (Thorsten Leemhuis)"
 <regressions@...mhuis.info>
To: Mathias Nyman <mathias.nyman@...ux.intel.com>,
 Thomas Gleixner <tglx@...utronix.de>
Cc: "Christian A. Ehrhardt" <lk@...e.de>, niklas.neronin@...ux.intel.com,
 Linux List Kernel Mailing <linux-kernel@...r.kernel.org>,
 Greg KH <gregkh@...uxfoundation.org>, linux-usb@...r.kernel.org,
 linux-x86_64@...r.kernel.org, netdev@...r.kernel.org,
 Linux kernel regressions list <regressions@...ts.linux.dev>,
 Randy Dunlap <rdunlap@...radead.org>,
 Mikhail Gavrilov <mikhail.v.gavrilov@...il.com>
Subject: Re: This is the fourth time I've tried to find what led to the
 regression of outgoing network speed and each time I find the merge commit
 8c94ccc7cd691472461448f98e2372c75849406c

On 21.02.24 14:44, Mathias Nyman wrote:
> On 21.2.2024 1.43, Randy Dunlap wrote:
>> On 2/20/24 15:41, Randy Dunlap wrote:
>>> {+ tglx]
>>> On 2/20/24 15:19, Mikhail Gavrilov wrote:
>>>> On Mon, Feb 19, 2024 at 2:41 PM Mikhail Gavrilov
>>>> <mikhail.v.gavrilov@...il.com> wrote:
>>>> I spotted network performance regression and it turned out, this was
>>>> due to the network card getting other interrupt. It is a side effect
>>>> of commit 57e153dfd0e7a080373fe5853c5609443d97fa5a.
>>> That's a merge commit (AFAIK, maybe not so much). The commit in
>>> mainline is:
>>>
>>> commit f977f4c9301c
>>> Author: Niklas Neronin <niklas.neronin@...ux.intel.com>
>>> Date:   Fri Dec 1 17:06:40 2023 +0200
>>>
>>>      xhci: add handler for only one interrupt line
>>>
>>>> Installing irqbalance daemon did not help. Maybe someone experienced
>>>> such a problem?
>>>
>>> Thomas, would you look at this, please?
>>>
>>> A network device and xhci (USB) driver are now sharing interrupts.
>>> This causes a large performance decrease for the networking device.
> 
> Short recap:

Thx for that. As the 6.8 release is merely two or three weeks away while
a fix is nowhere near in sight yet (afaics!) I start to wonder if we
should consider a revert here and try reapplying the culprit in a later
cycle when this problem is fixed.

Mathias, would that be an option? Or is there still hope that we see a
fix for this regression before the release of 6.8?

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
If I did something stupid, please tell me, as explained on that page.

#regzbot poke

> xhci (USB) and network device didn't share interrupts, or even interrupt
> the
> same CPU in either good or bad case.
> 
> A change in how many interrupts xhci driver requests changed which CPU
> the network device interrupts.
> 
> In the bad case Mikhail Gavrilovs network device was interrupting CPU0
> together with:
> - IR-IO-APIC    2-edge      timer
> - IR-PCI-MSIX-0000:07:00.0    1-edge      nvme1q1
> 
> In the good case network device was interrupting CPU27 together with:
> - IR-PCI-MSIX-0000:04:00.0   27-edge      nvme0q27
> - IR-PCI-MSIX-0000:07:00.0   28-edge      nvme1q28
> 
> Manually moving network device irq 87 from CPU0 to CPU23 helped.
> (echo 800000 > /proc/irq/87/smp_affinity)
> 
> Thanks
> -Mathias
>