netdev - Re: mlx5 broken affinity

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <d65d6156-07da-14dc-88ff-fcb5c0ef5f83@grimberg.me>
Date:   Wed, 1 Nov 2017 19:21:07 +0200
From:   Sagi Grimberg <sagi@...mberg.me>
To:     Jes Sorensen <jsorensen@...com>
Cc:     Networking <netdev@...r.kernel.org>, leonro@...lanox.com,
        Saeed Mahameed <saeedm@...lanox.com>,
        Kernel Team <kernel-team@...com>,
        Christoph Hellwig <hch@....de>
Subject: Re: mlx5 broken affinity

> Hi,

Hi Jes,

> The below patch seems to have broken PCI IRQ affinity assignments for mlx5.

I wouldn't call it breaking IRQ affinity assignments. It just makes
them automatic.

> Prior to this patch I could echo a value to /proc/irq/<x>/smp_affinity
> and it would get assigned. With this patch applied I get -EIO

Adding Christoph,

Ideally the affinity assignments would be better left alone, but
I wasn't aware that they are now immutable. Christoph?

> The actual affinity assignments seem to have changed too, but I assume
> this is a result of the generic allocation?

The affinity assignments should be giving you perfect linear assignment
of the rx/tx rings completion vectors to CPU cores:
first  -> 0
second -> 1
third  -> 2
...

Before, mlx5 spread affinity starting from the local numa node as it
relied on that when constructing RSS indirection table only to the local
numa node (which as a side effect hurt applications running on the far
node as the traffic was guaranteed to arrive rx rings located in the
"wrong" node).

Now the RSS indirection table is linear across both numa nodes just like
the irq affinity settings. Another thing that was improved was the
pre/post vectors which blacklisted any non rx/tx completion vectors from
the affinity assignments (like port events completion vectors from
example).

> With this applied I get:
> [root@...tbox /proc/irq]# cat 50/smp_affinity
> 000000,00000010
> [root@...tbox /proc/irq]# cat 100/smp_affinity
> 100000,00000000
> 
> Without I get:
> [root@...tbox /proc/irq]# cat 50/smp_affinity
> 000000,00000200
> [root@...tbox /proc/irq]# cat 100/smp_affinity
> 000100,00000000
> 
> I am not wildly familiar with the affinity assignment code. Is there
> something obvious I am missing here?

The affinity assignment is indeed changed, should be better though.
If you do not see a linear (diagonal) affinity assignment then
its a bug that needs to be fixed (probably in msi.c code).