linux-kernel - Re: SMP Affinity Issue with IRQ#

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Date:	Fri, 06 Jun 2008 18:21:20 -0600
From:	Robert Hancock <hancockr@...w.ca>
To:	Khaled Al-Hamwi <khaled.linux@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: SMP Affinity Issue with IRQ#

Khaled Al-Hamwi wrote:
> Hi list,
> 
> I am trying to evaluate the performance of a Linux box when working as
> IP forwarder.
> I have a hardware traffic generator IXIA T400 on one end.
> On the other end, I have a Linux box with kernel 2.6.16 and quad-core xeon CPU.
> 
> These two systems are connected through two crossover Gigabit cables and NICs.
> The traffic is sent from IXIA through one NIC to the forwarding machine
> and then it is forwarded back through the other NIC to IXIA.
> 
> I have two issues related to SMP affinity.
> The first one is that when I set the SMP affinity through
> /proc/irq/<IRQ#>/smp_affinity, it changes dynamically.
> Is there any load balancing in the system that changes the affinity
> after some time or after some packets are received?
> Is there a way to set it permanently?

Likely the irqbalance daemon is doing it. You should be able to stop it. 
(Or you have the kernel IRQ balancing config option enabled, which you 
likely shouldn't.)

> 
> The second issue is that changing the SMP affinity results in
> different delay and throughput measurements.
> I am assigning each NIC to a different CPU.
> If I used a different assignment but still each NIC is assigned to a
> different CPU, I am getting different performance results.
> I would expect that changing the assignment should yield the same
> performance results.
> The CPUs are identical and should have similar performance.
> Here, I have two examples of two different assignments:
> Example 1:
> /proc/irq/16/smp_affinity (eth0)  ->   CPU#1
> /proc/irq/20/smp_affinity (eth1)  ->   CPU#2
> Example 2:
> /proc/irq/16/smp_affinity (eth0)  ->   CPU#3
> /proc/irq/20/smp_affinity (eth1)  ->   CPU#2
> Which one of these two configurations can be used as a reference for
> performance evaluation?
> 
> Any ideas??

I believe the current Intel quad core CPUs are really two dual-core CPU 
dies in a common package. Splitting the network interrupts across the 
two physical CPU chips will likely give you worse performance.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/