linux-kernel - IRQ balancing on a router

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <20081003132117.GM16624@fi.muni.cz>
Date:	Fri, 3 Oct 2008 15:21:17 +0200
From:	Jan Kasprzak <kas@...muni.cz>
To:	arjan@...ux.intel.com
Cc:	linux-kernel@...r.kernel.org
Subject: IRQ balancing on a router

	Hello,

I have a dual-CPU router/firewall with five gigabit NICs. Recently I have
found that irqbalance (0.55 from Fedora 9/x86_64) gives a suboptimal
IRQ to CPU mapping on this box:

	During traffic spikes, it assings two NICs to one CPU, and the
other three to the second CPU. However, this does not account for
the fact that packets coming from the uplink interface are way more
expensive to handle than the rest of the traffic: most iptables rules
apply to the packets received from the uplink interface. The result is
that the CPU which receives IRQs for the uplink interface
is 100 % busy (softirq mostly), while the other one is 90% idle.

	Setting the IRQ mapping by hand (uplink to one CPU, all the other
NICs to the other CPU) makes a well balanced system (both CPUs 30-60 % busy).
I am not sure whether my configuration is too special, but it might be
worth trying to make irqbalance daemon cope also with this usage pattern.

	Another problem is that with one CPU 100 % busy in the kernel
the system latency of user-space programs is _way_ too high. For example,
MRTG graphs from my router used to have blank stripes (i.e.  snmpd
has failed to respond in time). Also the shell response time was bad,
even though I was logged in using SSH over the interface, which at that
time had its IRQ routed to the other CPU.

	With the same network load and manual IRQ to CPU assignment,
MRTG works well and both snmpd and shell response time is good.

-Yenya

-- 
| Jan "Yenya" Kasprzak  <kas at {fi.muni.cz - work | yenya.net - private}> |
| GPG: ID 1024/D3498839      Fingerprint 0D99A7FB206605D7 8B35FCDE05B18A5E |
| http://www.fi.muni.cz/~kas/    Journal: http://www.fi.muni.cz/~kas/blog/ |
>>  If you find yourself arguing with Alan Cox, you’re _probably_ wrong.  <<
>>     --James Morris in "How and Why You Should Become a Kernel Hacker"  <<
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/