linux-kernel - Why the number of /proc/interrupts doesn't change when nic is under heavy workload?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAEc1PS0doeaH59621ia7n2GT=WgRSsCit_EzGrCV93QH95XZqw@mail.gmail.com>
Date:	Sun, 15 Jan 2012 15:53:20 -0500
From:	Yuehai Xu <yuehaixu@...il.com>
To:	netdev@...r.kernel.org
Cc:	linux-kernel@...r.kernel.org, yhxu@...ne.edu
Subject: Why the number of /proc/interrupts doesn't change when nic is under
 heavy workload?

Hi All,

My nic of server is Intel Corporation 80003ES2LAN Gigabit Ethernet
Controller, the driver is e1000e, and my Linux version is 3.1.4. I
have a Memcached server running on this 8 core box, the weird thing is
that when my server is under heavy workload, the number of
/proc/interrupts doesn't change at all. Below are some details:
=======
cat /proc/interrupts | grep eth0
68:     330887     330861     331432     330544     330346     330227
   330830     330575   PCI-MSI-edge      eth0
=======
cat /proc/irq/68/smp_affinity
ff

I know when network is under heavy load, NAPI will disable nic
interrupt and poll ring buffer in nic. My question is, when is nic
interrupt enabled again? It seems that it will never be enabled if the
heavy workload doesn't stop, simply because the number showed by
/proc/interrupts doesn't change at all. In my case, one of core is
saturated by ksoftirqd, because lots of softirqs are pending to that
core. I just want to distribute these softirqs to other cores. Even
RPS is enabled, that core is still occupied by ksoftirq, nearly 100%.

I dive into the codes and find these statements:
__napi_schedule ==>
   local_irq_save(flags);
   ____napi_schedule(&__get_cpu_var(softnet_data), n);
   local_irq_restore(flags);

here "local_irq_save" actually invokes "cli" which disable interrupt
for the local core, is this the one that used in NAPI to disable nic
interrupt? Personally I don't think it is because it just disables
local cpu.

I also find "enable_irq/disable_irq/e1000_irq_enable/e1000_irq_disable"
under drivers/net/e1000e, are these used in NAPI to disable nic
interrupt, but I fail to get any clue that they are used in the code
path of NAPI?

My current situation is that, almost 60% of time of other 7 cores are
idle, while only one core which is occupied by ksoftirq is 100% busy.

Thanks very much!
Yuehai
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/