| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Sun, 15 Jun 2008 23:24:43 +0300
From: "Denys Fedoryshchenko" <denys@...p.net.lb>
To: netdev@...r.kernel.org
Subject: NAPI, rx_no_buffer_count, e1000, r8169 and other actors
Hi
Since i am using PC routers for my network, and i reach significant numbers
(for me significant) i start noticing minor problems. So all this talk about
networking performance in my case.
For example.
Sun server, AMD based (two CPU - AMD Opteron(tm) Processor 248).
e1000 connected over PCI-X ([ 4.919249] e1000: 0000:01:01.0: e1000_probe:
(PCI-X:100MHz:64-bit) 00:14:4f:20:89:f4)
All traffic processed over eth0, 5 VLAN, 1 second average around 110-200Mbps
of traffic. Host running also conntrack (max 1000000 entries, when packetloss
happen - around 256k entries). Around 1300 routes (FIB_TRIE) running. What is
worrying me, that ok, i win time by increasing rx descriptors from 256 to
4096, but how much time i win? if it "cracks" on 100 Mbps RX, it means by
interpolating descriptors increase from 256 to 4096 (4 times), i cannot
process more than 400Mbps RX?
The CPU is not so busy after all... maybe there is a way to change some
parameter to force NAPI poll interface more often?
I tried nice, changing realtime priority to FIFO, changing kernel to
preemptible... no luck, except increasing descriptors.
Router-Dora ~ # mpstat -P ALL 1
Linux 2.6.26-rc6-git2-build-0029 (Router-Dora) 06/15/08
22:51:02 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
22:51:03 all 1.00 0.00 0.00 0.00 2.50 29.00 0.00
67.50 12927.00
22:51:03 0 2.00 0.00 0.00 0.00 4.00 59.00 0.00
35.00 11935.00
22:51:03 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00
100.00 993.00
22:51:03 2 0.00 0.00 0.00 0.00 0.00 0.00 0.00
0.00 0.00
PID PPID USER STAT VSZ %MEM %CPU COMMAND
1544 1 root S 5824 0.2 0.0 /usr/sbin/snmpd -c /config/snmpd.conf
1530 1 squid S 2880 0.1 0.0 /usr/sbin/ripd -d
1524 1 squid S 2740 0.1 0.0 /usr/sbin/zebra -d
1 0 root S 2384 0.1 0.0 /bin/sh /init
1576 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty1
1577 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty2
1581 1115 root S 2384 0.1 0.0 /sbin/getty 38400 tty3
I have another host running, Core 2 Duo, e1000e+3 x e100, also conntrack, same
kernel configuration and similar amount of traffic, higher load (ifb + plenty
of shapers running) - almost no errors on default settings.
Linux 2.6.26-rc6-git2-build-0029 (Kup) 06/16/08
07:00:27 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
07:00:28 all 0.00 0.00 0.50 0.00 4.00 31.50 0.00
64.00 32835.00
07:00:29 all 0.00 0.00 0.50 0.00 2.50 29.00 0.00
68.00 33164.36
Third host r8169 (PCI! This is important, seems i am running out of PCI
capacity), 400Mbit/s rx+tx summary load, e1000e interface also - around
200Mbps load. What is worrying me - interrupts rate, it seems generated by
realtek card... is there any way to drop it down?
Also some packetloss, around 0.0005% (i prefer to have clear zero :-)) ). No
nat, no shapers, same kernel configuration.
17:36:51 CPU %user %nice %sys %iowait %irq %soft %steal
%idle intr/s
17:36:52 all 0.50 0.00 0.50 0.00 1.49 32.34 0.00
65.17 88993.07
17:36:53 all 0.00 0.00 0.50 0.00 0.50 32.00 0.00
67.00 88655.00
17:36:54 all 0.00 0.00 0.50 0.00 1.49 31.84 0.00
66.17 89484.00
MegaRouter-KARAM ~ # cat /proc/interrupts ;sleep 10;cat /proc/interrupts
CPU0 CPU1
0: 806263699 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge i8042
9: 0 0 IO-APIC-fasteoi acpi
12: 5 0 IO-APIC-edge i8042
16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7
19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
21: 1191830952 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0
23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
217: 3 1584682152 PCI-MSI-edge eth1
NMI: 806263639 806263443 Non-maskable interrupts
LOC: 0 806263442 Local timer interrupts
RES: 99130 71199 Rescheduling interrupts
CAL: 62651 3871 function call interrupts
TLB: 239 187 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0
CPU0 CPU1
0: 806273702 0 IO-APIC-edge timer
1: 2 0 IO-APIC-edge i8042
9: 0 0 IO-APIC-fasteoi acpi
12: 5 0 IO-APIC-edge i8042
16: 0 0 IO-APIC-fasteoi uhci_hcd:usb3
18: 0 0 IO-APIC-fasteoi ehci_hcd:usb1, uhci_hcd:usb7
19: 0 0 IO-APIC-fasteoi uhci_hcd:usb6
21: 1192549139 0 IO-APIC-fasteoi uhci_hcd:usb4, eth0
23: 1245 0 IO-APIC-fasteoi ehci_hcd:usb2, uhci_hcd:usb5
217: 3 1584840861 PCI-MSI-edge eth1
NMI: 806273642 806273446 Non-maskable interrupts
LOC: 0 806273445 Local timer interrupts
RES: 99130 71199 Rescheduling interrupts
CAL: 62653 3871 function call interrupts
TLB: 239 187 TLB shootdowns
TRM: 0 0 Thermal event interrupts
SPU: 0 0 Spurious interrupts
ERR: 0
MIS: 0
--
Denys Fedoryshchenko
Technical Manager
Virtual ISP S.A.L.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists