~~ -*-text-*- ------------------------------------------------------- Bisecting IRQ change: What change reduced the IRQs ------------------------------------------------------- Jesper Dangaard Brouer (jdb@comx.dk) ------------------------------------------------------- $LastChangedRevision: 786 $ $Date: 2008-11-20 20:44:51 +0100 (Thu, 20 Nov 2008) $ ------------------------------------------------------- git clone ~~~~~~~~~ +--------- cd /var/kernels/git/davem git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect-irqs +--------- Description / Reason to find ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I get really good performance (900-1200kpps) compared to 2.6.28 (git net-2.6). The cause of this problem (tracked down together with Robert Olsson) is that on 2.6.28 I have a lot less IRQs available. It seems max 34 IRQs. Due the reduced number of IRQs the NIU driver cannot get enough IRQs to the interfaces, and starts to use "IO-APIC" based IRQs. On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with the usb driver... Think thats must be my performance problem on 2.6.28. Known: Good and bad ~~~~~~~~~~~~~~~~~~~ GOOD: git bisect good v2.6.27 BAD: git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43 [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip HiSTORY: ~~~~~~~~ +-------- cd /var/kernels/git/davem/net-2.6-bisect-irqs/ git bisect start git bisect good v2.6.27 +-------- +-------------- git bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43 Bisecting: 3220 revisions left to test after this [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2 +-------------- CONFIG_LOCALVERSION="-bisect" +------------- cp ../net-2.6-bisect/.config . script make_oldconfig_01 make oldconfig exit #Script done, file is make_oldconfig_01 +------------- +---------------- time make -j6 bzImage modules # #real 9m22.739s #user 16m56.776s #sys 1m4.672s +---------------- Booted kernel: GOOD: irqs and (niu rmmod good) +---------------- git bisect good Bisecting: 1614 revisions left to test after this [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input +---------------- Compiling: +---------------- time make -j6 bzImage modules +---------------- Booted kernel: GOOD: irqs and (niu rmmod good) +----------- git bisect good Bisecting: 807 revisions left to test after this [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen() +----------- Compiling... +---------------- time make -j6 bzImage modules #real 10m1.561s #user 17m23.293s #sys 1m5.744s +---------------- Installing... Booted kernel: +---- dcu-router-ng:~# uname -a Linux dcu-router-ng 2.6.27-bisect #3 SMP PREEMPT Thu Nov 20 12:33:02 CET 2008 i686 GNU/Linux +---- Results: GOOD: irqs and (niu rmmod good) +------ git bisect good Bisecting: 403 revisions left to test after this [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse +------ Compiling... +---------------- time make -j6 bzImage modules #real 10m9.371s #user 17m21.781s #sys 1m6.052s +---------------- Installing... Booting ... +------- dcu-router-ng:~# uname -a Linux dcu-router-ng 2.6.27-bisect #4 SMP PREEMPT Thu Nov 20 12:50:39 CET 2008 i686 GNU/Linux +------- Results: GOOD: irqs and (niu rmmod good) +------- git-bisect good Bisecting: 223 revisions left to test after this [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files +------- +---------------- time make -j6 bzImage modules +---------------- Booting ... +-------- Linux dcu-router-ng 2.6.27-bisect #5 SMP PREEMPT Thu Nov 20 13:58:34 CET 2008 i686 GNU/Linux +-------- Results: BAD: irqs and (niu rmmod also BAD) +------- cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 125 0 0 0 IO-APIC-edge timer 1: 0 0 1 1 IO-APIC-edge i8042 3: 2 1 2 2 IO-APIC-edge serial 8: 0 2 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 1 2 1 0 IO-APIC-edge i8042 16: 103 108 108 112 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6, eth0 17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4, eth3 20: 0 0 0 0 PCI-MSI-edge eth2 21: 0 0 0 0 PCI-MSI-edge eth2 22: 24 23 23 23 IO-APIC-fasteoi uhci_hcd:usb5, eth2 23: 0 0 0 0 PCI-MSI-edge eth2 24: 0 0 0 0 PCI-MSI-edge eth2 25: 0 0 0 0 PCI-MSI-edge eth2 26: 0 0 0 0 PCI-MSI-edge eth2 27: 0 0 0 0 PCI-MSI-edge eth2 28: 0 0 0 0 PCI-MSI-edge eth2 29: 0 0 0 0 PCI-MSI-edge eth2 30: 0 0 0 0 PCI-MSI-edge eth2 31: 0 0 0 0 PCI-MSI-edge eth2 32: 0 0 0 0 PCI-MSI-edge eth2 34: 271 268 268 264 PCI-MSI-edge cciss0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 3301 2970 2594 2389 Local timer interrupts RES: 28 560 6 13 Rescheduling interrupts CAL: 50 104 99 62 Function call interrupts TLB: 241 224 287 279 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 MIS: 0 +------- OUTPUT "rmmod niu" (gives segfault) and "dmesg" +------- ------------[ cut here ]------------ kernel BUG at drivers/pci/msi.c:632! invalid opcode: 0000 [#1] PREEMPT SMP Modules linked in: ehci_hcd bnx2 uhci_hcd zlib_inflate serio_raw hpilo niu(-) Pid: 3036, comm: rmmod Not tainted (2.6.27-bisect #5) ProLiant DL380 G5 EIP: 0060:[] EFLAGS: 00010286 CPU: 2 EIP is at msi_free_irqs+0xdc/0xe0 EAX: f6b8f860 EBX: 00000030 ECX: f7156ba8 EDX: c0420500 ESI: f7156800 EDI: f7156ba8 EBP: f6431eb4 ESP: f6431ea8 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process rmmod (pid: 3036, ti=f6430000 task=f70f9b20 task.ti=f6430000) Stack: f7156800 f670c400 f7156800 f6431ebc c021ecb8 f6431ec8 c021ef41 f670c000 f6431edc f809d3f8 f7156800 f80a1ed4 f80a1ed4 f6431ee8 c0219c29 f7156858 f6431ef8 c026b0d4 f7156858 f7156914 f6431f0c c026b197 f80a1ea0 f80a1ed4 Call Trace: [] ? msix_free_all_irqs+0x8/0x10 [] ? pci_disable_msix+0x31/0x40 [] ? niu_pci_remove_one+0x88/0x8a [niu] [] ? pci_device_remove+0x19/0x40 [] ? __device_release_driver+0x54/0x80 [] ? driver_detach+0x97/0xa0 [] ? bus_remove_driver+0x75/0xa0 [] ? driver_unregister+0x39/0x40 [] ? pci_unregister_driver+0x21/0x80 [] ? niu_exit+0xd/0x10 [niu] [] ? sys_delete_module+0x114/0x1d0 [] ? remove_vma+0x3a/0x50 [] ? do_munmap+0x189/0x1e0 [] ? sysenter_do_call+0x12/0x21 [] ? quirk_disable_msi+0x30/0x50 Code: b7 43 08 8b 53 1c c1 e0 04 01 d0 ba 01 00 00 00 83 c0 0c 89 10 3b 7b 14 75 aa 8b 43 1c e8 3d 92 ef ff eb a0 5b 31 c0 5e 5f 5d c3 <0f> 0b eb fe 55 89 e5 e8 18 ff ff ff 5d c3 8d b6 00 00 00 00 55 EIP: [] msi_free_irqs+0xdc/0xe0 SS:ESP 0068:f6431ea8 ---[ end trace f72de2e283920207 ]--- +------- +------ git-bisect bad Bisecting: 89 revisions left to test after this [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function +------ Related config change? (make oldconfig) +------ script make_oldconfig_02 make oldconfig Script done, file is make_oldconfig_02 +------ +------ Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) ? ?Y This enables support for sparse irq, esp for msi/msi-x. the irq number will be bus/dev/fn + 12bit. You may need if you have lots of cards supports msi-x installed. If you don't know what to do here, say Y. +------ Compiling... +---------------- time make -j6 bzImage modules # #real 9m29.556s #user 17m10.396s #sys 1m5.056s +---------------- Booting ... +------- Linux dcu-router-ng 2.6.27-bisect #6 SMP PREEMPT Thu Nov 20 14:25:40 CET 2008 i686 GNU/Linux +------- The output from /proc/interrupts changed, very weird! BUT eth3 does use a "PCI-MSI-edge" interrupt. Guess this is a GOOD state even though it looks weird. Unloading NIU driver also GOOD. +--------- cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0x0: 124 1 0 0 IO-APIC-edge timer 0x1: 1 0 0 1 IO-APIC-edge i8042 0x3: 2 1 2 2 IO-APIC-edge serial 0x8: 1 0 0 1 IO-APIC-edge rtc 0x9: 0 0 0 0 IO-APIC-fasteoi acpi 0xc: 0 1 2 1 IO-APIC-edge i8042 0x10: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6 0x11: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 0x12: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 0x6000fe: 288 289 290 290 PCI-MSI-edge cciss0 0x16: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5 0xb00100: 0 0 0 0 PCI-MSI-edge eth2 0xb000ff: 0 0 0 0 PCI-MSI-edge eth2 0xb000fe: 0 0 0 0 PCI-MSI-edge eth2 0xb000fd: 0 0 0 0 PCI-MSI-edge eth2 0xb000fc: 0 0 0 0 PCI-MSI-edge eth2 0xb000fb: 0 0 0 0 PCI-MSI-edge eth2 0xb000fa: 0 0 0 0 PCI-MSI-edge eth2 0xb000f9: 0 0 0 0 PCI-MSI-edge eth2 0xb000f8: 0 0 0 0 PCI-MSI-edge eth2 0xb000f7: 0 0 0 0 PCI-MSI-edge eth2 0xb000f6: 0 0 0 0 PCI-MSI-edge eth2 0xb000f5: 0 0 0 0 PCI-MSI-edge eth2 0xb000f4: 0 0 0 0 PCI-MSI-edge eth2 0xb01100: 0 0 0 0 PCI-MSI-edge eth3 0xb010ff: 0 0 0 0 PCI-MSI-edge eth3 0xb010fe: 0 0 0 0 PCI-MSI-edge eth3 0xb010fd: 0 0 0 0 PCI-MSI-edge eth3 0xb010fc: 0 0 0 0 PCI-MSI-edge eth3 0xb010fb: 0 0 0 0 PCI-MSI-edge eth3 0xb010fa: 0 0 0 0 PCI-MSI-edge eth3 0xb010f9: 0 0 0 0 PCI-MSI-edge eth3 0xb010f8: 0 0 0 0 PCI-MSI-edge eth3 0xb010f7: 0 0 0 0 PCI-MSI-edge eth3 0xb010f6: 0 0 0 0 PCI-MSI-edge eth3 0x13: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 0x300100: 210 210 210 208 PCI-MSI-edge eth0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 3630 3265 3103 2711 Local timer interrupts RES: 34 226 12 417 Rescheduling interrupts CAL: 89 55 90 78 Function call interrupts TLB: 253 205 311 267 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 MIS: 0 +--------- Guess it a GOOD situation... +------ git-bisect good Bisecting: 44 revisions left to test after this [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI +------ Compiling ... +-------- time make -j6 bzImage modules #real 9m28.062s #user 17m7.492s #sys 1m4.248s +-------- Installing ... Booting ... +------ Linux dcu-router-ng 2.6.27-bisect #7 SMP PREEMPT Thu Nov 20 14:52:45 CET 2008 i686 GNU/Linux +------ Still looks GOOD (/proc/interrupts still looks weird). And rmmod NIU driver GOOD. +------ git-bisect good Bisecting: 22 revisions left to test after this [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4 +------ Compiling ... +-------- time make -j6 bzImage modules #real 0m34.622s #user 0m41.139s #sys 0m5.812s +-------- Install ... Booting ... +----- Linux dcu-router-ng 2.6.27-bisect #8 SMP PREEMPT Thu Nov 20 15:04:11 CET 2008 i686 GNU/Linux +----- Looks GOOD, and /proc/interrupts changed again! Now the interrupts are not i HEX anymore, but in decimal, but still strange/large numbers for MSI. Unloading NIU driver GOOD. +------ cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 124 0 0 0 IO-APIC-edge timer 1: 0 0 1 1 IO-APIC-edge i8042 3: 2 2 1 2 IO-APIC-edge serial 8: 0 0 1 1 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 1 2 1 0 IO-APIC-edge i8042 16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6 17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 6291710: 828 821 828 823 PCI-MSI-edge cciss0 22: 23 24 24 22 IO-APIC-fasteoi uhci_hcd:usb5 11534592: 0 0 0 0 PCI-MSI-edge eth2 11534591: 0 0 0 0 PCI-MSI-edge eth2 11534590: 0 0 0 0 PCI-MSI-edge eth2 11534589: 0 0 0 0 PCI-MSI-edge eth2 11534588: 0 0 0 0 PCI-MSI-edge eth2 11534587: 0 0 0 0 PCI-MSI-edge eth2 11534586: 0 0 0 0 PCI-MSI-edge eth2 11534585: 0 0 0 0 PCI-MSI-edge eth2 11534584: 0 0 0 0 PCI-MSI-edge eth2 11534583: 0 0 0 0 PCI-MSI-edge eth2 11534582: 0 0 0 0 PCI-MSI-edge eth2 11534581: 0 0 0 0 PCI-MSI-edge eth2 11534580: 0 0 0 0 PCI-MSI-edge eth2 11538688: 0 0 0 0 PCI-MSI-edge eth3 11538687: 0 0 0 0 PCI-MSI-edge eth3 11538686: 0 0 0 0 PCI-MSI-edge eth3 11538685: 0 0 0 0 PCI-MSI-edge eth3 11538684: 0 0 0 0 PCI-MSI-edge eth3 11538683: 0 0 0 0 PCI-MSI-edge eth3 11538682: 0 0 0 0 PCI-MSI-edge eth3 11538681: 0 0 0 0 PCI-MSI-edge eth3 11538680: 0 0 0 0 PCI-MSI-edge eth3 11538679: 0 0 0 0 PCI-MSI-edge eth3 11538678: 0 0 0 0 PCI-MSI-edge eth3 19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 3145984: 9993 9994 9987 9993 PCI-MSI-edge eth0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 10075 11448 8001 8787 Local timer interrupts RES: 297 17 349 26 Rescheduling interrupts CAL: 173 189 95 173 Function call interrupts TLB: 299 259 330 345 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 MIS: 0 +------ +------- git-bisect good Bisecting: 11 revisions left to test after this [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal +------- Compiling +-------- time make -j6 bzImage modules #real 9m30.767s #user 17m10.808s #sys 1m6.388s +-------- Installing ... Booting ... +----- Linux dcu-router-ng 2.6.27-bisect #9 SMP PREEMPT Thu Nov 20 15:28:17 CET 2008 i686 GNU/Linux +----- BAD kernel version, max IRQ is 34. And eth3 got assigned a IO-APIC-fasteoi shared with uhci_hcd:usb2. Also BAD unloading of NIU driver. BUG is some where in between: git log 922402f15a85f7a064926eb1db68cc52bc4d4a91..a1aca5de08a0cb840a90fb3f729a5940f8d21185 | grep ^commit | wc -l 11 commits +------- git-bisect bad Bisecting: 5 revisions left to test after this [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig +------- Compiling... +-------- time make -j6 bzImage modules +-------- Install ... Booting +------ Linux dcu-router-ng 2.6.27-bisect #10 SMP PREEMPT Thu Nov 20 15:56:10 CET 2008 i686 GNU/Linux +------ BAD kernel. BAD rmmod NIU driver. +--------- git bisect bad Bisecting: 2 revisions left to test after this [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out +--------- Compiling... +------ time make -j6 bzImage modules #real 7m23.814s #user 12m15.718s #sys 0m42.183s +------ Config change prompting: +----- Support sparse irq numbering (HAVE_SPARSE_IRQ) [Y/n/?] (NEW) Y +----- Installing ... Booting ... +------- Linux dcu-router-ng 2.6.27-bisect #11 SMP PREEMPT Thu Nov 20 16:19:29 CET 2008 i686 GNU/Linux +------- GOOD!!! +-------- cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 124 0 0 0 IO-APIC-edge timer 1: 0 0 1 1 IO-APIC-edge i8042 3: 2 2 2 2 IO-APIC-edge serial 8: 1 0 0 1 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 1 2 1 0 IO-APIC-edge i8042 16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6 17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 6291710: 285 288 293 287 PCI-MSI-edge cciss0 11534592: 0 0 0 0 PCI-MSI-edge eth2 11534591: 0 0 0 0 PCI-MSI-edge eth2 11534590: 0 0 0 0 PCI-MSI-edge eth2 11534589: 0 0 0 0 PCI-MSI-edge eth2 11534588: 0 0 0 0 PCI-MSI-edge eth2 11534587: 0 0 0 0 PCI-MSI-edge eth2 11534586: 0 0 0 0 PCI-MSI-edge eth2 11534585: 0 0 0 0 PCI-MSI-edge eth2 11534584: 0 0 0 0 PCI-MSI-edge eth2 11534583: 0 0 0 0 PCI-MSI-edge eth2 11534582: 0 0 0 0 PCI-MSI-edge eth2 11534581: 0 0 0 0 PCI-MSI-edge eth2 11534580: 0 0 0 0 PCI-MSI-edge eth2 22: 23 24 23 23 IO-APIC-fasteoi uhci_hcd:usb5 11538688: 0 0 0 0 PCI-MSI-edge eth3 11538687: 0 0 0 0 PCI-MSI-edge eth3 11538686: 0 0 0 0 PCI-MSI-edge eth3 11538685: 0 0 0 0 PCI-MSI-edge eth3 11538684: 0 0 0 0 PCI-MSI-edge eth3 11538683: 0 0 0 0 PCI-MSI-edge eth3 11538682: 0 0 0 0 PCI-MSI-edge eth3 11538681: 0 0 0 0 PCI-MSI-edge eth3 11538680: 0 0 0 0 PCI-MSI-edge eth3 11538679: 0 0 0 0 PCI-MSI-edge eth3 11538678: 0 0 0 0 PCI-MSI-edge eth3 19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 3145984: 244 242 238 241 PCI-MSI-edge eth0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 3715 3104 2853 2542 Local timer interrupts RES: 88 52 280 258 Rescheduling interrupts CAL: 76 75 93 59 Function call interrupts TLB: 245 241 312 283 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 MIS: 0 +--------- +--------- git-bisect good Bisecting: 1 revisions left to test after this [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n +---------- Compiling ... +------ time make -j6 bzImage modules +------ Install... Boot ... +------- Linux dcu-router-ng 2.6.27-bisect #12 SMP PREEMPT Thu Nov 20 16:33:11 CET 2008 i686 GNU/Linux +------- +-------- cat /proc/interrupts CPU0 CPU1 CPU2 CPU3 0: 124 0 0 0 IO-APIC-edge timer 1: 0 0 1 1 IO-APIC-edge i8042 3: 1 2 2 2 IO-APIC-edge serial 8: 2 0 0 0 IO-APIC-edge rtc 9: 0 0 0 0 IO-APIC-fasteoi acpi 12: 1 2 1 0 IO-APIC-edge i8042 16: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb1, ehci_hcd:usb6 17: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb2 18: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb3 6291710: 268 269 268 267 PCI-MSI-edge cciss0 11534592: 0 0 0 0 PCI-MSI-edge eth2 11534591: 0 0 0 0 PCI-MSI-edge eth2 11534590: 0 0 0 0 PCI-MSI-edge eth2 11534589: 0 0 0 0 PCI-MSI-edge eth2 11534588: 0 0 0 0 PCI-MSI-edge eth2 11534587: 0 0 0 0 PCI-MSI-edge eth2 11534586: 0 0 0 0 PCI-MSI-edge eth2 11534585: 0 0 0 0 PCI-MSI-edge eth2 11534584: 0 0 0 0 PCI-MSI-edge eth2 11534583: 0 0 0 0 PCI-MSI-edge eth2 11534582: 0 0 0 0 PCI-MSI-edge eth2 11534581: 0 0 0 0 PCI-MSI-edge eth2 11534580: 0 0 0 0 PCI-MSI-edge eth2 11538688: 0 0 0 0 PCI-MSI-edge eth3 11538687: 0 0 0 0 PCI-MSI-edge eth3 11538686: 0 0 0 0 PCI-MSI-edge eth3 11538685: 0 0 0 0 PCI-MSI-edge eth3 11538684: 0 0 0 0 PCI-MSI-edge eth3 11538683: 0 0 0 0 PCI-MSI-edge eth3 11538682: 0 0 0 0 PCI-MSI-edge eth3 11538681: 0 0 0 0 PCI-MSI-edge eth3 11538680: 0 0 0 0 PCI-MSI-edge eth3 11538679: 0 0 0 0 PCI-MSI-edge eth3 11538678: 0 0 0 0 PCI-MSI-edge eth3 19: 0 0 0 0 IO-APIC-fasteoi uhci_hcd:usb4 22: 25 23 24 25 IO-APIC-fasteoi uhci_hcd:usb5 3145984: 175 174 176 178 PCI-MSI-edge eth0 NMI: 0 0 0 0 Non-maskable interrupts LOC: 3508 2902 2765 2489 Local timer interrupts RES: 238 35 461 6 Rescheduling interrupts CAL: 61 90 59 81 Function call interrupts TLB: 257 220 299 300 TLB shootdowns TRM: 0 0 0 0 Thermal event interrupts SPU: 0 0 0 0 Spurious interrupts ERR: 0 MIS: 0 +-------- GOOD. +---------- git-bisect good 3235e936c0cc3589309280b6f59e5096779adae3 is first bad commit commit 3235e936c0cc3589309280b6f59e5096779adae3 Author: Thomas Gleixner Date: Wed Oct 15 13:16:00 2008 +0200 x86: remove sparse irq from Kconfig This code is not ready yet. Signed-off-by: Thomas Gleixner :040000 040000 6043e32465556e828de0fbb6aa497b277239af01 2dd75ba207990d83a3a4c7b7b16abccfe2d5e10d M arch +-------- Found bad commit: 3235e936c0cc3589309280b6f59e5096779adae3 Git bisect LOG ~~~~~~~~~~~~~~ +------- git-bisect log git-bisect start # good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27 git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5 # bad: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip git-bisect bad 92b29b86fe2e183d44eb467e5e74a5f718ef2e43 # good: [af5c2bd16ac2e5688c3bf46ea1f95112d696d294] x86: fix virt_addr_valid() with CONFIG_DEBUG_VIRTUAL=y, v2 git-bisect good af5c2bd16ac2e5688c3bf46ea1f95112d696d294 # good: [36ac1d2f323f8bf8bc10c25b88f617657720e241] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/dtor/input git-bisect good 36ac1d2f323f8bf8bc10c25b88f617657720e241 # good: [1aece34833721d64eb33fc15cd923c727296d3d3] container freezer: rename check_if_frozen() git-bisect good 1aece34833721d64eb33fc15cd923c727296d3d3 # good: [1d9a8a47d659f053abeca9ece45651b4d94780c8] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mszeredi/fuse git-bisect good 1d9a8a47d659f053abeca9ece45651b4d94780c8 # bad: [dd3a1db900f2a215a7d7dd71b836e149a6cf5fed] genirq: improve include files git-bisect bad dd3a1db900f2a215a7d7dd71b836e149a6cf5fed # good: [db4b5525caafd846ec20f95afbc6403c792e22cf] x86: apic_64.c - setup_APIC_timer has to be __cpuinit function git-bisect good db4b5525caafd846ec20f95afbc6403c792e22cf # good: [ba374c9baef910fbc5373541d98c50f15e82c3f8] x86: fix HPET compiler error when not using CONFIG_PCI_MSI git-bisect good ba374c9baef910fbc5373541d98c50f15e82c3f8 # good: [922402f15a85f7a064926eb1db68cc52bc4d4a91] x86: Add UV partition call v4 git-bisect good 922402f15a85f7a064926eb1db68cc52bc4d4a91 # bad: [a1aca5de08a0cb840a90fb3f729a5940f8d21185] genirq: remove artifacts from sparseirq removal git-bisect bad a1aca5de08a0cb840a90fb3f729a5940f8d21185 # bad: [3235e936c0cc3589309280b6f59e5096779adae3] x86: remove sparse irq from Kconfig git-bisect bad 3235e936c0cc3589309280b6f59e5096779adae3 # good: [4c66a73f0796dacc2ff0d4af75794ec843ceb3d1] x86: sparse_irq: fix typo in debug print out git-bisect good 4c66a73f0796dacc2ff0d4af75794ec843ceb3d1 # good: [7ef0c30dbf96a8d9a234e90c248eb19df3c031be] genirq: define nr_irqs for architectures with GENERIC_HARDIRQS=n git-bisect good 7ef0c30dbf96a8d9a234e90c248eb19df3c031be +------- Email ~~~~~ To: Thomas Gleixner David Miller , Jesper Dangaard Brouer , netdev , linux-kernel@vger.kernel.org, Robert Olsson Subj.: Regression: Bisected, IRQ and MSI allocations screwed without sparse irq Hi Thomas Gleixner, I have bisected a regression to your commit 3235e936c0cc3589309280b6f59e5096779adae3, "x86: remove sparse irq from Kconfig". Its actually not necessary your fault, as your commit simply removes the config option HAVE_SPARSE_IRQ. This revels the bug / regression I'm exposted to. Guess I should bisect again to find the exact faulty commit, but I'm rather sick of bisecting at the moment, and though you might have a better idea whats going wrong. I would rather spend my time performance tuning the multiqueue routing code... [The regression]: During my testing of the Sun Neptune based NICs. On kernel 2.6.27 I get really good performance (900-1200kpps) compared to 2.6.28 (davem git net-2.6). The cause of this problem (tracked down together with Robert Olsson) is that on 2.6.28 I have a lot less IRQs available. It seems max 34 IRQs. Due the reduced number of IRQs the NIU driver cannot get enough IRQs to the interfaces, and starts to use "IO-APIC" based IRQs. On kernel 2.6.28: My eth2 is using 10 IRQs all "PCI-MSI-edge". BUT my eth3 is using a single IRQ using "IO-APIC-fasteoi" and shared with the usb driver. That my performance problem on 2.6.28. [Other related bugs]: Is that unloading the "niu" driver will give a kernel BUG.