~~ -*-text-*- ------------------------------------------------------- Bisecting bug: NIU driver rmmod MSI-X bug ------------------------------------------------------- Jesper Dangaard Brouer (jdb@comx.dk) ------------------------------------------------------- $LastChangedRevision: 772 $ $Date: 2008-11-19 13:08:13 +0100 (Wed, 19 Nov 2008) $ ------------------------------------------------------- git clone ~~~~~~~~~ +--------- cd /var/kernels/git/davem git clone git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-2.6.git net-2.6-bisect +--------- Bug description ~~~~~~~~~~~~~~~ NIU driver rmmod kernel MSI-X bug. I found a strange behavior, rmmod'ing the niu driver will only cause a kernel BUG, if the driver was loaded at boot time. If I remove the niu.ko driver from /lib/modules/2.6.28-rc4-davem/kernel/drivers/net/ reboot the system. After that I can load and unload the niu.ko driver without problems... hmmm. Reproduce / test ~~~~~~~~~~~~~~~~ 1. Boot machine 2. rmmod niu 3. look at dmesg for kernel BUG output Install trick ~~~~~~~~~~~~~ Installing kernel in a seperate directory. +-------- export VER=`cat include/config/kernel.release` echo $VER export INSTALL_MOD_PATH=/var/kernels/git/install/ rm -rf $INSTALL_MOD_PATH/lib/modules/$VER/kernel/ make modules_install export BOOT="$INSTALL_MOD_PATH/boot/" [ -d $BOOT ] || mkdir $BOOT cp -v arch/x86/boot/bzImage $BOOT/vmlinuz-$VER cp -v arch/i386/boot/bzImage $BOOT/vmlinuz-$VER cp -v System.map $BOOT/System.map-$VER cp -v vmlinux $BOOT/vmlinux-$VER +-------- * Push to test host ~~~~~~~~~~~~~~~~~ export KERNEL=2.6.27-davem export KERNEL=2.6.28-rc2-davem export KERNEL=2.6.28-rc4-davem +-------- export HOST=ng export KERNEL=$VER pushd /var/kernels/git/install rsync -e ssh -avz boot/vmlinuz-${KERNEL} root@${HOST}:/boot/ rsync -e ssh -avz boot/vmlinux-${KERNEL} root@${HOST}:/boot/ rsync -e ssh -avz --delete lib/modules/${KERNEL} root@${HOST}:/lib/modules/ popd +-------- Create branch from tag: ~~~~~~~~~~~~~~~~~~~~~~~ Known good starting point. +----------- git branch tag_v2.6.27 v2.6.27 git checkout tag_v2.6.27 +----------- Start bisect ~~~~~~~~~~~~ +-------- git checkout master git bisect start git bisect good v2.6.27 git bisect bad master +-------- Create .config by <<>> or <<>> +------- bisect good #Bisecting: 2157 revisions left to test after this #[92b29b86fe2e183d44eb467e5e74a5f718ef2e43] #Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip +------- compile: time make -j6 bzImage modules Sort of "good" version, problem is that number of IRQs when down a lot! This probably means bad performance as I cannot get enough IRQs for the Sun NIC. +-------- bisect good #Bisecting: 1066 revisions left to test after this #[ea541686d8454efac4f2b5c0767affb12d4b6a52] #Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds +-------- compile: time make -j6 bzImage modules CRAP! - kernel (commit ea541686d8454efac4f2b5c0767affb12d4b6a52) will not boot on my system :-((( Stops with message: +------ hpet0: 3 comparators, 64-bit 14.318180 Mhz counter +------ <> removed "High Resolution Timer Support" in .config CONFIG_HIGH_RES_TIMERS and CONFIG_SCHED_HRTICK. Disable "HPET Timer Support" (under "Processor type and features"). Undefs CONFIG_HPET_TIMER and CONFIG_HPET_EMULATE_RTC. Compiling ... installing ... It was not the problem... now I get a "BUG: soft lockup" kstop EIP is a stop_cpu+0x37/0xb0. * Parallel process#1: ~~~~~~~~~~~~~~~~~~~~ <> Pick a new commit point a make a new seperate branch and try to see if we cab boot... random picked commit 9a1c3542768b5a58e45a9216921cd10a3bae1205 git checkout -b new_bisect_point01 9a1c3542768b5a58e45a9216921cd10a3bae1205 Compile on another tree davem/net-2.6-copy. Kernel named "-test". It can boot and unloading "niu" driver WORKS! If I understand bisect it should be possible to call: git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205 * Parallel process#2: ~~~~~~~~~~~~~~~~~~~~ <> use git-bisect skip +------ git-bisect skip Bisecting: 1066 revisions left to test after this [969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6 +------ STILL HANGS on boot :-((( * Back on track... ~~~~~~~~~~~~~~~~~~ +-------- git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205 Bisecting: 1000 revisions left to test after this [1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage +-------- compile... install... BAD: Booting (now 2.6.28-rc1-bisect) and unloading "niu" causes the bug! Mark bisecting as BAD. +---------- git-bisect bad Bisecting: 508 revisions left to test after this [36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6 +---------- compile ... install ... (back on ver. 2.6.27-bisect) GRRRR... Now I cannot boot again! :-((( * Try to find a new place ~~~~~~~~~~~~~~~~~~~~~~~ Random pick: dbacefc9c4f6bd365243db379473ab7041656d90 +-------- cd /var/kernels/git/davem/net-2.6-copy/ git checkout -b new_bisect_point02 dbacefc9c4f6bd365243db379473ab7041656d90 +-------- VER=2.6.27-rc1-test This version can boot and unloading niu works. * Back on track(2) ... but not :-( ~~~~~~~~~~~~~~~~~~ git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90 +-------- git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90 Bisecting: 508 revisions left to test after this [36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6 +--------- Hmmm... commit 36ec891895020f3bc9953c8b11d079c6d77d76bd was the same as before... which could not boot... * Try a bisect skip... ~~~~~~~~~~~~~~~~~~~~~ +--------- git bisect skip Bisecting: 508 revisions left to test after this [70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6 +--------- compile ... install ... boot... CANNOT BOOT !!! :-((( * Doing lucky guessing ~~~~~~~~~~~~~~~~~~~~~~ git-bisect good +-------------- git-bisect good Bisecting: 259 revisions left to test after this [22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev +-------------- ARGH!!! -- cannot boot this kernel, it hangs :-((( * Doing desperate guessing ~~~~~~~~~~~~~~~~~~~~~~~~~~ +-------- git-bisect good Bisecting: 132 revisions left to test after this [c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip +-------- compiling ... installing ... ... booting -- ARGH! cannot boot hang! * Doing hopeless guessing ~~~~~~~~~~~~~~~~~~~~~~~~~ This is hopeless, I should give up! +----------------- git-bisect good Bisecting: 65 revisions left to test after this [969907a956752f88dde4aa23fa8c033b9a939aee] Merge git://git.kernel.org/pub/scm/linux/kernel/git/bart/ide-2.6 +------------------ compiling ... installing ... still hangs after boot... :-((( ... GIVING UP!!! * git bisect log ~~~~~~~~~~~~~~~~ +------- git-bisect start # good: [3fa8749e584b55f1180411ab1b51117190bac1e5] Linux 2.6.27 git-bisect good 3fa8749e584b55f1180411ab1b51117190bac1e5 # bad: [5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776] rtnetlink: propagate error from dev_change_flags in do_setlink() git-bisect bad 5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776 # good: [29415c37f043d1d54dcf356601d738ff6633b72b] KVM: set debug registers after "schedulable" section git-bisect good 29415c37f043d1d54dcf356601d738ff6633b72b # good: [92b29b86fe2e183d44eb467e5e74a5f718ef2e43] Merge branch 'tracing-v28-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip git-bisect good 92b29b86fe2e183d44eb467e5e74a5f718ef2e43 # skip: [ea541686d8454efac4f2b5c0767affb12d4b6a52] Merge branch 'for-linus' of git://git.o-hand.com/linux-rpurdie-leds git-bisect skip ea541686d8454efac4f2b5c0767affb12d4b6a52 # good: [9a1c3542768b5a58e45a9216921cd10a3bae1205] pass fmode_t to blkdev_put() git-bisect good 9a1c3542768b5a58e45a9216921cd10a3bae1205 # bad: [1137fb670465b6b5d15b9db7d01707a5833ee3ae] arm ide breakage git-bisect bad 1137fb670465b6b5d15b9db7d01707a5833ee3ae # good: [dbacefc9c4f6bd365243db379473ab7041656d90] fs/buffer.c: uninline __remove_assoc_queue() git-bisect good dbacefc9c4f6bd365243db379473ab7041656d90 # skip: [36ec891895020f3bc9953c8b11d079c6d77d76bd] Merge git://git.kernel.org/pub/scm/linux/kernel/git/lethal/sh-2.6 git-bisect skip 36ec891895020f3bc9953c8b11d079c6d77d76bd # good: [70740d6c93030b339b4ad17fd58ee135dfc13913] Merge branch 'drm-next' of git://git.kernel.org/pub/scm/linux/kernel/git/airlied/drm-2.6 git-bisect good 70740d6c93030b339b4ad17fd58ee135dfc13913 # good: [22484856402bfa1ff3defe47f6029ab0418240d9] Merge git://git.kernel.org/pub/scm/linux/kernel/git/viro/bdev git-bisect good 22484856402bfa1ff3defe47f6029ab0418240d9 # good: [c3c9897c63ebb0b93b7f78724e38d6ee1da04041] Merge branch 'x86-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/linux-2.6-tip git-bisect good c3c9897c63ebb0b93b7f78724e38d6ee1da04041 +------- Compile: ~~~~~~~~ +------- time make -j6 bzImage modules +------- NOTES: ~~~~~~ git log 29415c37f043d1d54dcf356601d738ff6633b72b..5f9021cfdc3524a4c5e3d7ae2d049eb7adcd6776 NR_IRQS changed ... could this be releated? git show 7db282fa67b58daff8a57f9e1c93d4474b5908ff git show 1b4897688011cd05e07f00dcfe6af3331eb36a3c git show c78d0cf2925bffae8a6f00e7d9b8e971b0392edd