Message-ID: <878d7651-433a-46fe-a28b-1b7e893fcbe0@linux.ibm.com>
Date: Wed, 8 Oct 2025 16:27:00 +0530
From: Venkat Rao Bagalkote <venkat88@...ux.ibm.com>
To: namcao@...utronix.de, Madhavan Srinivasan <maddy@...ux.ibm.com>,
        LKML <linux-kernel@...r.kernel.org>,
        linuxppc-dev <linuxppc-dev@...ts.ozlabs.org>,
        Gautam Menghani <gautam@...ux.ibm.com>
Subject: [bisected][mainline]Kernel OOPs at msi_desc_to_pci_dev

Greetings!!


IBM CI has reported a kernel crash while running module load and unload 
testing on lpfc driver.


Traces:


[  177.928316] BUG: Kernel NULL pointer dereference on read at 0x00000008
[  177.928323] Faulting instruction address: 0xc0000000009c5d68
[  177.928331] Oops: Kernel access of bad area, sig: 11 [#1]
[  177.928337] LE PAGE_SIZE=64K MMU=Hash  SMP NR_CPUS=8192 NUMA pSeries
[  177.928346] Modules linked in: rpadlpar_io rpaphp nft_compat xsk_diag 
nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet 
nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct bonding nft_chain_nat 
nf_nat nf_conntrack tls nf_defrag_ipv6 nf_defrag_ipv4 ip_set rfkill 
nf_tables nfnetlink binfmt_misc dm_round_robin dm_multipath dm_mod 
pseries_rng vmx_crypto drm fuse drm_panel_orientation_quirks xfs sd_mod 
sg lpfc(-) nvmet_fc ibmvscsi ibmveth scsi_transport_srp nvmet nvme_fc 
nvme_fabrics nvme_core scsi_transport_fc
[  177.928436] CPU: 0 UID: 0 PID: 10129 Comm: modprobe Kdump: loaded Not 
tainted 6.17.0-rc4-auto-00024-g043439ad1a23-dirty #17 VOLUNTARY
[  177.928448] Hardware name: IBM,9009-42A POWER9 (architected) 0x4e0202 
0xf000005 of:IBM,FW950.A0 (VL950_141) hv:phyp pSeries
[  177.928455] NIP:  c0000000009c5d68 LR: c00000000010e550 CTR: 
c00000000010e52c
[  177.928462] REGS: c000000132e1f720 TRAP: 0300   Not tainted 
(6.17.0-rc4-auto-00024-g043439ad1a23-dirty)
[  177.928471] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  
CR: 48008222  XER: 20040000
[  177.928494] CFAR: c00000000000dbbc DAR: 0000000000000008 DSISR: 
40000000 IRQMASK: 0
[  177.928494] GPR00: c00000000010e550 c000000132e1f9c0 c000000001678100 
0000000000000000
[  177.928494] GPR04: c0000001329f75f0 c000000132e1fab8 c000000120459d00 
fffffffffffe0000
[  177.928494] GPR08: 0000000000000000 c0000001329f7540 0000000000000000 
c00800000d8e3560
[  177.928494] GPR12: c00000000010e52c c000000002ff0000 0000000000000000 
0000000000000000
[  177.928494] GPR16: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[  177.928494] GPR20: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[  177.928494] GPR24: 0000000000000000 0000000000000000 c00800000d905f38 
c000000012076394
[  177.928494] GPR28: 0000000000000000 c0000001329f7598 c000000120484680 
c0000000120760c8
[  177.928580] NIP [c0000000009c5d68] msi_desc_to_pci_dev+0x8/0x14
[  177.928596] LR [c00000000010e550] pseries_msi_ops_teardown+0x24/0x38
[  177.928609] Call Trace:
[  177.928613] [c000000132e1f9c0] [c00000000dd3a8b0] 0xc00000000dd3a8b0 
(unreliable)
[  177.928625] [c000000132e1f9e0] [c000000000244e44] 
msi_remove_device_irq_domain+0x98/0x18c
[  177.928637] [c000000132e1fa20] [c000000000244f6c] 
msi_device_data_release+0x34/0xa8
[  177.928647] [c000000132e1fa50] [c000000000af7648] 
release_nodes+0x64/0x13c
[  177.928663] [c000000132e1fa90] [c000000000af9770] 
devres_release_all+0xc0/0x130
[  177.928672] [c000000132e1fb00] [c000000000aee2a8] 
device_unbind_cleanup+0x2c/0xb0
[  177.928682] [c000000132e1fb30] [c000000000af0668] 
device_release_driver_internal+0x2dc/0x32c
[  177.928692] [c000000132e1fb80] [c000000000af077c] 
driver_detach+0x8c/0x160
[  177.928701] [c000000132e1fbc0] [c000000000aedab0] 
bus_remove_driver+0x94/0x140
[  177.928710] [c000000132e1fc40] [c000000000af16a8] 
driver_unregister+0x48/0x88
[  177.928720] [c000000132e1fcb0] [c0000000009b808c] 
pci_unregister_driver+0x3c/0x184
[  177.928733] [c000000132e1fd00] [c00800000d8e18e4] 
lpfc_exit+0x38/0xe754 [lpfc]
[  177.928825] [c000000132e1fd30] [c00000000027bfd8] 
__do_sys_delete_module.constprop.0+0x1ac/0x3ec
[  177.928836] [c000000132e1fe10] [c000000000033338] 
system_call_exception+0x138/0x330
[  177.928847] [c000000132e1fe50] [c00000000000d05c] 
system_call_vectored_common+0x15c/0x2ec
[  177.928860] ---- interrupt: 3000 at 0x7fffb17d848c
[  177.928867] NIP:  00007fffb17d848c LR: 00007fffb17d848c CTR: 
0000000000000000
[  177.928873] REGS: c000000132e1fe80 TRAP: 3000   Not tainted 
(6.17.0-rc4-auto-00024-g043439ad1a23-dirty)
[  177.928880] MSR:  800000000280f033 
<SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44004882  XER: 00000000
[  177.928902] IRQMASK: 0
[  177.928902] GPR00: 0000000000000081 00007fffcc632ff0 00007fffb18c7100 
00000100200c1e58
[  177.928902] GPR04: 0000000000000800 00000001015e71d8 0000000000000000 
00000000000000da
[  177.928902] GPR08: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[  177.928902] GPR12: 0000000000000000 00007fffb20cc5a0 0000000000000000 
0000000000000000
[  177.928902] GPR16: 00007fffcc6358b8 0000000000000001 00000001015e7c10 
00000100200c02f8
[  177.928902] GPR20: 0000000000000000 00000001015e8b38 00000100200c1df0 
0000000000000000
[  177.928902] GPR24: 00007fffcc6358c8 00007fffcc6358c8 00000100200d0430 
00000100200c02d0
[  177.928902] GPR28: 00000100200c1e58 0000000000000000 00000100200c1df0 
0000000000100000
[  177.928979] NIP [00007fffb17d848c] 0x7fffb17d848c
[  177.928985] LR [00007fffb17d848c] 0x7fffb17d848c
[  177.928991] ---- interrupt: 3000
[  177.928997] Code: ebc10020 38210030 e8010010 ebe1fff8 7c0803a6 
4e800020 ebc10020 4bffff7c 60000000 60000000 7c0802a6 60000000 
<e8630008> 3863ff38 4e800020 3c4c00cb
[  177.929034] ---[ end trace 0000000000000000 ]---
[  177.935447] pstore: backend (nvram) writing error (-1)
[  177.935452]


Bisection logs:


daaa574aba6f9c683408b58a7ab2dc775ece2f98 is the first bad commit
commit daaa574aba6f9c683408b58a7ab2dc775ece2f98 (HEAD)
Author: Nam Cao <namcao@...utronix.de>
Date:   Mon Aug 11 11:28:56 2025 +0200

     powerpc/pseries/msi: Switch to msi_create_parent_irq_domain()

     Move away from the legacy MSI domain setup, switch to use
     msi_create_parent_irq_domain().

     Signed-off-by: Nam Cao <namcao@...utronix.de>
     Signed-off-by: Madhavan Srinivasan <maddy@...ux.ibm.com>
     Link: 
https://patch.msgid.link/c7a6d8f27fd217021dea4daad777e81a525ae460.1754903590.git.namcao@linutronix.de

  arch/powerpc/include/asm/pci-bridge.h  |   2 --
  arch/powerpc/platforms/pseries/Kconfig |   1 +
  arch/powerpc/platforms/pseries/msi.c   | 114 
++++++++++++++++++++++++++++++++++++++++++++++++------------------------------------------------------------------
  3 files changed, 49 insertions(+), 68 deletions(-)


git bisect start
# status: waiting for both good and bad commits
# bad: [c746c3b5169831d7fb032a1051d8b45592ae8d78] Merge tag 
'for-6.18-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/kdave/linux
git bisect bad c746c3b5169831d7fb032a1051d8b45592ae8d78
# status: waiting for good commit(s), bad commit known
# good: [038d61fd642278bab63ee8ef722c50d10ab01e8f] Linux 6.16
git bisect good 038d61fd642278bab63ee8ef722c50d10ab01e8f
# good: [e25079858627916b22c4a789005a90a9fae808d8] Merge branch 
'net-better-drop-accounting'
git bisect good e25079858627916b22c4a789005a90a9fae808d8
# bad: [05a54fa773284d1a7923cdfdd8f0c8dabb98bd26] Merge tag 
'sound-6.18-rc1' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tiwai/sound
git bisect bad 05a54fa773284d1a7923cdfdd8f0c8dabb98bd26
# bad: [ae28ed4578e6d5a481e39c5a9827f27048661fdd] Merge tag 
'bpf-next-6.18' of 
git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next
git bisect bad ae28ed4578e6d5a481e39c5a9827f27048661fdd
# good: [6855f06042ae8d134f96c63feb5dfb3943c6d789] Merge tag 
'i2c-for-6.17-rc8' of 
git://git.kernel.org/pub/scm/linux/kernel/git/wsa/linux
git bisect good 6855f06042ae8d134f96c63feb5dfb3943c6d789
# bad: [30d4efb2f5a515a60fe6b0ca85362cbebea21e2f] Merge tag 
'for-linus-6.18-rc1-tag' of 
git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip
git bisect bad 30d4efb2f5a515a60fe6b0ca85362cbebea21e2f
# good: [a9401710a5f5681abd2a6f21f9e76bc9f2e81891] Merge tag 
'v6.18-rc-part1-smb3-common' of git://git.samba.org/ksmbd
git bisect good a9401710a5f5681abd2a6f21f9e76bc9f2e81891
# good: [fe68bb2861808ed5c48d399bd7e670ab76829d55] Merge tag 
'microblaze-v6.18' of git://git.monstr.eu/linux-2.6-microblaze
git bisect good fe68bb2861808ed5c48d399bd7e670ab76829d55
# good: [feafee284579d29537a5a56ba8f23894f0463f3d] Merge tag 
'arm64-upstream' of 
git://git.kernel.org/pub/scm/linux/kernel/git/arm64/linux
git bisect good feafee284579d29537a5a56ba8f23894f0463f3d
# good: [9cc220a422113f665e13364be1411c7bba9e3e30] Merge tag 
's390-6.18-1' of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux
git bisect good 9cc220a422113f665e13364be1411c7bba9e3e30
# bad: [b48b6cc8c655d8cdcf5124ba9901b74c8f759668] powerpc/pseries: 
Enable HVPIPE event message interrupt
git bisect bad b48b6cc8c655d8cdcf5124ba9901b74c8f759668
# good: [7f9bcf13069731fac48d8b44086fab179fbc04c9] powerpc/cpm2: Drop 
legacy-of-mm-gpiochip.h header
git bisect good 7f9bcf13069731fac48d8b44086fab179fbc04c9
# good: [f0ac60e6e311062f1a452d93376055787db4b070] powerpc/powernv/pci: 
Switch to use msi_create_parent_irq_domain()
git bisect good f0ac60e6e311062f1a452d93376055787db4b070
# bad: [814ef095f12c9fa142043ee689500f3a41bb6dab] powerpc/pseries: Add 
papr-hvpipe char driver for HVPIPE interfaces
git bisect bad 814ef095f12c9fa142043ee689500f3a41bb6dab
# bad: [043439ad1a23cd3f65628310d1f5a06e61f8b431] powerpc/pseries: 
Define papr-hvpipe ioctl
git bisect bad 043439ad1a23cd3f65628310d1f5a06e61f8b431
# bad: [daaa574aba6f9c683408b58a7ab2dc775ece2f98] powerpc/pseries/msi: 
Switch to msi_create_parent_irq_domain()
git bisect bad daaa574aba6f9c683408b58a7ab2dc775ece2f98
# first bad commit: [daaa574aba6f9c683408b58a7ab2dc775ece2f98] 
powerpc/pseries/msi: Switch to msi_create_parent_irq_domain()


If you happen to fix this, please add below tag.


Reported-by: Venkat Rao Bagalkote <venkat88@...ux.ibm.com>



Regards,

Venkat.

Powered by blists - more mailing lists