Message-ID: <aafeb4a9-31ea-43ad-b807-fd082cc0c9ad@linux.ibm.com>
Date: Mon, 23 Jun 2025 19:20:03 +0530
From: Venkat Rao Bagalkote <venkat88@...ux.ibm.com>
To: LKML <linux-kernel@...r.kernel.org>,
        Linux Next Mailing List <linux-next@...r.kernel.org>,
        Madhavan Srinivasan <maddy@...ux.ibm.com>,
        Stephen Rothwell <sfr@...b.auug.org.au>, viro@...iv.linux.org.uk,
        linux-fsdevel@...r.kernel.org, riteshh@...ux.ibm.com
Subject: [linux-next-20250620] Fails to boot to IBM Power Server

Greetings!!!


IBM CI has reported a boot issue, while trying to boot from 
linux-next-20250620 repo.


Git Bisect is pointing to below commit as the first bad commit.


a9ea6b0629a5a91d11db9318fba45a2e058babb1 is the first bad commit
commit a9ea6b0629a5a91d11db9318fba45a2e058babb1
Author: Al Viro <viro@...iv.linux.org.uk>
Date:   Tue Jun 17 00:09:51 2025 -0400

     replace collect_mounts()/drop_collected_mounts() with safer variant

     collect_mounts() has several problems - one can't iterate over the 
results
     directly, so it has to be done with callback passed to 
iterate_mounts();
     it also has oopsable race with d_invalidate(); it also creates 
temporary
     clones of mounts invisibly for sync umount (IOW, you can have 
umount return
     with filesystem not mounted in any other locations and yet have it 
still
     busy as umount(2) returns).

     A saner approach is to give caller an array of struct path that 
would pin
     every mount in a subtree, without cloning any mounts.

             * collect_mounts()/drop_collected_mounts()/iterate_mounts() 
is gone
             * collect_paths(where, preallocated, size) gives either 
ERR_PTR(-E...) or
     a pointer to array of struct path, one for each chunk of tree 
visible under
     'where' (i.e. the first element is a copy of where, followed by 
(mount,root)
     for everything mounted under it - the same set collect_mounts() 
would give).
     Unlike collect_mounts(), the mounts are *not* cloned - we just get 
(pinning)
     references to roots of subtree in the caller's namespaces.
             Array is terminated by {NULL, NULL} struct path.  If it 
fits into
     preallocated array (on-stack, normally), that's where it goes; 
otherwise
     it's allocated by kmalloc_array().  Passing 0 as size means that 
'preallocated'
     is ignored (and expected to be NULL).
             * drop_collected_paths(paths, preallocated) is given the 
array returned
     by collect_paths() and the preallocated array used passed to the 
same.  All
     mount/dentry references are dropped and array is kfree'd if it's 
not equal to
     'preallocated'.
             * instead of iterate_mounts(), users should just iterate 
over array
     of struct path - nothing exotic is needed for that.  Existing users 
(all in
     audit_tree.c) are converted.

     Fixes: 80b5dce8c59b0 ("vfs: Add a function to lazily unmount all 
mounts from any dentry")
     Signed-off-by: Al Viro <viro@...iv.linux.org.uk>

  fs/namespace.c        | 97 
+++++++++++++++++++++++++++++++--------------------
  fs/pnode.h            |  2 --
  include/linux/mount.h |  6 ++--
  kernel/audit_tree.c   | 63 ++++++++++++++++++---------------
  4 files changed, 95 insertions(+), 73 deletions(-)



Traces:


[   26.465091] Kernel attempted to read user page (0) - exploit attempt? 
(uid: 0)
[   26.465146] BUG: Kernel NULL pointer dereference on read at 0x00000000
[   26.465146] BUG: Kernel NULL pointer dereference on read at 0x00000000
[   26.465178] Faulting instruction address: 0xc00000000067a4e0
[   26.465178] Faulting instruction address: 0xc00000000067a4e0
[   26.465206] Oops: Kernel access of bad area, sig: 11 [#1]
[   26.465206] Oops: Kernel access of bad area, sig: 11 [#1]
[   26.465232] LE PAGE_SIZE=64K MMU=Radix  SMP NR_CPUS=8192 NUMA pSeries
[   26.465232] LE PAGE_SIZE=64K MMU=Radix  SMP NR_CPUS=8192 NUMA pSeries
[   26.465264] Modules linked in: nft_compat nf_tables nfnetlink 
rpadlpar_io rpaphp xsk_diag bonding rfkill binfmt_misc mlx5_ib ib_uverbs 
ib_core vmx_crypto pseries_rng drm drm_panel_orientation_quirks ext4 
crc16 mbcache jbd2 dm_service_time sd_mod sg nvme_tcp nvme_fabrics 
ibmvfc nvme_core mlx5_core scsi_transport_fc ibmveth mlxfw psample 
dm_multipath dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic 
uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp 
libiscsi_tcp libiscsi scsi_transport_iscsi fuse
[   26.465264] Modules linked in: nft_compat nf_tables nfnetlink 
rpadlpar_io rpaphp xsk_diag bonding rfkill binfmt_misc mlx5_ib ib_uverbs 
ib_core vmx_crypto pseries_rng drm drm_panel_orientation_quirks ext4 
crc16 mbcache jbd2 dm_service_time sd_mod sg nvme_tcp nvme_fabrics 
ibmvfc nvme_core mlx5_core scsi_transport_fc ibmveth mlxfw psample 
dm_multipath dm_mirror dm_region_hash dm_log dm_mod be2iscsi bnx2i cnic 
uio cxgb4i cxgb4 tls libcxgbi libcxgb qla4xxx iscsi_boot_sysfs iscsi_tcp 
libiscsi_tcp libiscsi scsi_transport_iscsi fuse
[   26.465508] CPU: 24 UID: 0 PID: 1196 Comm: osqueryd Kdump: loaded Not 
tainted 6.16.0-rc2-next-20250619-autotest #1 VOLUNTARY
[   26.465508] CPU: 24 UID: 0 PID: 1196 Comm: osqueryd Kdump: loaded Not 
tainted 6.16.0-rc2-next-20250619-autotest #1 VOLUNTARY

[   26.465615] NIP:  c00000000067a4e0 LR: c0000000003035e0 CTR: 
c00000000021b030
[   26.465615] NIP:  c00000000067a4e0 LR: c0000000003035e0 CTR: 
c00000000021b030
[   26.465649] REGS: c000000081b3f400 TRAP: 0300   Not tainted 
(6.16.0-rc2-next-20250619-autotest)
[   26.465649] REGS: c000000081b3f400 TRAP: 0300   Not tainted 
(6.16.0-rc2-next-20250619-autotest)
[   26.465692] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  
CR: 24608862  XER: 20040000
[   26.465692] MSR:  800000000280b033 <SF,VEC,VSX,EE,FP,ME,IR,DR,RI,LE>  
CR: 24608862  XER: 20040000
[   26.465740] CFAR: c0000000003035dc DAR: 0000000000000000 DSISR: 
40000000 IRQMASK: 0
[   26.465740] GPR00: c0000000003035e0 c000000081b3f6a0 c000000001658100 
c000000002bb0e38
[   26.465740] GPR04: c000000081b3f798 0000000000000010 c000000081b3f4e0 
0000000000000018
[   26.465740] GPR08: c000000081a88e80 0000000000000000 0000000000000000 
c008000006c5fc88
[   26.465740] GPR12: c00000000021b030 c000000c7db72b00 0000000000000000 
0000000000000000
[   26.465740] GPR16: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[   26.465740] GPR20: 0000000000000000 0000000000000018 c000000060abb810 
c000000002b777a0
[   26.465740] GPR24: c000000002b776e0 c00000007f673f70 c00000007f6c3930 
c000000002bb0e38
[   26.465740] GPR28: 0000000000000000 c00000007f673e20 c00000007f6c3900 
ffffffffffffffea
[   26.465740] CFAR: c0000000003035dc DAR: 0000000000000000 DSISR: 
40000000 IRQMASK: 0
[   26.465740] GPR00: c0000000003035e0 c000000081b3f6a0 c000000001658100 
c000000002bb0e38
[   26.465740] GPR04: c000000081b3f798 0000000000000010 c000000081b3f4e0 
0000000000000018
[   26.465740] GPR08: c000000081a88e80 0000000000000000 0000000000000000 
c008000006c5fc88
[   26.465740] GPR12: c00000000021b030 c000000c7db72b00 0000000000000000 
0000000000000000
[   26.465740] GPR16: 0000000000000000 0000000000000000 0000000000000000 
0000000000000000
[   26.465740] GPR20: 0000000000000000 0000000000000018 c000000060abb810 
c000000002b777a0
[   26.465740] GPR24: c000000002b776e0 c00000007f673f70 c00000007f6c3930 
c000000002bb0e38
[   26.465740] GPR28: 0000000000000000 c00000007f673e20 c00000007f6c3900 
ffffffffffffffea
[   26.466083] NIP [c00000000067a4e0] collect_paths+0x5c/0x2c4
[   26.466083] NIP [c00000000067a4e0] collect_paths+0x5c/0x2c4
[   26.466113] LR [c0000000003035e0] audit_add_tree_rule+0x38c/0x6c0
[   26.466113] LR [c0000000003035e0] audit_add_tree_rule+0x38c/0x6c0
[   26.466146] Call Trace:
[   26.466146] Call Trace:
[   26.466158] [c000000081b3f6a0] [c000000081b3f6e0] 0xc000000081b3f6e0 
(unreliable)
[   26.466158] [c000000081b3f6a0] [c000000081b3f6e0] 0xc000000081b3f6e0 
(unreliable)
[   26.466195] [c000000081b3f720] [c0000000003035e0] 
audit_add_tree_rule+0x38c/0x6c0
[   26.466195] [c000000081b3f720] [c0000000003035e0] 
audit_add_tree_rule+0x38c/0x6c0
[   26.466232] [c000000081b3f8f0] [c0000000002f5698] 
audit_add_rule+0xc4/0x368
[   26.466232] [c000000081b3f8f0] [c0000000002f5698] 
audit_add_rule+0xc4/0x368
[   26.466267] [c000000081b3f960] [c0000000002f6d1c] 
audit_rule_change+0x84/0x240
[   26.466267] [c000000081b3f960] [c0000000002f6d1c] 
audit_rule_change+0x84/0x240
[   26.466302] [c000000081b3f9a0] [c0000000002f2bcc] 
audit_receive_msg+0x370/0x131c
[   26.466302] [c000000081b3f9a0] [c0000000002f2bcc] 
audit_receive_msg+0x370/0x131c
[   26.466338] [c000000081b3fac0] [c0000000002f3c94] 
audit_receive+0x11c/0x220
[   26.466338] [c000000081b3fac0] [c0000000002f3c94] 
audit_receive+0x11c/0x220
[   26.466371] [c000000081b3fb40] [c000000000e7d964] 
netlink_unicast+0x328/0x3bc
[   26.466371] [c000000081b3fb40] [c000000000e7d964] 
netlink_unicast+0x328/0x3bc
[   26.466408] [c000000081b3fbb0] [c000000000e7dc18] 
netlink_sendmsg+0x220/0x528
[   26.466408] [c000000081b3fbb0] [c000000000e7dc18] 
netlink_sendmsg+0x220/0x528
[   26.466443] [c000000081b3fca0] [c000000000d7d6b8] 
__sys_sendto+0x1fc/0x28c
[   26.466443] [c000000081b3fca0] [c000000000d7d6b8] 
__sys_sendto+0x1fc/0x28c
[   26.466477] [c000000081b3fdf0] [c000000000d7d784] sys_sendto+0x3c/0x4c
[   26.466477] [c000000081b3fdf0] [c000000000d7d784] sys_sendto+0x3c/0x4c
[   26.466509] [c000000081b3fe10] [c000000000033338] 
system_call_exception+0x138/0x330
[   26.466509] [c000000081b3fe10] [c000000000033338] 
system_call_exception+0x138/0x330
[   26.466547] [c000000081b3fe50] [c00000000000d05c] 
system_call_vectored_common+0x15c/0x2ec
[   26.466547] [c000000081b3fe50] [c00000000000d05c] 
system_call_vectored_common+0x15c/0x2ec
[   26.466588] ---- interrupt: 3000 at 0x7fff92f507c4
[   26.466588] ---- interrupt: 3000 at 0x7fff92f507c4
[   26.466611] NIP:  00007fff92f507c4 LR: 00007fff92f507c4 CTR: 
0000000000000000
[   26.466611] NIP:  00007fff92f507c4 LR: 00007fff92f507c4 CTR: 
0000000000000000
[   26.466645] REGS: c000000081b3fe80 TRAP: 3000   Not tainted 
(6.16.0-rc2-next-20250619-autotest)
[   26.466645] REGS: c000000081b3fe80 TRAP: 3000   Not tainted 
(6.16.0-rc2-next-20250619-autotest)
[   26.466685] MSR:  800000000280f033 
<SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44602461  XER: 00000000
[   26.466685] MSR:  800000000280f033 
<SF,VEC,VSX,EE,PR,FP,ME,IR,DR,RI,LE>  CR: 44602461  XER: 00000000
[   26.466734] IRQMASK: 0
[   26.466734] GPR00: 000000000000014f 00007fffe4904780 000000011dc16b98 
0000000000000026
[   26.466734] GPR04: 00007fffe4904808 0000000000000470 0000000000000000 
00007fffe49047f8
[   26.466734] GPR08: 000000000000000c 0000000000000000 0000000000000000 
0000000000000000
[   26.466734] GPR12: 0000000000000000 00007fff932ad2c0 0000000000000020 
00000001584e4310
[   26.466734] GPR16: 00000001585664c0 000000011ddb6b98 0000000000000000 
0000000000000000
[   26.466734] GPR20: 0000000156e56d20 000000015707ee80 00000001585664d0 
0000000000000018
[   26.466734] GPR24: 0000000000000000 000000011dd66b98 00007fffe4904808 
0000000000000026
[   26.466734] GPR28: 0000000000000470 0000000000000000 0000000000000000 
000000000000000c
[   26.466734] IRQMASK: 0
[   26.466734] GPR00: 000000000000014f 00007fffe4904780 000000011dc16b98 
0000000000000026
[   26.466734] GPR04: 00007fffe4904808 0000000000000470 0000000000000000 
00007fffe49047f8
[   26.466734] GPR08: 000000000000000c 0000000000000000 0000000000000000 
0000000000000000
[   26.466734] GPR12: 0000000000000000 00007fff932ad2c0 0000000000000020 
00000001584e4310
[   26.466734] GPR16: 00000001585664c0 000000011ddb6b98 0000000000000000 
0000000000000000
[   26.466734] GPR20: 0000000156e56d20 000000015707ee80 00000001585664d0 
0000000000000018
[   26.466734] GPR24: 0000000000000000 000000011dd66b98 00007fffe4904808 
0000000000000026
[   26.466734] GPR28: 0000000000000470 0000000000000000 0000000000000000 
000000000000000c
[   26.467050] NIP [00007fff92f507c4] 0x7fff92f507c4
[   26.467050] NIP [00007fff92f507c4] 0x7fff92f507c4
[   26.467073] LR [00007fff92f507c4] 0x7fff92f507c4
[   26.467073] LR [00007fff92f507c4] 0x7fff92f507c4
[   26.467096] ---- interrupt: 3000
[   26.467096] ---- interrupt: 3000
[   26.467112] Code: 7c7c1b78 3be0ffea 3b7b8d38 7f63db78 f8010010 
f821ff81 90a1002c e94d0c78 f9410048 39400000 f9210040 f8810038 
<eb5c0000> 48a8a2b9 60000000 e92d0908
[   26.467112] Code: 7c7c1b78 3be0ffea 3b7b8d38 7f63db78 f8010010 
f821ff81 90a1002c e94d0c78 f9410048 39400000 f9210040 f8810038 
<eb5c0000> 48a8a2b9 60000000 e92d0908
[   26.467187] ---[ end trace 0000000000000000 ]---
[   26.467187] ---[ end trace 0000000000000000 ]---
[   26.469587] pstore: backend (nvram) writing error (-1)
[   26.469587] pstore: backend (nvram) writing error (-1)



Git Bisect logs:


git bisect log
git bisect start
# status: waiting for both good and bad commits
# good: [e04c78d86a9699d136910cfc0bdcf01087e3267e] Linux 6.16-rc2
git bisect good e04c78d86a9699d136910cfc0bdcf01087e3267e
# status: waiting for bad commit, 1 good commit known
# bad: [2c923c845768a0f0e34b8161d70bc96525385782] Add linux-next 
specific files for 20250619
git bisect bad 2c923c845768a0f0e34b8161d70bc96525385782
# bad: [c567e1e73106808756096712e1e24ff0e55bc869] Merge branch 'main' of 
git://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git
git bisect bad c567e1e73106808756096712e1e24ff0e55bc869
# bad: [84f70e2114659fb2aef508a2dabe474b9500e1d0] Merge branch 
'xtensa-for-next' of git://github.com/jcmvbkbc/linux-xtensa.git
git bisect bad 84f70e2114659fb2aef508a2dabe474b9500e1d0
# bad: [8705d9293cf77242a2549cb2db0fac17ea8bc07b] Merge branch 
'mm-unstable' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm
git bisect bad 8705d9293cf77242a2549cb2db0fac17ea8bc07b
# bad: [70d02212789008ab8c7854eec7781442883a06ac] Merge branch 
'mtd/fixes' of git://git.kernel.org/pub/scm/linux/kernel/git/mtd/linux.git
git bisect bad 70d02212789008ab8c7854eec7781442883a06ac
# bad: [504e9fee35da228b1481cf7821e99118b78a396a] Merge branch 'fixes' 
of git://git.kernel.org/pub/scm/linux/kernel/git/s390/linux.git
git bisect bad 504e9fee35da228b1481cf7821e99118b78a396a
# good: [5adb635077d1b4bd65b183022775a59a378a9c00] Merge tag 
'selinux-pr-20250618' of 
git://git.kernel.org/pub/scm/linux/kernel/git/pcmoore/selinux
git bisect good 5adb635077d1b4bd65b183022775a59a378a9c00
# good: [75856c59ae536a369ac79c1b8f5f5c002a9f5c70] mm/hugetlb: remove 
unnecessary holding of hugetlb_lock
git bisect good 75856c59ae536a369ac79c1b8f5f5c002a9f5c70
# bad: [c918c63d5c13ac59fef9eb02b7688cb464fdb32c] Merge branch 'fixes' 
of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git
git bisect bad c918c63d5c13ac59fef9eb02b7688cb464fdb32c
# good: [4f24bfcc398eb77aa41fe1bb1621d8c2cca5368d] Merge tag 
'sched_ext-for-6.16-rc2-fixes' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tj/sched_ext
git bisect good 4f24bfcc398eb77aa41fe1bb1621d8c2cca5368d
# good: [74b4cc9b8780bfe8a3992c9ac0033bf22ac01f19] Merge tag 
'cgroup-for-6.16-rc2-fixes' of 
git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup
git bisect good 74b4cc9b8780bfe8a3992c9ac0033bf22ac01f19
# good: [336f36773aec57d6bb6e33e46f6121bca13d33a0] Merge branch 
'misc-6.16' into next-fixes
git bisect good 336f36773aec57d6bb6e33e46f6121bca13d33a0
# bad: [a9ea6b0629a5a91d11db9318fba45a2e058babb1] replace 
collect_mounts()/drop_collected_mounts() with safer variant
git bisect bad a9ea6b0629a5a91d11db9318fba45a2e058babb1
# first bad commit: [a9ea6b0629a5a91d11db9318fba45a2e058babb1] replace 
collect_mounts()/drop_collected_mounts() with safer variant



If you happen to fix this issue, please add below tag.


Reported-by: Venkat Rao Bagalkote <venkat88@...ux.ibm.com>



Regards,

Venkat.

Powered by blists - more mailing lists