linux-kernel - Re: Are jump labels broken on 6.11-rc1?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20240805143522.GA623936@frogsfrogsfrogs>
Date: Mon, 5 Aug 2024 07:35:22 -0700
From: "Darrick J. Wong" <djwong@...nel.org>
To: Peter Zijlstra <peterz@...radead.org>
Cc: Chandan Babu R <chandanbabu@...nel.org>,
	Matthew Wilcox <willy@...radead.org>,
	xfs <linux-xfs@...r.kernel.org>,
	linux-fsdevel <linux-fsdevel@...r.kernel.org>,
	linux-kernel <linux-kernel@...r.kernel.org>, x86@...nel.org,
	tglx@...utronix.de
Subject: Re: Are jump labels broken on 6.11-rc1?

On Wed, Jul 31, 2024 at 12:55:57PM +0200, Peter Zijlstra wrote:
> On Tue, Jul 30, 2024 at 10:33:41PM -0700, Darrick J. Wong wrote:
> 
> > Sooooo... it turns out that somehow your patch got mismerged on the
> > first go-round, and that worked.  The second time, there was no
> > mismerge, which mean that the wrong atomic_cmpxchg() callsite was
> > tested.
> > 
> > Looking back at the mismerge, it actually changed
> > __static_key_slow_dec_cpuslocked, which had in 6.10:
> > 
> > 	if (atomic_dec_and_test(&key->enabled))
> > 		jump_label_update(key);
> > 
> > Decrement, then return true if the value was set to zero.  With the 6.11
> > code, it looks like we want to exchange a 1 with a 0, and act only if
> > the previous value had been 1.
> > 
> > So perhaps we really want this change?  I'll send it out to the fleet
> > and we'll see what it reports tomorrow morning.
> 
> Bah yes, I missed we had it twice. Definitely both sites want this.
> 
> I'll tentatively merge the below patch in tip/locking/urgent. I can
> rebase if there is need.

Hi Peter,

This morning, I noticed the splat below with -rc2.

WARNING: CPU: 0 PID: 8578 at kernel/jump_label.c:295 __static_key_slow_dec_cpuslocked.part.0+0x50/0x60

Line 295 is the else branch of this code:

	if (atomic_cmpxchg(&key->enabled, 1, 0) == 1)
		jump_label_update(key);
	else
		WARN_ON_ONCE(!static_key_slow_try_dec(key));

Apparently static_key_slow_try_dec returned false?  Looking at that
function, I suppose the atomic_read of key->enabled returned 0, since it
didn't trigger the "WARN_ON_ONCE(v < 0)" code.  Does that mean the value
must have dropped from positive N to 0 without anyone ever taking the
jump_label_mutex?

Unfortunately I'm a little too covfid-brained to figure this out today.
:(

--D

[  104.985250] run fstests xfs/1877 at 2024-08-04 15:46:00
[  107.669818] XFS (sda4): EXPERIMENTAL exchange-range feature enabled. Use at your own risk!
[  107.672983] XFS (sda4): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
[  107.676466] XFS (sda4): Mounting V5 Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[  107.732210] XFS (sda4): Ending clean mount
[  107.744657] XFS (sda4): EXPERIMENTAL online scrub feature in use. Use at your own risk!
[  120.562636] Direct I/O collision with buffered writes! File: d2d4/d50c/d4a2/fe7 Comm: fsstress
[  132.078284] Direct I/O collision with buffered writes! File: d820/d6b9/d6f5/fbcc Comm: fsstress
[  134.261151] Direct I/O collision with buffered writes! File: d13e/d261/d896/fc9e Comm: fsstress
[  172.025695] Direct I/O collision with buffered writes! File: rt/p2/d1/f194c Comm: fsstress
[  238.690588] Direct I/O collision with buffered writes! File: dca1/dd3f/d2346/f26e3 Comm: fsstress
[  374.677179] Direct I/O collision with buffered writes! File: de70/dd75/de74/f2f38 Comm: fsstress
[  399.084085] Direct I/O collision with buffered writes! File: dd50/d1584/d1f9/f1e65 Comm: fsstress
[  547.895266] ------------[ cut here ]------------
[  547.905274] WARNING: CPU: 0 PID: 8578 at kernel/jump_label.c:295 __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  547.914707] Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_set_hash_ip ip_set_hash_net xt_set nft_compat ip_set_hash_mac ip_set nf_tables libcrc32c nfnetlink bfq sha512_ssse3 sha512_generic pvpanic_mmio sha256_ssse3 pvpanic sch_fq_codel configfs fuse ip_tables x_tables overlay nfsv4 af_packet
[  547.934623] CPU: 0 UID: 0 PID: 8578 Comm: xfs_scrub Not tainted 6.11.0-rc2-djwx #rc2 d9817e54d8a35981d261570492175bf5e1b3bc11
[  547.941392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20171121_152543-x86-ol7-builder-01.us.oracle.com-4.el7.1 04/01/2014
[  547.948411] RIP: 0010:__static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  547.958769] Code: 74 16 e8 a3 f7 ff ff 84 c0 74 1f 5b 48 c7 c7 60 96 13 82 e9 82 15 6f 00 e8 dd fb ff ff 5b 48 c7 c7 60 96 13 82 e9 70 15 6f 00 <0f> 0b eb dd 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 55
[  547.990804] RSP: 0018:ffffc9000a033c58 EFLAGS: 00010246
[  547.996006] RAX: 0000000000000000 RBX: ffffffffa0755a40 RCX: 0000000000000000
[  548.005476] RDX: 0000000000000000 RSI: 000000000000011b RDI: ffffffffa0755a40
[  548.020780] RBP: 0000000000000001 R08: ffffe8ffffc1d808 R09: ffffe8ffffc1d820
[  548.022169] R10: ffffc9000a033b28 R11: 0000000092a11c8d R12: 00000000ffffffa1
[  548.023686] R13: ffff888101deca00 R14: ffff888102b5d000 R15: 0000007f8ffa83f7
[  548.031184] FS:  00007f41b1000680(0000) GS:ffff88842d000000(0000) knlGS:0000000000000000
[  548.039927] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  548.044941] CR2: 00007f41842b43a8 CR3: 00000002d0745000 CR4: 00000000003506f0
[  548.049473] Call Trace:
[  548.050026]  <TASK>
[  548.050543]  ? __warn+0x7c/0x120
[  548.051235]  ? __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  548.053966]  ? report_bug+0x1a7/0x210
[  548.059439]  ? handle_bug+0x3c/0x60
[  548.063914]  ? exc_invalid_op+0x13/0x60
[  548.070776]  ? asm_exc_invalid_op+0x16/0x20
[  548.074709]  ? __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  548.078116]  ? __static_key_slow_dec_cpuslocked.part.0+0x2d/0x60
[  548.081773]  static_key_slow_dec+0x3e/0x60
[  548.084107]  xchk_teardown+0x1a2/0x1d0 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.089682]  xfs_scrub_metadata+0x448/0x5c0 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.098202]  xfs_ioc_scrubv_metadata+0x389/0x550 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.101209]  xfs_file_ioctl+0x8f0/0xe80 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.110146]  ? preempt_count_add+0x4a/0xa0
[  548.116282]  ? up_write+0x64/0x180
[  548.117009]  ? shmem_file_write_iter+0x5a/0x90
[  548.117987]  ? preempt_count_add+0x4a/0xa0
[  548.122761]  ? vfs_write+0x3a2/0x4a0
[  548.125244]  __x64_sys_ioctl+0x8a/0xb0
[  548.125987]  do_syscall_64+0x47/0x100
[  548.126895]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  548.128056] RIP: 0033:0x7f41b491ec5b
[  548.128874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[  548.133010] RSP: 002b:00007f41b0fff4c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  548.134754] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f41b491ec5b
[  548.136731] RDX: 00007f41b0fff610 RSI: 00000000c0285840 RDI: 0000000000000004
[  548.138545] RBP: 00007f41b0fff610 R08: 0000000000000000 R09: 0000000000000064
[  548.140068] R10: 00007f41b0fff235 R11: 0000000000000246 R12: 00007ffe4738a3b0
[  548.141605] R13: 000000000000001d R14: 00007ffe4738a558 R15: 00007f418824e4e0
[  548.143161]  </TASK>
[  548.143659] ---[ end trace 0000000000000000 ]---
[  564.628369] Direct I/O collision with buffered writes! File: d23/d2b9/db54/f14fc Comm: fsstress
[  677.002836] u16:0 (11) used greatest stack depth: 11408 bytes left
[  706.851715] Direct I/O collision with buffered writes! File: d2ee3/da3b/d3258/f255e Comm: fsstress
[  853.054101] Direct I/O collision with buffered writes! File: d1b4b/d2ac/d8b5/f13f8 Comm: fsstress
[  895.426556] Direct I/O collision with buffered writes! File: d747b/d577c/d5893/f6014 Comm: fsstress
[ 1037.454921] u16:5 (67) used greatest stack depth: 11016 bytes left
[ 1064.482959] Direct I/O collision with buffered writes! File: d60b4/d6e91/d6852/f6a87 Comm: fsstress
[ 1159.694683] XFS (sda3): Unmounting Filesystem eef3561a-2a86-49c8-9aa1-5024529cd3e7
[ 1170.139417] XFS (sda4): Unmounting Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[ 1179.548273] XFS (sda4): EXPERIMENTAL exchange-range feature enabled. Use at your own risk!
[ 1179.553727] XFS (sda4): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
[ 1179.558023] XFS (sda4): Mounting V5 Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[ 1179.590448] XFS (sda4): Ending clean mount
[ 1179.600826] XFS (sda4): Unmounting Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d


[  104.985250] run fstests xfs/1877 at 2024-08-04 15:46:00
[  107.669818] XFS (sda4): EXPERIMENTAL exchange-range feature enabled. Use at your own risk!
[  107.672983] XFS (sda4): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
[  107.676466] XFS (sda4): Mounting V5 Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[  107.732210] XFS (sda4): Ending clean mount
[  107.744657] XFS (sda4): EXPERIMENTAL online scrub feature in use. Use at your own risk!
[  120.562636] Direct I/O collision with buffered writes! File: d2d4/d50c/d4a2/fe7 Comm: fsstress
[  132.078284] Direct I/O collision with buffered writes! File: d820/d6b9/d6f5/fbcc Comm: fsstress
[  134.261151] Direct I/O collision with buffered writes! File: d13e/d261/d896/fc9e Comm: fsstress
[  172.025695] Direct I/O collision with buffered writes! File: rt/p2/d1/f194c Comm: fsstress
[  238.690588] Direct I/O collision with buffered writes! File: dca1/dd3f/d2346/f26e3 Comm: fsstress
[  374.677179] Direct I/O collision with buffered writes! File: de70/dd75/de74/f2f38 Comm: fsstress
[  399.084085] Direct I/O collision with buffered writes! File: dd50/d1584/d1f9/f1e65 Comm: fsstress
[  547.895266] ------------[ cut here ]------------
[  547.905274] WARNING: CPU: 0 PID: 8578 at kernel/jump_label.c:295 __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  547.914707] Modules linked in: xfs rpcsec_gss_krb5 auth_rpcgss nft_chain_nat xt_REDIRECT nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 ip6t_REJECT nf_reject_ipv6 ipt_REJECT nf_reject_ipv4 xt_tcpudp ip_set_hash_ip ip_set_hash_net xt_set nft_compat ip_set_hash_mac ip_set nf_tables libcrc32c nfnetlink bfq sha512_ssse3 sha512_generic pvpanic_mmio sha256_ssse3 pvpanic sch_fq_codel configfs fuse ip_tables x_tables overlay nfsv4 af_packet
[  547.934623] CPU: 0 UID: 0 PID: 8578 Comm: xfs_scrub Not tainted 6.11.0-rc2-djwx #rc2 d9817e54d8a35981d261570492175bf5e1b3bc11
[  547.941392] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS ?-20171121_152543-x86-ol7-builder-01.us.oracle.com-4.el7.1 04/01/2014
[  547.948411] RIP: 0010:__static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  547.958769] Code: 74 16 e8 a3 f7 ff ff 84 c0 74 1f 5b 48 c7 c7 60 96 13 82 e9 82 15 6f 00 e8 dd fb ff ff 5b 48 c7 c7 60 96 13 82 e9 70 15 6f 00 <0f> 0b eb dd 66 66 2e 0f 1f 84 00 00 00 00 00 90 0f 1f 44 00 00 55
[  547.990804] RSP: 0018:ffffc9000a033c58 EFLAGS: 00010246
[  547.996006] RAX: 0000000000000000 RBX: ffffffffa0755a40 RCX: 0000000000000000
[  548.005476] RDX: 0000000000000000 RSI: 000000000000011b RDI: ffffffffa0755a40
[  548.020780] RBP: 0000000000000001 R08: ffffe8ffffc1d808 R09: ffffe8ffffc1d820
[  548.022169] R10: ffffc9000a033b28 R11: 0000000092a11c8d R12: 00000000ffffffa1
[  548.023686] R13: ffff888101deca00 R14: ffff888102b5d000 R15: 0000007f8ffa83f7
[  548.031184] FS:  00007f41b1000680(0000) GS:ffff88842d000000(0000) knlGS:0000000000000000
[  548.039927] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  548.044941] CR2: 00007f41842b43a8 CR3: 00000002d0745000 CR4: 00000000003506f0
[  548.049473] Call Trace:
[  548.050026]  <TASK>
[  548.050543]  ? __warn+0x7c/0x120
[  548.051235]  ? __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  548.053966]  ? report_bug+0x1a7/0x210
[  548.059439]  ? handle_bug+0x3c/0x60
[  548.063914]  ? exc_invalid_op+0x13/0x60
[  548.070776]  ? asm_exc_invalid_op+0x16/0x20
[  548.074709]  ? __static_key_slow_dec_cpuslocked.part.0+0x50/0x60
[  548.078116]  ? __static_key_slow_dec_cpuslocked.part.0+0x2d/0x60
[  548.081773]  static_key_slow_dec+0x3e/0x60
[  548.084107]  xchk_teardown+0x1a2/0x1d0 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.089682]  xfs_scrub_metadata+0x448/0x5c0 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.098202]  xfs_ioc_scrubv_metadata+0x389/0x550 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.101209]  xfs_file_ioctl+0x8f0/0xe80 [xfs 6c824b8f28c8b3bc861384b1edb6577088bd4dda]
[  548.110146]  ? preempt_count_add+0x4a/0xa0
[  548.116282]  ? up_write+0x64/0x180
[  548.117009]  ? shmem_file_write_iter+0x5a/0x90
[  548.117987]  ? preempt_count_add+0x4a/0xa0
[  548.122761]  ? vfs_write+0x3a2/0x4a0
[  548.125244]  __x64_sys_ioctl+0x8a/0xb0
[  548.125987]  do_syscall_64+0x47/0x100
[  548.126895]  entry_SYSCALL_64_after_hwframe+0x4b/0x53
[  548.128056] RIP: 0033:0x7f41b491ec5b
[  548.128874] Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1c 48 8b 44 24 18 64 48 2b 04 25 28 00 00
[  548.133010] RSP: 002b:00007f41b0fff4c0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
[  548.134754] RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f41b491ec5b
[  548.136731] RDX: 00007f41b0fff610 RSI: 00000000c0285840 RDI: 0000000000000004
[  548.138545] RBP: 00007f41b0fff610 R08: 0000000000000000 R09: 0000000000000064
[  548.140068] R10: 00007f41b0fff235 R11: 0000000000000246 R12: 00007ffe4738a3b0
[  548.141605] R13: 000000000000001d R14: 00007ffe4738a558 R15: 00007f418824e4e0
[  548.143161]  </TASK>
[  548.143659] ---[ end trace 0000000000000000 ]---
[  564.628369] Direct I/O collision with buffered writes! File: d23/d2b9/db54/f14fc Comm: fsstress
[  677.002836] u16:0 (11) used greatest stack depth: 11408 bytes left
[  706.851715] Direct I/O collision with buffered writes! File: d2ee3/da3b/d3258/f255e Comm: fsstress
[  853.054101] Direct I/O collision with buffered writes! File: d1b4b/d2ac/d8b5/f13f8 Comm: fsstress
[  895.426556] Direct I/O collision with buffered writes! File: d747b/d577c/d5893/f6014 Comm: fsstress
[ 1037.454921] u16:5 (67) used greatest stack depth: 11016 bytes left
[ 1064.482959] Direct I/O collision with buffered writes! File: d60b4/d6e91/d6852/f6a87 Comm: fsstress
[ 1159.694683] XFS (sda3): Unmounting Filesystem eef3561a-2a86-49c8-9aa1-5024529cd3e7
[ 1170.139417] XFS (sda4): Unmounting Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[ 1179.548273] XFS (sda4): EXPERIMENTAL exchange-range feature enabled. Use at your own risk!
[ 1179.553727] XFS (sda4): EXPERIMENTAL parent pointer feature enabled. Use at your own risk!
[ 1179.558023] XFS (sda4): Mounting V5 Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d
[ 1179.590448] XFS (sda4): Ending clean mount
[ 1179.600826] XFS (sda4): Unmounting Filesystem e1e9e96c-0d94-4a7e-8405-86e4d16d331d