| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <aA3FL6e1HVAw1J+w@xsang-OptiPlex-9020>
Date: Sun, 27 Apr 2025 13:48:31 +0800
From: Oliver Sang <oliver.sang@...el.com>
To: Arnd Bergmann <arnd@...db.de>
CC: <oe-lkp@...ts.linux.dev>, kernel test robot <lkp@...el.com>,
<linux-kernel@...r.kernel.org>, Ingo Molnar <mingo@...nel.org>, "Linus
Torvalds" <torvalds@...ux-foundation.org>, John Stultz <jstultz@...gle.com>,
Thomas Gleixner <tglx@...utronix.de>, Stephen Boyd <sboyd@...nel.org>,
Borislav Petkov <bp@...en8.de>, Dave Hansen <dave.hansen@...ux.intel.com>,
<x86@...nel.org>, "H. Peter Anvin" <hpa@...or.com>, <oliver.sang@...el.com>
Subject: Re: [linus:master] [x86/cpu] f388f60ca9:
BUG:soft_lockup-CPU##stuck_for#s![swapper:#]
hi, Arnd,
On Thu, Apr 24, 2025 at 09:59:38AM +0200, Arnd Bergmann wrote:
> On Thu, Apr 24, 2025, at 04:12, Oliver Sang wrote:
> > On Tue, Apr 22, 2025 at 12:16:33PM +0200, Arnd Bergmann wrote:
>
> Cc: x86 and timekeeping maintainers, see
> https://lore.kernel.org/lkml/202504211553.3ba9400-lkp@intel.com/
> for the thread so far.
>
> >> > [ 721.016779][ C0] hardirqs last disabled at (159506):
> >> > sysvec_apic_timer_interrupt (arch/x86/kernel/apic/apic.c:1049)
> >> > [ 721.016779][ C0] softirqs last enabled at (159174): handle_softirqs
> >> > (kernel/softirq.c:408 kernel/softirq.c:589)
> >> > [ 721.016779][ C0] softirqs last disabled at (159159): __do_softirq
> >> > (kernel/softirq.c:596)
> >> > [ 721.016779][ C0] CPU: 0 UID: 0 PID: 1 Comm: swapper Not tainted
> >> > 6.14.0-rc3-00037-gf388f60ca904 #1
> >> > [ 721.016779][ C0] Hardware name: QEMU Standard PC (i440FX + PIIX,
> >> > 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
> >> > [ 721.016779][ C0] EIP: timekeeping_notify
> >> > (kernel/time/timekeeping.c:1522)
> >>
> >> Timekeeping code could be related, I see that CONFIG_X86_TSC
> >> is disabled for i486SX configurations, so even if a TSC is present
> >> in the emulated machine, it is not being used to measure time
> >> accurately.
> >>
> >> > -CONFIG_X86_CMPXCHG64=y
> >>
> >> This could be another issue, if there is code that relies on
> >> the cx8/cmpxchg8b feature to be used. Since this is a non-SMP
> >> kernel, this is less likely to be the cause of the problem.
> >
> > thanks a lot for all these details!
> >
> >>
> >> Can you try what happens when you enable the two options, either
> >> by changing CONFIG_M486SX to CONFIG_M586TSC, or with a patch
> >> like the one below? Note that CONFIG_X86_CMPXCHG64 recently
> >> got renamed to CONFIG_X86_CX8, but they are the exact same thing.
> >
> > I applied your patch directly upon f388f60ca9 (change for X86_CMPXCHG64
> > instead of X86_CX8 as you metnioned), commit id is
> > c1f7ef63239411313163a7b1bff654236f48351c
> >
> ...
> > by running same tests, now it backs to the clean status like
> > fc2d5cbe541032e7 (parent of f388f60ca9)
>
> Thanks for confirming. So a 486-targeted kernel still passes
> your tests on modern hardware if we force TSC and CX8 to
> be enabled, but the boot fails if the options are turned
> off in Kconfig (though available in emulated hardware).
>
> To be completely sure, you could re-run the same test with
> just one of these enabled, but I'm rather sure that the TSC
> is the root cause.
just FYI. we rerun the tests. if only enable X86_TSC, the config diff is
--- /pkg/linux/i386-randconfig-r071-20250410/gcc-12/f388f60ca9041a95c9b3f157d316ed7c8f297e44/.config 2025-04-15 15:41:17.009901645 +0800
+++ /pkg/linux/i386-randconfig-r071-20250410/gcc-12/801597ddaae3bdc15546df3d0eba6e9e4e157b8d/.config 2025-04-25 14:24:49.488257697 +0800
@@ -351,6 +351,7 @@ CONFIG_X86_F00F_BUG=y
CONFIG_X86_INVD_BUG=y
CONFIG_X86_ALIGNMENT_16=y
CONFIG_X86_INTEL_USERCOPY=y
+CONFIG_X86_TSC=y
CONFIG_X86_MINIMUM_CPU_FAMILY=4
CONFIG_IA32_FEAT_CTL=y
CONFIG_X86_VMX_FEATURE_NAMES=y
the various issues still exists:
fc2d5cbe541032e7 f388f60ca9041a95c9b3f157d31 801597ddaae3bdc15546df3d0eb
---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
:496 29% 145:494 31% 153:500 last_state.booting
:496 7% 35:494 14% 71:500 dmesg.BUG:kernel_hang_in_boot_stage
:496 0% :494 0% 1:500 dmesg.BUG:soft_lockup-CPU##stuck_for#s![kworker##:#]
:496 9% 45:494 5% 25:500 dmesg.BUG:soft_lockup-CPU##stuck_for#s![swapper:#]
:496 0% 1:494 0% :500 dmesg.EIP:__timer_delete_sync
:496 1% 5:494 1% 3:500 dmesg.EIP:_raw_spin_unlock_irq
:496 0% 2:494 0% 1:500 dmesg.EIP:_raw_spin_unlock_irqrestore
:496 0% 1:494 0% :500 dmesg.EIP:console_emit_next_record
:496 0% :494 0% 1:500 dmesg.EIP:finish_task_switch
:496 0% 1:494 1% 4:500 dmesg.EIP:handle_softirqs
:496 1% 3:494 1% 4:500 dmesg.EIP:lock_acquire
:496 0% 2:494 0% 2:500 dmesg.EIP:lock_release
:496 0% :494 0% 1:500 dmesg.EIP:preempt_schedule_thunk
:496 0% 1:494 0% :500 dmesg.EIP:queue_delayed_work_on
:496 0% :494 0% 1:500 dmesg.EIP:rmqueue_pcplist
:496 9% 45:494 5% 25:500 dmesg.EIP:timekeeping_notify
:496 3% 14:494 2% 8:500 dmesg.INFO:rcu_preempt_detected_stalls_on_CPUs/tasks
:496 0% :494 0% 1:500 dmesg.INFO:rcu_preempt_self-detected_stall_on_CPU
:496 6% 32:494 9% 46:500 dmesg.INFO:task_blocked_for_more_than#seconds
:496 9% 45:494 5% 26:500 dmesg.Kernel_panic-not_syncing:softlockup:hung_tasks
if only enable X86_CMPXCHG64:
--- /pkg/linux/i386-randconfig-r071-20250410/gcc-12/f388f60ca9041a95c9b3f157d316ed7c8f297e44/.config 2025-04-15 15:41:17.009901645 +0800
+++ /pkg/linux/i386-randconfig-r071-20250410/gcc-12/dce28f73d4220df77c7fd6c41e854f1ef0af1d02/.config 2025-04-25 14:30:26.838037407 +0800
@@ -351,7 +351,8 @@ CONFIG_X86_F00F_BUG=y
CONFIG_X86_INVD_BUG=y
CONFIG_X86_ALIGNMENT_16=y
CONFIG_X86_INTEL_USERCOPY=y
-CONFIG_X86_MINIMUM_CPU_FAMILY=4
+CONFIG_X86_CMPXCHG64=y
+CONFIG_X86_MINIMUM_CPU_FAMILY=5
CONFIG_IA32_FEAT_CTL=y
CONFIG_X86_VMX_FEATURE_NAMES=y
CONFIG_CPU_SUP_INTEL=y
@@ -5745,6 +5746,7 @@ CONFIG_GENERIC_NET_UTILS=y
# CONFIG_PRIME_NUMBERS is not set
CONFIG_RATIONAL=y
CONFIG_GENERIC_IOMAP=y
+CONFIG_ARCH_USE_CMPXCHG_LOCKREF=y
CONFIG_ARCH_HAS_FAST_MULTIPLIER=y
CONFIG_ARCH_USE_SYM_ANNOTATIONS=y
various issues gone:
fc2d5cbe541032e7 f388f60ca9041a95c9b3f157d31 dce28f73d4220df77c7fd6c41e8
---------------- --------------------------- ---------------------------
fail:runs %reproduction fail:runs %reproduction fail:runs
| | | | |
:496 29% 145:494 0% :500 last_state.booting
:496 7% 35:494 0% :500 dmesg.BUG:kernel_hang_in_boot_stage
:496 9% 45:494 0% :500 dmesg.BUG:soft_lockup-CPU##stuck_for#s![swapper:#]
:496 0% 1:494 0% :500 dmesg.EIP:__timer_delete_sync
:496 1% 5:494 0% :500 dmesg.EIP:_raw_spin_unlock_irq
:496 0% 2:494 0% :500 dmesg.EIP:_raw_spin_unlock_irqrestore
:496 0% 1:494 0% :500 dmesg.EIP:console_emit_next_record
:496 0% 1:494 0% :500 dmesg.EIP:handle_softirqs
:496 1% 3:494 0% :500 dmesg.EIP:lock_acquire
:496 0% 2:494 0% :500 dmesg.EIP:lock_release
:496 0% 1:494 0% :500 dmesg.EIP:queue_delayed_work_on
:496 9% 45:494 0% :500 dmesg.EIP:timekeeping_notify
:496 3% 14:494 0% :500 dmesg.INFO:rcu_preempt_detected_stalls_on_CPUs/tasks
:496 6% 32:494 0% :500 dmesg.INFO:task_blocked_for_more_than#seconds
:496 9% 45:494 0% :500 dmesg.Kernel_panic-not_syncing:softlockup:hung_tasks
> I tried reproducing the problem locally
> with your .config on a qemu/tcg emulation running on an
> arm64 host, but this seems to run fine, including the
> rcutorture tests.
>
> Comparing my results with your log file, I see that your
> crash happens while changing the clocksource:
>
> Your dmesg:
> [ 92.548514][ T1] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
> [ 721.016745][ C0] watchdog: BUG: soft lockup - CPU#0 stuck for 626s! [swapper:1]
>
> My dmesg:
> [ 1.154511][ T1] hpet0: 3 comparators, 64-bit 100.000000 MHz counter
> [ 1.157896][ T1] clocksource: Switched to clocksource tsc-early
>
> There are also clearly some differences between TCG and KVM in
> the handling of TSC, e.g. I get this warning from qemu itself
> for the SandyBridge CPU:
>
> qemu-system-i386: warning: TCG doesn't support requested feature: CPUID.01H:ECX.tsc-deadline [bit 24]
>
> I tried a few other variations, including KVM on an x86 laptop
> (using kvmclock or tsc-early clocksource), but none of them failed
> the way yours did.
>
> Arnd
>
Powered by blists - more mailing lists