linux-kernel - [BUG] Guest OSes die simultaneously (bisected)

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <3d8f5987-e09c-4dd2-a9c0-8ba22c9e948a@paulmck-laptop>
Date: Wed, 3 Jan 2024 14:22:23 -0800
From: "Paul E. McKenney" <paulmck@...nel.org>
To: Like Xu <like.xu@...ux.intel.com>, Andi Kleen <ak@...ux.intel.com>,
	Kan Liang <kan.liang@...ux.intel.com>,
	Luwei Kang <luwei.kang@...el.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Paolo Bonzini <pbonzini@...hat.com>
Cc: linux-perf-users@...r.kernel.org, linux-kernel@...r.kernel.org,
	kvm@...r.kernel.org, Breno Leitao <leitao@...ian.org>,
	Arnaldo Carvalho de Melo <acme@...nel.org>,
	Ingo Molnar <mingo@...hat.com>
Subject: [BUG] Guest OSes die simultaneously (bisected)

Hello!

Since some time between v5.19 and v6.4, long-running rcutorture tests
would (rarely but intolerably often) have all guests on a given host die
simultaneously with something like an instruction fault or a segmentation
violation.

Each bisection step required 20 hosts running 10 hours each, and
this eventually fingered commit c59a1f106f5c ("KVM: x86/pmu: Add
IA32_PEBS_ENABLE MSR emulation for extended PEBS").  Although this commit
is certainly messing with things that could possibly cause all manner
of mischief, I don't immediately see a smoking gun.  Except that the
commit prior to this one is rock solid.

Just to make things a bit more exciting, bisection in mainline proved
to be problematic due to bugs of various kinds that hid this one.  I was
therefore forced to bisect among the commits backported to the internal
v5.19-based kernel, which fingered the backported version of the patch
called out above.

Please note that this is not (yet) an emergency.  I will just continue
to run rcutorture on v5.19-based hypervisors in the meantime.

Any suggestions for debugging or fixing?

							Thanx, Paul