linux-kernel - Re: [RFC] Avoid PIT SMP lockups

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <453404F6.5040202@vmware.com>
Date:	Mon, 16 Oct 2006 15:17:26 -0700
From:	Zachary Amsden <zach@...are.com>
To:	caglar@...dus.org.tr
Cc:	Gerd Hoffmann <kraxel@...e.de>, john stultz <johnstul@...ibm.com>,
	Andi Kleen <ak@...e.de>, lkml <linux-kernel@...r.kernel.org>
Subject: Re: [RFC] Avoid PIT SMP lockups

S.Çag(lar Onur wrote:
> 12 Eki 2006 Per 10:28 tarihinde, Gerd Hoffmann s,unlar? yazm?s,t?: 
>   
>> Try switching the vmware configuration to "other OS".  This turns off
>> os-specific binary patching.  The alternatives code might have broken
>> assumptions vmware does about the linux kernel code ...
>>     
>
> I did before, i tried these combinations
>
> * Guest Os: Linux, Version: Other Linux 2.6.x kernel
> * Guest Os: Linux, Version: Other Linux
> * Guest Os: Other, Version: Other
>
> all of them ends up with panic.
>   

We don't do any binary patching. In general, most of the guest OS types 
turn on various performance hints or activate fairly expensive 
workarounds to known OS bugs, but they don't make the system less 
correct by limiting the functionality available to the OS kernel.

It sounds to me like an interrupt is firing in the alternatives code. 
Perhaps some callee is doing a raw local_irq_enable from somewhere 
inside here, or nobody took care to disable interrupts in the first 
place. You need to have interrupts disabled here.

OMG! When did we start calling alternative_instructions from init with 
_after_ local_irq_enable()? This is totally wrong. Even on native 
hardware, you can be hit with a timer interrupt in this window where we 
are patching the kernel. It does not surprise me that you hit this 
repeatedly. It surprises me that other people don't! The kernel has 
managed to take enough time to get from the last timer IRQ (likely in 
lpj calibration, as you noted in the output) to the alternative patching 
point that 90% of the time, another timer IRQ is pending. This can 
happen on any machine, not just VMware, and I'm surprised it doesn't 
happen with other emulators as well.

It might only happen with SMP because the difficulty of getting good 
enough TSC / timer IRQ synchronization during boot increases 
exponentially with SMP configurations. And it might pass 10% of the time 
because you were lucky enough not to fire off another timer interrupt yet.

My nasty quick patch might not apply - the only tree I've got is a very 
hacked 2.6.18-rc6-mm1+local-patches thing, but the fix should be obvious 
enough.

Zach

View attachment "hotfix-alternatives-irq-safety.patch" of type "text/plain" (844 bytes)