linux-kernel - Another preempt folding issue?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <52FA6D4B.7020709@canonical.com>
Date:	Tue, 11 Feb 2014 19:34:51 +0100
From:	Stefan Bader <stefan.bader@...onical.com>
To:	Peter Zijlstra <peterz@...radead.org>,
	Paolo Bonzini <pbonzini@...hat.com>
CC:	Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
	kvm@...r.kernel.org
Subject: Another preempt folding issue?

Hi Peter,

I am currently looking at a weird issue that manifest itself when trying to run
kvm enabled qemu on a i386 host (v3.13 kernel, oh and potentially important the
cpu is 64bit capable, so qemu-system-x86_64 is called). Sooner or later this
causes softlockup messages on the host. I tracked this down to __vcpu_run in
arch/x86/kvm/x86.c which does a loop which in that case never seems to make
progress or exit.

What I found is that vcpu_enter_guest will exit quickly without causing the loop
to exit when need_resched() is true. Looking at a crash dump I took, this was
the case (thread_info->flags had TIF_NEED_RESCHED set). So after immediately
returning __vcpu_run has the following code:

        if (need_resched()) {
                srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx);
                kvm_resched(vcpu); // now cond_resched();
                vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
        }

The kvm_resched basically would end up doing a cond_resched() which now checks
preempt_count() to be 0. If that is zero it will do the reschedule, otherwise it
just does nothing. Looking at the percpu variables in the dump, I saw that
the preempt_count was 0x8000000 (actually it was 0x80110000 but that was me
triggering the kexec crashdump with sysrq-c).

I saw that there have been some changes in the upstream kernel and have picked
the following:
1) x86, acpi, idle: Restructure the mwait idle routines
2) x86, idle: Use static_cpu_has() for CLFLUSH workaround, add barriers
3) sched/preempt: Fix up missed PREEMPT_NEED_RESCHED folding
4) sched/preempt/x86: Fix voluntary preempt for x86

Patch 1) and 2) as dependencies of 3) (to get the mwait function correct and to
the other file). Finally 4) is fixing up 3). [maybe worth suggesting to do for
3.13.y stable].

Still, with all those I got the softlockup. Since I knew from the dump info that
something is wrong with the folding, I made the pragmatic approach and added the
following:

        if (need_resched()) {
                srcu_read_unlock(&kvm->srcu, vcpu->srcu_idx);
+               preempt_fold_need_resched();
                kvm_resched(vcpu); // now cond_resched();
                vcpu->srcu_idx = srcu_read_lock(&kvm->srcu);
        }

And this lets the kvm guest run without the softlockups! However I am less than
convinced that this is the right thing to do. Somehow something done when
converting the preempt_count into percpu has caused at least the i386 side to
get into this mess (as there has not been any whining about 64bit). Just fail to
see what.

-Stefan

Download attachment "signature.asc" of type "application/pgp-signature" (902 bytes)