linux-kernel - xen: IPI interrupts not resumed early enough on suspend/resume

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <1317654626.21903.72.camel@zakaz.uk.xensource.com>
Date:	Mon, 3 Oct 2011 16:10:26 +0100
From:	Ian Campbell <Ian.Campbell@...rix.com>
To:	Thomas Gleixner <tglx@...utronix.de>,
	Jeremy Fitzhardinge <Jeremy.Fitzhardinge@...rix.com>,
	Konrad Rzeszutek Wilk <konrad.wilk@...cle.com>
CC:	xen-devel <xen-devel@...ts.xensource.com>,
	linux-kernel <linux-kernel@...r.kernel.org>
Subject: xen: IPI interrupts not resumed early enough on suspend/resume

Hi Thomas,

Recently I've been chasing an issue where a Xen guest will fail to
resume about 1 time in 100. I eventually managed to bisect this back to
676dc3cf5bc3 "xen: Use IRQF_FORCE_RESUME".

The Xen suspend procedure (drivers/xen/manage.c:do_suspend()) is roughly
(I've omitted some uninteresting parts) as follows:
  dpm_suspend_start()
  dpm_suspend_noirq()
  stop_machine()
   -> xen_suspend()
        syscore_suspend()
        HYPERVISOR_suspend() /* Hypercall, returns on resume */
        xen_irq_resume() /* Re-establishes evtchn<->irq bindings */
        syscore_resume()
  dpm_resume_noirq()
  dpm_resume_end()

The resume process appears to be coming to a halt at the end of the
stop_machine invocation of xen_suspend(), i.e. after syscore_resume()
but before dpm_resume_noirq().

Looking at the stack traces of all VCPUs when this happens it appears
that they are all idle, which suggests we are missing an event to cause
a reschedule out of the stop_machine thread back into the suspending
thread.

One of the effects of 676dc3cf5bc3 was to move the unmasking of the
timer and IPI interrupts from xen_irq_resume() (i.e. within the
stop_machine region) to dpm_resume_noirq() (i.e. outside the
stop_machine region). Since the IPI interrupts includes the reschedule
IPI I rather suspect that is the reason for the problem. I added a hack
to unmask the reched* IPIs at xen_irq_resume() time and so far it seems
to fix things, which backs up my gut feeling.

I can see a few options for how I might go about solving this in a
non-hacky way, which approach do you think would be preferable:

      * Add "IRQF_RESUME_EARLY", driven from syscore_resume, and use it
        for these interrupts.
      * register syscore ops for the Xen event channel subsystem to
        unmask the IPIs earlier (would probably look a lot like the code
        removed by 676dc3cf5bc3).
      * add syscore_ops to Xen smp subsystem to unmask the specific IPIs
        (which it binds at start of day) earlier.
      * push dpm_(suspend|resume)_noirq down into stop machine region
      * use something other than stop_machine to quiesce system and move
        to cpu0 for suspend (doesn't seem sensible to reproduce that
        functionality).

Routing IPIs through the regular IRQ path seems a little bit unusual but
it looks like powerpc does something similar in smp_request_message_ipi
and mpic_request_ipis and that code uses the syscore approach. Does
applying that here too seem sane?

Any preference / advice?

Thanks,
Ian.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/