linux-kernel - Re: S3 resume regression [1cf4f629d9d2 ("cpu/hotplug: Move online calls to hotplugged cpu")]

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160526183207.GX4329@intel.com>
Date:	Thu, 26 May 2016 21:32:07 +0300
From:	Ville Syrjälä <ville.syrjala@...ux.intel.com>
To:	"Rafael J. Wysocki" <rafael.j.wysocki@...el.com>
Cc:	Steven Rostedt <rostedt@...dmis.org>,
	Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
	Thomas Gleixner <tglx@...utronix.de>,
	linux-arch@...r.kernel.org, Rik van Riel <riel@...hat.com>,
	"Srivatsa S. Bhat" <srivatsa@....edu>,
	Peter Zijlstra <peterz@...radead.org>,
	Arjan van de Ven <arjan@...ux.intel.com>,
	Rusty Russell <rusty@...tcorp.com.au>,
	Oleg Nesterov <oleg@...hat.com>, Tejun Heo <tj@...nel.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Paul McKenney <paulmck@...ux.vnet.ibm.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Paul Turner <pjt@...gle.com>, linux-kernel@...r.kernel.org,
	rui.zhang@...el.com, len.brown@...el.com,
	Linux PM <linux-pm@...r.kernel.org>,
	Linux ACPI <linux-acpi@...r.kernel.org>
Subject: Re: S3 resume regression [1cf4f629d9d2 ("cpu/hotplug: Move online
 calls to hotplugged cpu")]

On Wed, May 18, 2016 at 10:24:24AM +0300, Ville Syrjälä wrote:
> On Wed, May 18, 2016 at 01:14:42AM +0200, Rafael J. Wysocki wrote:
> > On 5/16/2016 9:39 PM, Ville Syrjälä wrote:
> > > On Wed, May 11, 2016 at 04:34:06PM +0300, Ville Syrjälä wrote:
> > >> On Wed, May 11, 2016 at 08:44:45AM -0400, Steven Rostedt wrote:
> > >>> On Wed, 11 May 2016 15:21:16 +0300
> > >>> Ville Syrjälä <ville.syrjala@...ux.intel.com> wrote:
> > >>>
> > >>>> Yeah can't get anything from the machine at that point. netconsole
> > >>>> didn't help either, and no serial on this machine. And IIRC I've
> > >>>> tried ramoops on this thing in the past but unfortunately the memory
> > >>>> got cleared on reboot.
> > >>>>
> > >>> Can you look at the documentation in the kernel code at
> > >>>
> > >>> Documentation/power/basic-pm-debugging.txt And follow the procedures
> > >>> for testing suspend to RAM (although it requires mostly running the
> > >>> same tests as for hibernation suspending).
> > >>>
> > >>> You can also use the tool s2ram for this as well.
> > >>>
> > >>> See Documentation/power/s2ram.txt
> > >>>
> > >>> Perhaps this can give us a bit more light onto the problem.
> > >>>
> > >>> Basically the above does partial suspend and resume, and can pinpoint
> > >>> problem areas down to a more select location.
> > >> All the pm_test modes work fine. The only difference between them was
> > >> that 'platform' required me to manually wake up the machine (hitting a
> > >> key was sufficient), whereas the others woke up without help.
> > >>
> > >> pm_trace gave me
> > >> [    1.306633]   Magic number: 0:185:178
> > >> [    1.322880]   hash matches ../drivers/base/power/main.c:1070
> > >> [    1.339270] acpi device:0e: hash matches
> > >> [    1.355414]  platform: hash matches
> > >>
> > >> which is the TRACE_SUSPEND in __device_suspend_noirq(), so no help
> > >> there.
> > >>
> > >> I guess I could try to sprinkle more TRACE_RESUMEs around into some
> > >> early resume code. If anyone has good ideas where to put them it
> > >> might speed things up a bit.
> > > So I did a bunch of that and found that it gets stuck somewhere
> > > around executing the _WAK method:
> > > platform_resume_noirq
> > >   acpi_pm_finish
> > >    acpi_leave_sleep_state
> > >     acpi_hw_sleep_dispatch
> > >      acpi_hw_legacy_wake
> > >       acpi_hw_execute_sleep_method
> > >        acpi_evaluate_object
> > >         acpi_ns_evaluate
> > >          acpi_ps_execute_method
> > >           acpi_ps_parse_aml
> > >
> > > It also seesm that adding a few TRACE_RESUME()s or an msleep() right
> > > after enable_nonboot_cpus() can avoid the hang, sometimes.
> > >
> > > I've attached the DSDT in case anyone is interested in looking at it.
> > >
> > 
> > What if you comment out the execution of _WAK (line 318 of 
> > drivers/acpi/acpica/hwsleep.c in 4.6)?  Does that make any difference?
> 
> Indeed it does. Tried with acpi_idle and intel_idle, and both appear to
> resume just fine with that hack.
> 
> -       acpi_hw_execute_sleep_method(METHOD_PATHNAME__WAK, sleep_state);
> +       //acpi_hw_execute_sleep_method(METHOD_PATHNAME__WAK, sleep_state);
> +       printk(KERN_CRIT "skipping _WAK\n");

Continuing with my detective work a bit, I decided to hack the DSDT a
bit to see if I can narrow the it down further, and looks like I found
it on the first guess. The following change stops it from hanging.

@ -5056,7 +5056,7 @@
         If (LEqual (Arg0, 0x03))
         {
             Store (0x01, \SPNF)
-	    TRAP (0x46)
+	    //TRAP (0x46)
             P8XH (0x00, 0x03)
         }

So what does that do? Let's see:

    OperationRegion (IO_T, SystemIO, 0x0800, 0x10)
    Field (IO_T, ByteAcc, NoLock, Preserve)
    {
        Offset (0x08), 
        TRP0,   8
    }

    OperationRegion (GNVS, SystemMemory, 0x3F5E0C7C, 0x0200)
    Field (GNVS, AnyAcc, Lock, Preserve)
    {
        OSYS,   16, 
        SMIF,   8,
    ...

    Method (TRAP, 1, Serialized)
    {
        Store (Arg0, SMIF) /* \SMIF */
        Store (0x00, TRP0) /* \TRP0 */
        Return (SMIF) /* \SMIF */
    }

and a dump of the IOTR registers shows:

0x1e80: 0x0000fe01
0x1e84: 0x00020001
0x1e98: 0x000c0801
0x1e9c: 0x000200f0

which seems to be telling me that ports 0x800-0x80f and 
0xfe00-0xfe03 would trigger an SMI.

So the next question is how do the idle drivers and cpu hotplug
fit into this picture. Do we need to force the second HT into
a specific C state before the SMI or something?

-- 
Ville Syrjälä
Intel OTC