netdev - Re: [PATCH v3 01/11] xen/manage: keep track of the on-going suspend mode

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20200930212944.GA3138@dev-dsk-anchalag-2a-9c2d1d96.us-west-2.amazon.com>
Date:   Wed, 30 Sep 2020 21:29:44 +0000
From:   Anchal Agarwal <anchalag@...zon.com>
To:     <boris.ostrovsky@...cle.com>
CC:     <tglx@...utronix.de>, <mingo@...hat.com>, <bp@...en8.de>,
        <hpa@...or.com>, <x86@...nel.org>, <jgross@...e.com>,
        <linux-pm@...r.kernel.org>, <linux-mm@...ck.org>,
        <kamatam@...zon.com>, <sstabellini@...nel.org>,
        <konrad.wilk@...cle.com>, <roger.pau@...rix.com>,
        <axboe@...nel.dk>, <davem@...emloft.net>, <rjw@...ysocki.net>,
        <len.brown@...el.com>, <pavel@....cz>, <peterz@...radead.org>,
        <eduval@...zon.com>, <sblbir@...zon.com>,
        <xen-devel@...ts.xenproject.org>, <vkuznets@...hat.com>,
        <netdev@...r.kernel.org>, <linux-kernel@...r.kernel.org>,
        <dwmw@...zon.co.uk>, <benh@...nel.crashing.org>
Subject: Re: [PATCH v3 01/11] xen/manage: keep track of the on-going suspend mode

On Mon, Sep 28, 2020 at 02:49:56PM -0400, boris.ostrovsky@...cle.com wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> On 9/25/20 6:28 PM, Anchal Agarwal wrote:
> > On Fri, Sep 25, 2020 at 04:02:58PM -0400, boris.ostrovsky@...cle.com wrote:
> >> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> >>
> >>
> >>
> >> On 9/25/20 3:04 PM, Anchal Agarwal wrote:
> >>> On Tue, Sep 22, 2020 at 11:17:36PM +0000, Anchal Agarwal wrote:
> >>>> On Tue, Sep 22, 2020 at 12:18:05PM -0400, boris.ostrovsky@...cle.com wrote:
> >>>>> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 9/21/20 5:54 PM, Anchal Agarwal wrote:
> 
> >>>>> Also, wrt KASLR stuff, that issue is still seen sometimes but I haven't had
> >>>>> bandwidth to dive deep into the issue and fix it.
> >>
> >> So what's the plan there? You first mentioned this issue early this year and judged by your response it is not clear whether you will ever spend time looking at it.
> >>
> > I do want to fix it and did do some debugging earlier this year just haven't
> > gotten back to it. Also, wanted to understand if the issue is a blocker to this
> > series?
> 
> 
> Integrating code with known bugs is less than ideal.
> 
So for this series to be accepted, KASLR needs to be fixed along with other
comments of course? 
> 
> 3% failure for this feature seems to be a manageable number from the reproducability perspective --- you should be able to script this and each iteration should take way under a minute, no?
> 
>
Yes it should be doable. The % is not constant here that's the max I have seen.
Also, if at worse it takes a min per run and I have to run 2000-3000 runs to
produce failure that will still be slower. I have to dig in to see if I can find
a better way. 

> > I had some theories when debugging around this like if the random base address picked by kaslr for the
> > resuming kernel mismatches the suspended kernel and just jogging my memory, I didn't find that as the case.
> > Another hunch was if physical address of registered vcpu info at boot is different from what suspended kernel
> > has and that can cause CPU's to get stuck when coming online.
> 
> 
> I'd think if this were the case you'd have 100% failure rate. And we are also re-registering vcpu info on xen restore and I am not aware of any failures due to KASLR.
> 
What I meant there wrt VCPU info was that VCPU info is not unregistered during hibernation,
so Xen still remembers the old physical addresses for the VCPU information, created by the
booting kernel. But since the hibernation kernel may have different physical
addresses for VCPU info and if mismatch happens, it may cause issues with resume. 
During hibernation, the VCPU info register hypercall is not invoked again.
> 
> > The issue was only
> > reproducible 3% of the time out of 3000 runs hence its hard to just reproduce this.
> >
> > Moreover, I also wanted to get an insight on if hibernation works correctly with KASLR
> > generally and its only Xen causing the issue?
> 
> 
> With KASLR being on by default I'd be surprised if it didn't.
>
Thant makes it xen specific then. Also, I have not seen the issue on KVM based
instances.
> 
> -boris
> 
- Anchal