linux-kernel - Re: Boot regression with bacf6b499e11 ("x86/mm: Use a struct to reduce parameters for SME PGD mapping") on top of -rc8

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAEJqkggx7NkzvB3DFkDjOYbEMNeeMKibtPFDPbrpo46rriua9A@mail.gmail.com>
Date:   Sat, 20 Jan 2018 13:01:36 +0100
From:   Gabriel C <nix.or.die@...il.com>
To:     Tom Lendacky <thomas.lendacky@....com>
Cc:     Laura Abbott <labbott@...hat.com>, Borislav Petkov <bp@...e.de>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Thomas Gleixner <tglx@...utronix.de>,
        Ingo Molnar <mingo@...nel.org>,
        Brijesh Singh <brijesh.singh@....com>, X86 ML <x86@...nel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: Boot regression with bacf6b499e11 ("x86/mm: Use a struct to
 reduce parameters for SME PGD mapping") on top of -rc8

2018-01-20 7:15 GMT+01:00 Tom Lendacky <thomas.lendacky@....com>:
> On 1/19/2018 11:25 PM, Gabriel C wrote:
>> 2018-01-20 5:02 GMT+01:00 Laura Abbott <labbott@...hat.com>:
>>> On 01/19/2018 06:23 PM, Gabriel C wrote:
>>>>
>>>> 2018-01-20 2:23 GMT+01:00 Laura Abbott <labbott@...hat.com>:
>>>>>
>>>>> Hi,
>>>>
>>>>
>>>> Hi ,
>>>>
>>>>>
>>>>> Fedora got multiple reports of an early bootup crash post -rc8.
>>>>> Bisection showed bacf6b499e11 ("x86/mm: Use a struct to reduce
>>>>> parameters for SME PGD mapping") . It doesn't revert cleanly
>>>>> but if I revert the few other changes in arch/x86/mm/mem_encrypt.c
>>>>> as well it boots up fine.
>>>>>
>>>>> Annoyingly, I can't seem to get any actual kernel logs even with
>>>>> earlyprintk. It just reboots immediately (triple fault?). This
>>>>> happens on both of my Lenovo machines and I can ask other reporters
>>>>> for details as well.
>>>>>
>>>>
>>>> I tested these patches on 2 Lenovo Ideapad both with Skylake CPUs
>>>> on a older dual Xeon box , on 2 Toshibas with AMD APUs , on a RYZEN box ,
>>>> on dual EPYC box .. ofc on EPYC with mem_encrypt=on on the Intel CPUs
>>>> disabled.
>>>>
>>>> Also tested on top 4.14.13 , 4.14.14 as well on top 4.15.0-rc7 and on
>>>> current master/rc8++ without to see something like this.
>>>>
>>>> Also we pushed these patches on 4.14.13/14 and didn't got any reports
>>>> about
>>>> something like this.
>>>>
>>>> What Lenovo boxes are these ? maybe I find one to reproduce.
>>>>
>>>>
>>>>> $ git bisect log
>>>>> # bad: [ec835f8104a21f4d4eeb9d316ee71d2b4a7f00de] Merge tag
>>>>> 'trace-v4.15-rc4-3' of
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/rostedt/linux-trace
>>>>> # good: [a8750ddca918032d6349adbf9a4b6555e7db20da] Linux 4.15-rc8
>>>>> git bisect start 'origin/master' 'v4.15-rc8'
>>>>> # bad: [79683f80e4f07dba13cc08d0ebcf5c7b0aa1bf68] Merge tag
>>>>> 'mmc-v4.15-rc2-3' of
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/ulfh/mmc
>>>>> git bisect bad 79683f80e4f07dba13cc08d0ebcf5c7b0aa1bf68
>>>>> # good: [161f72ed6dbe7fb176585091d3b797125d310399] Merge tag
>>>>> 'mac80211-for-davem-2018-01-15' of
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/jberg/mac80211
>>>>> git bisect good 161f72ed6dbe7fb176585091d3b797125d310399
>>>>> # good: [88dc7fca18001fd883e5ace775afa316b68c8f2c] Merge branch
>>>>> 'x86-pti-for-linus' of
>>>>> git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip
>>>>> git bisect good 88dc7fca18001fd883e5ace775afa316b68c8f2c
>>>>> # bad: [d47924417319e3b6a728c0b690f183e75bc2a702] x86/intel_rdt/cqm:
>>>>> Prevent
>>>>> use after free
>>>>> git bisect bad d47924417319e3b6a728c0b690f183e75bc2a702
>>>>> # good: [fc90ccfd286eabb05ec54521367df8663cf0bbbf] Revert "x86/apic:
>>>>> Remove
>>>>> init_bsp_APIC()"
>>>>> git bisect good fc90ccfd286eabb05ec54521367df8663cf0bbbf
>>>>> # bad: [bacf6b499e11760aef73a3bb5ce4e5eea74a3fd4] x86/mm: Use a struct to
>>>>> reduce parameters for SME PGD mapping
>>>>> git bisect bad bacf6b499e11760aef73a3bb5ce4e5eea74a3fd4
>>>>> # good: [1303880179e67c59e801429b7e5d0f6b21137d99] x86/mm: Clean up
>>>>> register
>>>>> saving in the __enc_copy() assembly code
>>>>> git bisect good 1303880179e67c59e801429b7e5d0f6b21137d99
>>>>> # first bad commit: [bacf6b499e11760aef73a3bb5ce4e5eea74a3fd4] x86/mm:
>>>>> Use a
>>>>> struct to reduce parameters for SME PGD mapping
>>>>>
>>>>>
>>>>> Configuration is at
>>>>>
>>>>> https://git.kernel.org/pub/scm/linux/kernel/git/jwboyer/fedora.git/commit/?h=rawhide
>>>>> Note that I do think this is something in the Fedora configuration
>>>>> because a generic "make defconfig" booted just fine.
>>>>
>>>>
>>>> But maybe some of the Fedora patches ?
>>>>
>>>> Can you try an kernel with the config but without any patches ?
>>>> Or a defconfig and just enable  CONFIG_AMD_MEM_ENCRYPT ?
>>>>
>>>
>>> The bisect was a vanilla kernel without Fedora patches.
>>
>> Ok . I did an build ( v4.15-rc8-225-g8dd903d2cf7b ) on my AMD
>> Workstation  ( EPYC CPU )
>
> I've tried multiple config combinations on my EPYC system and have not
> been able to reproduce this issue and have not had any boot issues with
> mem_encrypt=on or mem_encrypt=off.  I don't have access to a non-AMD box
> at the moment, but I'm really scratching my head on this one.
>
>>  with your 64bit config and one on the Ideapad ( Intel i7-6498DU ) ..
>> I disabled Selinux since I don't use it here and module  signing.
>>
>> Also with your config my serial setup won't work and the kernel hangs
>> but mem_encrypt=on/off works just fine.
>
> If your using the EPYC serial device and you haven't enabled legacy
> serial device support (if your BIOS supports that), then you're using
> it as a platform device.  The fedora config has not set the
> CONFIG_X86_AMD_PLATFORM_DEVICE setting so you won't get the module
> to load and give you serial output.
>
> I'm confused when you say the kernel hangs but mem_encrypt=on/off works
> just fine, can you explain that a bit more?
>

I have to remove console=ttyS.. etc from kernel command line to boot
but this is not about mem_encrypt  , since once removed the kernel works
just fine with mem_encrypt=on/off.

>>
>> Also I notice on the Workstation it takes forever to boot untill
>> 'DMA-API' reports out-of-memory
>> ( dunno how much memory it need but the box has 128GB of RAM )..
>
> I haven't seen this before.
>
> Thanks,
> Tom
>
>>
>>
>> Can you tell us your Lenovo models  please  ?
>>
>> Reagrds,
>>
>> Gabriel C
>>