linux-kernel - Re: [GIT PULL] PCI changes for v4.20

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Tue, 13 Nov 2018 19:47:58 +0000
From:   "Lendacky, Thomas" <Thomas.Lendacky@....com>
To:     Borislav Petkov <bp@...en8.de>, Ingo Molnar <mingo@...nel.org>,
        Bjorn Helgaas <helgaas@...nel.org>
CC:     Jonathan Cameron <Jonathan.Cameron@...wei.com>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        "linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        Lorenzo Pieralisi <lorenzo.pieralisi@....com>,
        Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
        Bart Van Assche <bvanassche@....org>,
        Jens Axboe <axboe@...nel.dk>,
        Thomas Gleixner <tglx@...utronix.de>,
        Peter Zijlstra <a.p.zijlstra@...llo.nl>
Subject: Re: [GIT PULL] PCI changes for v4.20

On 11/13/2018 08:41 AM, Lendacky, Thomas wrote:
> On 11/13/2018 04:20 AM, Borislav Petkov wrote:
>> On Tue, Nov 13, 2018 at 08:17:12AM +0100, Ingo Molnar wrote:
>>>
>>> * Bjorn Helgaas <helgaas@...nel.org> wrote:
>>>
>>>> PCI changes:
>>>>
>>>>   - Pay attention to device-specific _PXM node values (Jonathan Cameron)
>>>
>>> There's a new boot regression, my AMD ThreadRipper system (MSI X399 SLI 
>>> PLUS (MS-7B09)) hangs during early bootup, and I have bisected it down to 
>>> this commit:
>>>
>>>   bad7dcd94f39: ACPI/PCI: Pay attention to device-specific _PXM node values
>>>
>>> Reverting it solves the hang.
>>>
>>> Unfortunately there's no console output when it hangs, even with 
>>> earlyprintk. It just hangs after the "loading initrd" line.
>>>
>>> Config is an Ubuntu-ish config with PROVE_LOCKING=y and a few other debug 
>>> options.
>>>
>>> All my other testsystems boot fine with similar configs, so it's probably 
>>> something specific to this system.
>>
>> Lemme add Tom, he might have an idea.
> 
> I'm not seeing any issues on my EPYC system.  Let me see if I can locate a
> Threadripper system to test on.

Based upon the link that Bjorn referenced in another email, I was able to
re-create the problem by having my EPYC system return early from
acpi_numa_init() with a -ENOENT (skipping the SRAT table). This resulted
in the following GPF:

[   11.157840] general protection fault: 0000 [#1] SMP NOPTI
[   11.158785] CPU: 3 PID: 1 Comm: swapper/0 Not tainted 4.20.0-rc2-zp-linux #3
[   11.158785] Hardware name: ******
[   11.158785] RIP: 0010:get_partial_node.isra.76+0x33/0x2b0
[   11.158785] Code: 89 e5 41 57 41 56 41 55 41 54 53 48 83 e4 f0 48 83 c4 80 48 85 f6 48 89 7c 24 30 48 89 54 24 10 89 4c 24 0c 0f 84 d5 00 00 00 <48> 83 7e 08 00 0f 84 ca 00 00 00 48 89 f7 48 89 74 24 38 e8 95 5e
[   11.158785] RSP: 0018:ffffc900001078b0 EFLAGS: 00010002
[   11.158785] RAX: 0000000000000000 RBX: 0000000000000202 RCX: 00000000006080c0
[   11.158785] RDX: ffff889ffdae7150 RSI: 4c7a584873359cf2 RDI: ffff888107c07000
[   11.158785] RBP: ffffc90000107958 R08: ffff888107c07000 R09: 0000000000000001
[   11.158785] R10: 00000000006080c0 R11: 0000000000000002 R12: ffff889ffdae7140
[   11.158785] R13: ffff888107c07000 R14: ffff888107c07000 R15: 0000000000000002
[   11.158785] FS:  0000000000000000(0000) GS:ffff889ffdac0000(0000) knlGS:0000000000000000
[   11.158785] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   11.158785] CR2: 0000000000000000 CR3: 00008014bc20a000 CR4: 00000000003406e0
[   11.158785] Call Trace:
[   11.158785]  ? acpi_os_release_object+0xa/0x10
[   11.158785]  ? acpi_ds_result_pop+0xf8/0x10c
[   11.158785]  ? acpi_ds_create_operand+0x227/0x24e
[   11.158785]  ___slab_alloc+0x100/0x540
[   11.158785]  ? acpi_ds_create_operands+0x72/0xd7
[   11.158785]  ? alloc_desc+0x35/0x210
[   11.158785]  ? acpi_ns_check_object_type+0x123/0x1c0
[   11.158785]  ? alloc_desc+0x35/0x210
[   11.158785]  __slab_alloc+0x1c/0x33
[   11.158785]  kmem_cache_alloc_node_trace+0xac/0x210
[   11.158785]  alloc_desc+0x35/0x210
[   11.158785]  __irq_alloc_descs+0x1c4/0x230
[   11.158785]  __irq_domain_alloc_irqs+0x54/0x2e0
[   11.158785]  mp_map_pin_to_irq+0x2cf/0x330
[   11.158785]  acpi_register_gsi_ioapic+0x78/0x170
[   11.158785]  ? mmio_resource_enabled.part.0+0x60/0x60
[   11.158785]  acpi_pci_irq_enable+0xcd/0x280
[   11.158785]  ? mmio_resource_enabled.part.0+0x60/0x60
[   11.158785]  ? mmio_resource_enabled.part.0+0x60/0x60
[   11.158785]  do_pci_enable_device+0x5b/0x100
[   11.158785]  ? pci_bus_read_config_word+0x56/0x70
[   11.158785]  pci_enable_device_flags+0xe0/0x130
[   11.158785]  pci_enable_bridge+0x52/0x90
[   11.158785]  pci_enable_device_flags+0x8c/0x130
[   11.158785]  quirk_usb_early_handoff+0x63/0x6b0
[   11.158785]  ? bus_find_device+0x87/0xd0
[   11.158785]  ? mmio_resource_enabled.part.0+0x60/0x60
[   11.158785]  pci_fixup_device+0xe8/0x1a0
[   11.158785]  pci_apply_final_quirks+0x68/0x127
[   11.158785]  ? pci_proc_init+0x68/0x68
[   11.158785]  do_one_initcall+0x4b/0x1cb
[   11.158785]  ? init_setup+0x1b/0x28
[   11.158785]  kernel_init_freeable+0x1be/0x26b
[   11.158785]  ? loglevel+0x5b/0x5b
[   11.158785]  ? rest_init+0xb0/0xb0
[   11.158785]  kernel_init+0xa/0x110
[   11.158785]  ret_from_fork+0x22/0x40
[   11.158785] Modules linked in:
[   11.158785] ---[ end trace ba1c80a146740c8b ]---
[   11.158785] RIP: 0010:get_partial_node.isra.76+0x33/0x2b0
[   11.158785] Code: 89 e5 41 57 41 56 41 55 41 54 53 48 83 e4 f0 48 83 c4 80 48 85 f6 48 89 7c 24 30 48 89 54 24 10 89 4c 24 0c 0f 84 d5 00 00 00 <48> 83 7e 08 00 0f 84 ca 00 00 00 48 89 f7 48 89 74 24 38 e8 95 5e
[   11.158785] RSP: 0018:ffffc900001078b0 EFLAGS: 00010002
[   11.158785] RAX: 0000000000000000 RBX: 0000000000000202 RCX: 00000000006080c0
[   11.158785] RDX: ffff889ffdae7150 RSI: 4c7a584873359cf2 RDI: ffff888107c07000
[   11.158785] RBP: ffffc90000107958 R08: ffff888107c07000 R09: 0000000000000001
[   11.158785] R10: 00000000006080c0 R11: 0000000000000002 R12: ffff889ffdae7140
[   11.158785] R13: ffff888107c07000 R14: ffff888107c07000 R15: 0000000000000002
[   11.158785] FS:  0000000000000000(0000) GS:ffff889ffdac0000(0000) knlGS:0000000000000000
[   11.158785] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   11.158785] CR2: 0000000000000000 CR3: 00008014bc20a000 CR4: 00000000003406e0
[   11.158785] Kernel panic - not syncing: Fatal exception
[   11.158785] ---[ end Kernel panic - not syncing: Fatal exception ]---

In acpi_get_node(), if I replace "return acpi_map_pxm_to_node(pxm);" with
"return acpi_map_pxm_to_online_node(pxm);" then the system successfully
boots.  I'm just not sure if that should be the proper approach or if
NUMA_NO_NODE should be returned if the _PXM value is outside the defined
entries. 

I was also able to trigger this GPF by returning a bogus _PXM value on the
EPYC system that had a valid SRAT table.  So it definitely would be worth
validating the PXM value before returning it.

Thanks,
Tom

> 
> It seems very strange that the commit in question would cause a hang so
> early. Do you have a serial console hooked up for the earlyprintk? Is the
> serial port set up in legacy mode (e.g. 0x3f8 as opposed to being an MMIO
> device that would require a driver)?
> 
> Can you dump the ACPI tables / run them through iasl to see what the _PXM
> values are in the DSDT table?
> 
> Thanks,
> Tom
> 
>>