Message-ID: <40b2da4a-a205-3cf2-0c78-c94c28b2d3f4@bell.net>
Date:   Thu, 16 Feb 2023 13:59:11 -0500
From:   Matt Fagnani <matt.fagnani@...l.net>
To:     Vasant Hegde <vasant.hegde@....com>,
        Felix Kuehling <felix.kuehling@....com>,
        Bjorn Helgaas <helgaas@...nel.org>,
        Baolu Lu <baolu.lu@...ux.intel.com>,
        "Huang, Shimmer" <Shimmer.Huang@....com>,
        "Liu, Aaron" <Aaron.Liu@....com>, Jason Gunthorpe <jgg@...dia.com>
Cc:     Joerg Roedel <jroedel@...e.de>,
        "regressions@...ts.linux.dev" <regressions@...ts.linux.dev>,
        Thorsten Leemhuis <regressions@...mhuis.info>,
        Linux PCI <linux-pci@...r.kernel.org>,
        "Pan, Xinhui" <Xinhui.Pan@....com>, amd-gfx@...ts.freedesktop.org,
        LKML <linux-kernel@...r.kernel.org>,
        Bjorn Helgaas <bhelgaas@...gle.com>,
        "iommu@...ts.linux.dev" <iommu@...ts.linux.dev>,
        "Deucher, Alexander" <Alexander.Deucher@....com>,
        Christian König <christian.koenig@....com>
Subject: Re: [regression, bisected, pci/iommu] Bug 216865 - Black screen when amdgpu started during 6.2-rc1 boot with AMD IOMMU enabled

Vasant,

I applied your four patches to 6.2-rc8 and built that. The black screen, 
null pointer dereference, and warnings didn't happen when booting 
6.2-rc8 with your patches. There were errors that the IOMMU wasn't 
restarted when amdgpu and amdkfd was starting though at kernel: kfd kfd: 
amdgpu: Failed to resume IOMMU for device 1002:9874. I don't know if 
those IOMMU errors were expected or not, but I did see those types of 
messages when I used amd_iommu=off to work around the black screen 
before. I didn't use amd_iommu=off when testing 6.2-rc8 with your 
patches. There were also a different amdgpu warning at 
drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu8_smumgr.c:98 
smu8_send_msg_to_smc_with_parameter+0x103/0x140 and errors about amdgpu 
timeouts on 1/3 boots. Plasma took over a minute to start and shut down 
on that boot which was unusually long. I've seen those sorts of amdgpu 
warnings and errors infrequently before so they might be unrelated to 
the IOMMU problem. The part of the journal where those errors started 
was the following.

Feb 16 13:22:31 kernel: [drm] amdgpu kernel modesetting enabled.
Feb 16 13:22:31 kernel: amdgpu: Topology: Add APU node [0x0:0x0]
Feb 16 13:22:31 kernel: [drm] initializing kernel modesetting (CARRIZO 
0x1002:0x9874 0x103C:0x8332 0xCA).
Feb 16 13:22:31 kernel: [drm] register mmio base: 0xF0400000
Feb 16 13:22:31 kernel: [drm] register mmio size: 262144
Feb 16 13:22:31 kernel: [drm] add ip block number 0 <vi_common>
Feb 16 13:22:31 kernel: [drm] add ip block number 1 <gmc_v8_0>
Feb 16 13:22:31 kernel: [drm] add ip block number 2 <cz_ih>
Feb 16 13:22:31 kernel: [drm] add ip block number 3 <gfx_v8_0>
Feb 16 13:22:31 kernel: [drm] add ip block number 4 <sdma_v3_0>
Feb 16 13:22:31 kernel: [drm] add ip block number 5 <powerplay>
Feb 16 13:22:31 kernel: [drm] add ip block number 6 <dm>
Feb 16 13:22:31 kernel: [drm] add ip block number 7 <uvd_v6_0>
Feb 16 13:22:31 kernel: [drm] add ip block number 8 <vce_v3_0>
Feb 16 13:22:31 kernel: [drm] add ip block number 9 <acp_ip>
Feb 16 13:22:31 kernel: amdgpu 0000:00:01.0: amdgpu: Fetched VBIOS from VFCT
Feb 16 13:22:31 kernel: amdgpu: ATOM BIOS: 113-C75100-031
Feb 16 13:22:31 kernel: [drm] UVD is enabled in physical mode
Feb 16 13:22:31 kernel: [drm] VCE enabled in physical mode
Feb 16 13:22:31 kernel: Console: switching to colour dummy device 80x25
Feb 16 13:22:31 kernel: amdgpu 0000:00:01.0: vgaarb: deactivate vga console
Feb 16 13:22:31 kernel: amdgpu 0000:00:01.0: amdgpu: Trusted Memory Zone 
(TMZ) feature not supported
Feb 16 13:22:31 kernel: [drm] vm size is 64 GB, 2 levels, block size is 
10-bit, fragment size is 9-bit
Feb 16 13:22:31 kernel: amdgpu 0000:00:01.0: amdgpu: VRAM: 512M 
0x000000F400000000 - 0x000000F41FFFFFFF (512M used)
Feb 16 13:22:31 kernel: amdgpu 0000:00:01.0: amdgpu: GART: 1024M 
0x000000FF00000000 - 0x000000FF3FFFFFFF
Feb 16 13:22:31 kernel: [drm] Detected VRAM RAM=512M, BAR=512M
Feb 16 13:22:31 kernel: [drm] RAM width 64bits UNKNOWN
Feb 16 13:22:31 kernel: [drm] amdgpu: 512M of VRAM memory ready
Feb 16 13:22:31 kernel: [drm] amdgpu: 3704M of GTT memory ready.
Feb 16 13:22:31 kernel: [drm] GART: num cpu pages 262144, num gpu pages 
262144
Feb 16 13:22:31 kernel: [drm] PCIE GART of 1024M enabled (table at 
0x000000F400600000).
Feb 16 13:22:31 kernel: amdgpu: hwmgr_sw_init smu backed is smu8_smu
Feb 16 13:22:31 kernel: [drm] Found UVD firmware Version: 1.91 Family ID: 11
Feb 16 13:22:31 kernel: [drm] UVD ENC is disabled
Feb 16 13:22:31 kernel: [drm] Found VCE firmware Version: 52.4 Binary ID: 3
Feb 16 13:22:31 kernel: amdgpu: smu version 27.18.00
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: values for Engine clock
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         300000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         480000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         533340
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         576000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         626090
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         685720
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         720000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         757900
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: Validation clocks:
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    engine_max_clock: 75790
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    memory_max_clock: 93300
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    level           : 8
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: values for Display clock
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         300000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         400000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         496560
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         626090
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         685720
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         757900
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         800000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         847060
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: Validation clocks:
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    engine_max_clock: 75790
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    memory_max_clock: 93300
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    level           : 8
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: values for Memory clock
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         667000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:         933000
Feb 16 13:22:31 kernel: [drm] DM_PPLIB: Validation clocks:
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    engine_max_clock: 75790
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    memory_max_clock: 93300
Feb 16 13:22:31 kernel: [drm] DM_PPLIB:    level           : 8
Feb 16 13:22:31 kernel: [drm] Display Core initialized with v3.2.215!
Feb 16 13:22:32 kernel: [drm] UVD initialized successfully.
Feb 16 13:22:32 kernel: [drm] VCE initialized successfully.
Feb 16 13:22:32 kernel: kfd kfd: amdgpu: Allocated 3969056 bytes on gart
Feb 16 13:22:32 kernel: amdgpu: sdma_bitmap: f
Feb 16 13:22:32 kernel: kfd kfd: amdgpu: Failed to resume IOMMU for 
device 1002:9874
Feb 16 13:22:32 kernel: kfd kfd: amdgpu: device 1002:9874 NOT added due 
to errors
Feb 16 13:22:32 kernel: amdgpu 0000:00:01.0: amdgpu: SE 1, SH per SE 1, 
CU per SH 8, active_cu_number 6
Feb 16 13:22:32 kernel: [drm] Initialized amdgpu 3.49.0 20150101 for 
0000:00:01.0 on minor 0
Feb 16 13:22:32 kernel: fbcon: amdgpudrmfb (fb0) is primary device
Feb 16 13:22:33 kernel: Console: switching to colour frame buffer device 
170x48
Feb 16 13:22:33 kernel: amdgpu 0000:00:01.0: [drm] fb0: amdgpudrmfb 
frame buffer device
Feb 16 13:22:33 kernel: audit: type=1334 audit(1676571753.397:17): 
prog-id=21 op=LOAD
Feb 16 13:22:33 kernel: audit: type=1130 audit(1676571753.419:18): pid=1 
uid=0 auid=4294967295 ses=4294967295 subj=kernel msg='unit=dbus-broker 
comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? 
terminal=? res=success'
Feb 16 13:22:33 kernel: audit: type=1130 audit(1676571753.456:19): pid=1 
uid=0 auid=4294967295 ses=4294967295 subj=kernel 
msg='unit=dracut-initqueue comm="systemd" exe="/usr/lib/systemd/systemd" 
hostname=? addr=? terminal=? res=success'
Feb 16 13:22:33 kernel: audit: type=1130 audit(1676571753.492:20): pid=1 
uid=0 auid=4294967295 ses=4294967295 subj=kernel 
msg='unit=systemd-fsck-root comm="systemd" 
exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Feb 16 13:22:33 kernel: EXT4-fs (dm-0): mounted filesystem 
00107de9-54ef-4784-a03f-61802ed0b350 with ordered data mode. Quota mode: 
none.
Feb 16 13:22:36 kernel: ------------[ cut here ]------------
Feb 16 13:22:36 kernel: smu8_send_msg_to_smc_with_parameter(0x0009, 0x0) 
timed out after 2814625 us
Feb 16 13:22:36 kernel: WARNING: CPU: 1 PID: 112 at 
drivers/gpu/drm/amd/amdgpu/../pm/powerplay/smumgr/smu8_smumgr.c:98 
smu8_send_msg_to_smc_with_parameter+0x103/0x140 [amdgpu]
Feb 16 13:22:36 kernel: Modules linked in: amdgpu i2c_algo_bit 
drm_ttm_helper ttm iommu_v2 mfd_core drm_buddy gpu_sched 
drm_display_helper drm_kms_helper hid_logitech_hidpp drm 
crct10dif_pclmul crc32_pclmul crc32c_intel r8169 sd_mod 
ghash_clmulni_intel t10_pi sha512_ssse3 crc64_rocksoft_generic 
crc64_rocksoft wdat_wdt sp5100_tco hid_logitech_dj crc64 cec video wmi 
fuse dm_multipath
Feb 16 13:22:36 kernel: CPU: 1 PID: 112 Comm: kworker/1:3 Not tainted 
6.2.0-rc8+ #94
Feb 16 13:22:36 kernel: Hardware name: HP HP Laptop 15-bw0xx/8332, BIOS 
F.52 12/03/2019
Feb 16 13:22:36 kernel: Workqueue: events amdgpu_vce_idle_work_handler 
[amdgpu]
Feb 16 13:22:36 kernel: RIP: 
0010:smu8_send_msg_to_smc_with_parameter+0x103/0x140 [amdgpu]
Feb 16 13:22:36 kernel: Code: 20 48 c7 c7 28 1c c1 c0 48 89 c1 48 f7 ea 
48 89 c8 44 89 e9 48 c1 f8 3f 48 c1 fa 07 48 29 c2 49 89 d0 44 89 e2 e8 
c5 28 48 e0 <0f> 0b eb b0 bd ea ff ff ff eb a9 48 8b 7b 40 be c0 01 00 
00 48 8b
Feb 16 13:22:36 kernel: RSP: 0018:ffffb997004c7db8 EFLAGS: 00010282
Feb 16 13:22:36 kernel: RAX: 000000000000004b RBX: ffff8b4e4f596800 RCX: 
0000000000000000
Feb 16 13:22:36 kernel: RDX: 0000000000000001 RSI: ffffffffa14cf075 RDI: 
00000000ffffffff
Feb 16 13:22:36 kernel: RBP: 00000000ffffffc2 R08: 0000000000000000 R09: 
ffffb997004c7c68
Feb 16 13:22:36 kernel: R10: 0000000000000003 R11: ffffffffa1d42e48 R12: 
0000000000000009
Feb 16 13:22:36 kernel: R13: 0000000000000000 R14: 00000003ded365a4 R15: 
0000000000000002
Feb 16 13:22:36 kernel: FS:  0000000000000000(0000) 
GS:ffff8b4f37480000(0000) knlGS:0000000000000000
Feb 16 13:22:36 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Feb 16 13:22:36 kernel: CR2: 00007f950a698364 CR3: 0000000033c10000 CR4: 
00000000001506e0
Feb 16 13:22:36 kernel: Call Trace:
Feb 16 13:22:36 kernel:  <TASK>
Feb 16 13:22:36 kernel:  smum_send_msg_to_smc+0xba/0xf0 [amdgpu]
Feb 16 13:22:36 kernel:  smu8_dpm_powergate_vce+0x15a/0x180 [amdgpu]
Feb 16 13:22:36 kernel:  pp_set_powergating_by_smu+0xed/0x1f0 [amdgpu]
Feb 16 13:22:36 kernel: amdgpu_dpm_set_powergating_by_smu+0x84/0xf0 [amdgpu]
Feb 16 13:22:36 kernel:  amdgpu_dpm_enable_vce+0x29/0xa0 [amdgpu]
Feb 16 13:22:36 kernel:  process_one_work+0x1c8/0x380
Feb 16 13:22:36 kernel:  worker_thread+0x4d/0x380
Feb 16 13:22:36 kernel:  ? _raw_spin_lock_irqsave+0x23/0x50
Feb 16 13:22:36 kernel:  ? __pfx_worker_thread+0x10/0x10
Feb 16 13:22:36 kernel:  kthread+0xe9/0x110
Feb 16 13:22:36 kernel:  ? __pfx_kthread+0x10/0x10
Feb 16 13:22:36 kernel:  ret_from_fork+0x2c/0x50
Feb 16 13:22:36 kernel:  </TASK>
Feb 16 13:22:36 kernel: ---[ end trace 0000000000000000 ]---
Feb 16 13:22:39 kernel: amdgpu: 
smu8_send_msg_to_smc_with_parameter(0x0004) aborted; SMU still servicing 
msg (0x0009)
Feb 16 13:22:41 kernel: amdgpu: 
smu8_send_msg_to_smc_with_parameter(0x0007) aborted; SMU still servicing 
msg (0x0009)

I'm attaching the kernel log for the boot of 6.2-rc8 + patches with the 
IOMMU errors and amdgpu warnings and timeouts.

Thanks,

Matt

On 2/16/23 00:25, Vasant Hegde wrote:
> Felix, Jason, Matt,
>
>
> On 2/16/2023 6:05 AM, Felix Kuehling wrote:
>> [+Shimmer, Aaron]
>>
>> Am 2023-02-15 um 10:39 schrieb Bjorn Helgaas:
>>> [+cc Christian, Xinhui, amd-gfx]
>>>
>>> On Fri, Jan 06, 2023 at 01:48:11PM +0800, Baolu Lu wrote:
>>>> On 1/5/23 11:27 PM, Felix Kuehling wrote:
>>>>> Am 2023-01-05 um 09:46 schrieb Deucher, Alexander:
>>>>>>> -----Original Message-----
>>>>>>> From: Hegde, Vasant <Vasant.Hegde@....com>
>>>>>>> On 1/5/2023 4:07 PM, Baolu Lu wrote:
>>>>>>>> On 2023/1/5 18:27, Vasant Hegde wrote:
>>>>>>>>> On 1/5/2023 6:39 AM, Matt Fagnani wrote:
>>>>>>>>>> I built 6.2-rc2 with the patch applied. The same black
>>>>>>>>>> screen problem happened with 6.2-rc2 with the patch. I
>>>>>>>>>> tried to use early kdump with 6.2-rc2 with the patch
>>>>>>>>>> twice by panicking the kernel with sysrq+alt+c after the
>>>>>>>>>> black screen happened. The system rebooted after about
>>>>>>>>>> 10-20 seconds both times, but no kdump and dmesg files
>>>>>>>>>> were saved in /var/crash. I'm attaching the lspci -vvv
>>>>>>>>>> output as requested. ...
>>>>>>>>> Looking into lspci output, it doesn't list ACS feature
>>>>>>>>> for Graphics card. So with your fix it didn't enable PASID
>>>>>>>>> and hence it failed to boot. ...
>>>>>>>> So do you mind telling why does the PASID need to be enabled
>>>>>>>> for the graphic device? Or in another word, what does the
>>>>>>>> graphic driver use the PASID for? ...
>>>>>> The GPU driver uses the pasid for shared virtual memory between
>>>>>> the CPU and GPU.  I.e., so that the user apps can use the same
>>>>>> virtual address space on the GPU and the CPU.  It also uses
>>>>>> pasid to take advantage of recoverable device page faults using
>>>>>> PRS. ...
>>>>> Agreed. This applies to GPU computing on some older AMD APUs that
>>>>> take advantage of memory coherence and IOMMUv2 address translation
>>>>> to create a shared virtual address space between the CPU and GPU.
>>>>> In this case it seems to be a Carrizo APU. It is also true for
>>>>> Raven APUs. ...
>>>> Thanks for the explanation.
>>>>
>>>> This is actually the problem that commit 201007ef707a was trying to
>>>> fix.  The PCIe fabric routes Memory Requests based on the TLP
>>>> address, ignoring any PASID (PCIe r6.0, sec 2.2.10.4), so a TLP with
>>>> PASID that should go upstream to the IOMMU may instead be routed as
>>>> a P2P Request if its address falls in a bridge window.
>>>>
>>>> In SVA case, the IOMMU shares the address space of a user
>>>> application.  The user application side has no knowledge about the
>>>> PCI bridge window.  It is entirely possible that the device is
>>>> programed with a P2P address and results in a disaster.
>>> Is this stalled?  We explored the idea of changing the PCI core so
>>> that for devices that use ATS/PRI, we could enable PASID without
>>> checking for ACS [1], but IIUC we ultimately concluded that it was
>>> based on a misunderstanding of how ATS Translation Requests are routed
>>> and that an AMD driver change would be required [2].
>>>
>>> So it seems like we still have this regression, and we're running out
>>> of time before v6.2.
>>>
>>> [1] https://lore.kernel.org/all/20230114073420.759989-1-baolu.lu@linux.intel.com/
>>> [2] https://lore.kernel.org/all/Y91X9MeCOsa67CC6@nvidia.com/
>> If I understand this correctly, the HW or the BIOS is doing something wrong
>> about reporting ACS. I don't know what the GPU driver can do other than add some
>> quirk to stop using AMD IOMMUv2 on this HW/BIOS.
>>
>> It looks like the problem is triggered when the driver calls
>> amd_iommu_init_device. That's when the first WARNs happen, soon followed by a
>> kernel oops in report_iommu_fault. The driver doesn't know anything is wrong
>> because amd_iommu_init_device seems to return "success". And the oops is not in
>> the GPU driver either.
> WARN is fixed and its in Joerg's tree.
> https://lore.kernel.org/all/20230111121503.5931-1-vasant.hegde@amd.com/
>
> report_iommu_fault() happened because in amd_iommu_init_device() path it failed
> to attach devices to new domain and returned error. But it didn't put devices
> back to old domain properly. It left in incosistent state and resulted in IO
> page fault. I have proposed series to handle device to domain attachment failure
> and better handling of subsequent report_iommu_fault().
> https://lore.kernel.org/linux-iommu/20230215052642.6016-1-vasant.hegde@amd.com/
>
>
> @Matt,
>    Can you please help to verify above patches on your system where you hit the
> issue originally?
>    (Grab above two series, apply it on top of latest kernel and test it)
>
> -Vasant
>
View attachment "6.2-rc8-amd-iommu-patches-journalctl-k-1.txt" of type "text/plain" (110142 bytes)

Powered by blists - more mailing lists