Date: Mon, 15 Apr 2024 14:57:25 -0400
From: Eric Wagner <ewagner12@...il.com>
To: Jason Gunthorpe <jgg@...pe.ca>
Cc: Joerg Roedel <joro@...tes.org>, Will Deacon <will@...nel.org>, 
	Suravee Suthikulpanit <suravee.suthikulpanit@....com>, Robin Murphy <robin.murphy@....com>, 
	iommu@...ts.linux.dev, linux-kernel@...r.kernel.org
Subject: Re: Kernel 6.7 regression doesn't boot if using AMD eGPU

Apologies if I made a mistake in the first bisect, I'm new to kernel
debugging.

I tested cedc811c76778bdef91d405717acee0de54d8db5 (x86/amd) and
3613047280ec42a4e1350fdc1a6dd161ff4008cc (core) directly and both were good.
Then I ran git bisect again with e8cca466a84a75f8ff2a7a31173c99ee6d1c59d2
as the bad and 6e6c6d6bc6c96c2477ddfea24a121eb5ee12b7a3 as the good and the
bisect log is attached. It ended up at the same commit as before.

I've also attached a picture of the boot screen that occurs when it hangs.
0000:05:00.0 is the PCIe bus address of the RX 580 eGPU that's causing the
problem.

On Mon, Apr 15, 2024 at 12:30 PM Jason Gunthorpe <jgg@...pe.ca> wrote:

> On Sat, Apr 13, 2024 at 06:04:12PM -0400, Eric Wagner wrote:
> >    On my Thinkpad T14s G3 AMD (Ryzen 7 6850U) laptop connected to an AMD
> >    RX 580 in Akitio Node Thunderbolt 3 eGPU. Booting with the eGPU
> >    connected hangs on kernels 6.7 and 6.8, but worked on 6.6. For
> >    debugging, I find that adding the kernel parameter amd_iommu=off seems
> >    to fix the issue and allows booting with the eGPU on 6.7.
> >    I tried bisecting the issue between 6.6 and 6.7 and ended up with:
> >    "e8cca466a84a75f8ff2a7a31173c99ee6d1c59d2 is the first bad commit" in
> >    the attached. This seems to indicate an amd iommu issue.
> >    Two others also reported the same issue on AMD Ryzen 7 7840 with AMD
> RX
> >    6000 connected as eGPU
> >    ([1]https://gitlab.freedesktop.org/drm/amd/-/issues/3182).
> >    Let me know if you need more information.
> >
> > References
> >
> >    1. https://gitlab.freedesktop.org/drm/amd/-/issues/3182
>
> > Bisecting: 366 revisions left to test after this (roughly 9 steps)
> > [74e9347ebc5be452935fe4f3eddb150aa5a6f4fe] Merge tag
> 'loongarch-fixes-6.6-3' of git://
> git.kernel.org/pub/scm/linux/kernel/git/chenhuacai/linux-loongson
> > Bisecting: 182 revisions left to test after this (roughly 8 steps)
> > [f6176471542d991137543af2ef1c18dae3286079] Merge tag
> 'mtd/fixes-for-6.6-rc7' of git://
> git.kernel.org/pub/scm/linux/kernel/git/mtd/linux
> > Bisecting: 87 revisions left to test after this (roughly 7 steps)
> > [fe3cfe869d5e0453754cf2b4c75110276b5e8527] Merge tag 'phy-fixes-6.6' of
> git://git.kernel.org/pub/scm/linux/kernel/git/phy/linux-phy
> > Bisecting: 43 revisions left to test after this (roughly 6 steps)
> > [c76c067e488ccd55734c3e750799caf2c5956db6] s390/pci: Use dma-iommu layer
> > Bisecting: 27 revisions left to test after this (roughly 5 steps)
> > [aa5cabc4ce8e6b45d170d162dc54b1bac1767c47] Merge tag 'arm-smmu-updates'
> of git://git.kernel.org/pub/scm/linux/kernel/git/will/linux into arm/smmu
> > Bisecting: 14 revisions left to test after this (roughly 4 steps)
> > [bbc70e0aec287e164344b1a071bd46466a4f29b3] iommu/dart: Remove the
> force_bypass variable
> > Bisecting: 9 revisions left to test after this (roughly 3 steps)
> > [e82c175e63229ea495a0a0b5305a98b5b6ee5346] Revert "iommu/vt-d: Remove
> unused function"
> > Bisecting: 5 revisions left to test after this (roughly 2 steps)
> > [92bce97f0c341d3037b0f364b6839483f6a41cae] s390/pci: Fix reset of IOMMU
> software counters
> > Bisecting: 3 revisions left to test after this (roughly 2 steps)
> > [3613047280ec42a4e1350fdc1a6dd161ff4008cc] Merge tag 'v6.6-rc7' into core
> > Bisecting: 2 revisions left to test after this (roughly 1 step)
> > [f7da9c081517daba70f9f9342e09d7a6322ba323] iommu/tegra-smmu: Drop
> unnecessary error check for for debugfs_create_dir()
> > Bisecting: 1 revision left to test after this (roughly 1 step)
> > [9e13ec61de2a51195b122a79461431d8cb99d7b5] iommu/virtio: Add
> __counted_by for struct viommu_request and use struct_size()
> > Bisecting: 0 revisions left to test after this (roughly 0 steps)
> > [6e6c6d6bc6c96c2477ddfea24a121eb5ee12b7a3] iommu: Avoid unnecessary
> cache invalidations
> > e8cca466a84a75f8ff2a7a31173c99ee6d1c59d2 is the first bad commit
> > commit e8cca466a84a75f8ff2a7a31173c99ee6d1c59d2
> > Merge: 6e6c6d6bc6 f7da9c0815 aa5cabc4ce 9e13ec61de e82c175e63 cedc811c76
> 3613047280 92bce97f0c
> > Author: Joerg Roedel <jroedel@...e.de>
> > Date:   Fri Oct 27 09:13:40 2023 +0200
> >
> >     Merge branches 'iommu/fixes', 'arm/tegra', 'arm/smmu', 'virtio',
> 'x86/vt-d', 'x86/amd', 'core' and 's390' into next
>
> Do you have the good/bad log on this? It doesn't look like bisect
> tested enough stuff to really conclude the merge is the bad thing, at
> a minimum it should be testing all the bases of the merge. Do you have
> --first-parent set or something?
>
> I would test cedc811c76778bdef91d405717acee0de54d8db5 (x86/amd) and
> 3613047280ec42a4e1350fdc1a6dd161ff4008cc (core) directly. Most likely
> cedc will be bad problem.
>
> If one of them is bad then restart the bisection with that as the bad
> and 6e6c6d6bc6 as the good.
>
> (or run bisect again with e8cca466a84a75f8ff2a7a31173c99ee6d1c59d2 as
> the bad and 6e6c6d6bc6c96c2477ddfea24a121eb5ee12b7a3 as the good
> without --first-parent)
>
> Jason
>

Content of type "text/html" skipped

View attachment "bisect2.log" of type "text/x-log" (1945 bytes)

Download attachment "20240415_133212.jpg" of type "image/jpeg" (2084419 bytes)

Powered by blists - more mailing lists