Message-ID: <CAH1Ww+QkS54PW4LZ42KtqGtjACmsAt_Y_rd3fiR5PUHQR65Hpg@mail.gmail.com>
Date:   Wed, 3 Feb 2021 09:56:06 +0100
From:   Daniel Gomez <daniel@...c.com>
To:     Christian König <christian.koenig@....com>
Cc:     Daniel Vetter <daniel@...ll.ch>,
        amd-gfx list <amd-gfx@...ts.freedesktop.org>,
        dri-devel <dri-devel@...ts.freedesktop.org>,
        Alex Deucher <alexander.deucher@....com>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: Re: [amdgpu] deadlock

On Wed, 3 Feb 2021 at 09:51, Christian König <christian.koenig@....com> wrote:
>
> Am 03.02.21 um 09:48 schrieb Daniel Vetter:
> > On Wed, Feb 3, 2021 at 9:36 AM Christian König <christian.koenig@....com> wrote:
> >> Hi Daniel,
> >>
> >> this is not a deadlock, but rather a hardware lockup.
> > Are you sure? Ime getting stuck in dma_fence_wait has generally good
> > chance of being a dma_fence deadlock. GPU hang should never result in
> > a forever stuck dma_fence.
>
> Yes, I'm pretty sure. Otherwise the hardware clocks wouldn't go up like
> this.
>
> Question is rather why we end up in the userptr handling for GFX? Our
> ROCm OpenCL stack shouldn't use this.
>
> > Daniel, can you pls re-hang your machine and then dump backtraces of
> > all tasks into dmesg with sysrq-t, and then attach that? Without al
yes, I'll try it again.
> > the backtraces it's tricky to construct the full dependency chain of
> > what's going on. Also is this plain -rc6, not some more patches on
> > top?
It's plain -rc6 in regards to amdgpu and dma fences. I have some other patches
on top but they are v4l2 related.
>
> Yeah, that's still a good idea to have.
>
> Christian.
>
> > -Daniel
> >
> >> Which OpenCl stack are you using?
Radeon Software for Linux 20.20.
https://www.amd.com/en/support/kb/release-notes/rn-amdgpu-unified-linux-20-20

I've also noticed the 'latest' linux-firmware adds support for the
20.45 version but
I haven't tested it yet since I couldn't bring it up properly the
support as I was doing for
the previous versions. Somehow the libamdocl64.so got reduced from 80 Mb to
1.2 Mb and I couldn't figure out what has been changed.

Should I use ROCm instead of Radeon for Linux?

> >>
> >> Regards,
> >> Christian.
> >>
> >> Am 03.02.21 um 09:33 schrieb Daniel Gomez:
> >>> Hi all,
> >>>
> >>> I have a deadlock with the amdgpu mainline driver when running in parallel two
> >>> OpenCL applications. So far, we've been able to replicate it easily by executing
> >>> clinfo and MatrixMultiplication (from AMD opencl-samples). It's quite old the
> >>> opencl-samples so, if you have any other suggestion for testing I'd be very
> >>> happy to test it as well.
> >>>
> >>> How to replicate the issue:
> >>>
> >>> # while true; do /usr/bin/MatrixMultiplication --device gpu \
> >>>       --deviceId 0 -x 1000 -y 1000 -z 1000 -q -t -i 50; done
> >>> # while true; do clinfo; done
> >>>
> >>> Output:
> >>>
> >>> After a minute or less (sometimes could be more) I can see that
> >>> MatrixMultiplication and clinfo hang. In addition, with radeontop you can see
> >>> how the Graphics pipe goes from ~50% to 100%. Also the shader clocks
> >>> goes up from ~35% to ~96%.
> >>>
> >>> clinfo keeps printing:
> >>> ioctl(7, DRM_IOCTL_SYNCOBJ_WAIT, 0x7ffe46e5f950) = -1 ETIME (Timer expired)
> >>>
> >>> And MatrixMultiplication prints the following (strace) if you try to
> >>> kill the process:
> >>>
> >>> sched_yield()                           = 0
> >>> futex(0x557e945343b8, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0,
> >>> NULL, FUTEX_BITSET_MATCH_ANYstrace: Process 651 detached
> >>>    <detached ...>
> >>>
> >>> After this, the gpu is not functional at all and you'd need a power cycle reset
> >>> to restore the system.
> >>>
> >>> Hardware info:
> >>> CPU: AMD Ryzen Embedded V1605B with Radeon Vega Gfx (8) @ 2.000GHz
> >>> GPU: AMD ATI Radeon Vega Series / Radeon Vega Mobile Series
> >>>
> >>> 03:00.0 VGA compatible controller: Advanced Micro Devices, Inc.
> >>> [AMD/ATI] Raven Ridge [Radeon Vega Series / Radeon Vega Mobile Series]
> >>> (rev 83)
> >>>       DeviceName: Broadcom 5762
> >>>       Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Raven Ridge
> >>> [Radeon Vega Series / Radeon Vega Mobile Series]
> >>>       Kernel driver in use: amdgpu
> >>>       Kernel modules: amdgpu
> >>>
> >>> Linux kernel info:
> >>>
> >>> root@...222:~# uname -a
> >>> Linux qt5222 5.11.0-rc6-qtec-standard #2 SMP Tue Feb 2 09:41:46 UTC
> >>> 2021 x86_64 x86_64 x86_64 GNU/Linux
> >>>
> >>> By enabling the kernel locks stats I could see the MatrixMultiplication is
> >>> hanged in the amdgpu_mn_invalidate_gfx function:
> >>>
> >>> [  738.359202] 1 lock held by MatrixMultiplic/653:
> >>> [  738.359206]  #0: ffff88810e364fe0
> >>> (&adev->notifier_lock){+.+.}-{3:3}, at:
> >>> amdgpu_mn_invalidate_gfx+0x34/0xa0 [amdgpu]
> >>>
> >>> I can see in the the amdgpu_mn_invalidate_gfx function: the
> >>> dma_resv_wait_timeout_rcu uses wait_all (fences) and MAX_SCHEDULE_TIMEOUT so, I
> >>> guess the code gets stuck there waiting forever. According to the
> >>> documentation: "When somebody tries to invalidate the page tables we block the
> >>> update until all operations on the pages in question are completed, then those
> >>> pages are marked  as accessed and also dirty if it wasn’t a read only access."
> >>> Looks like the fences are deadlocked and therefore, it never returns. Could it
> >>> be possible? any hint to where can I look to fix this?
> >>>
> >>> Thank you  in advance.
> >>>
> >>> Here the full dmesg output:
> >>>
> >>> [  738.337726] INFO: task MatrixMultiplic:653 blocked for more than 122 seconds.
> >>> [  738.344937]       Not tainted 5.11.0-rc6-qtec-standard #2
> >>> [  738.350384] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs"
> >>> disables this message.
> >>> [  738.358240] task:MatrixMultiplic state:D stack:    0 pid:  653
> >>> ppid:     1 flags:0x00004000
> >>> [  738.358254] Call Trace:
> >>> [  738.358261]  ? dma_fence_default_wait+0x1eb/0x230
> >>> [  738.358276]  __schedule+0x370/0x960
> >>> [  738.358291]  ? dma_fence_default_wait+0x117/0x230
> >>> [  738.358297]  ? dma_fence_default_wait+0x1eb/0x230
> >>> [  738.358305]  schedule+0x51/0xc0
> >>> [  738.358312]  schedule_timeout+0x275/0x380
> >>> [  738.358324]  ? dma_fence_default_wait+0x1eb/0x230
> >>> [  738.358332]  ? mark_held_locks+0x4f/0x70
> >>> [  738.358341]  ? dma_fence_default_wait+0x117/0x230
> >>> [  738.358347]  ? lockdep_hardirqs_on_prepare+0xd4/0x180
> >>> [  738.358353]  ? _raw_spin_unlock_irqrestore+0x39/0x40
> >>> [  738.358362]  ? dma_fence_default_wait+0x117/0x230
> >>> [  738.358370]  ? dma_fence_default_wait+0x1eb/0x230
> >>> [  738.358375]  dma_fence_default_wait+0x214/0x230
> >>> [  738.358384]  ? dma_fence_release+0x1a0/0x1a0
> >>> [  738.358396]  dma_fence_wait_timeout+0x105/0x200
> >>> [  738.358405]  dma_resv_wait_timeout_rcu+0x1aa/0x5e0
> >>> [  738.358421]  amdgpu_mn_invalidate_gfx+0x55/0xa0 [amdgpu]
> >>> [  738.358688]  __mmu_notifier_release+0x1bb/0x210
> >>> [  738.358710]  exit_mmap+0x2f/0x1e0
> >>> [  738.358723]  ? find_held_lock+0x34/0xa0
> >>> [  738.358746]  mmput+0x39/0xe0
> >>> [  738.358756]  do_exit+0x5c3/0xc00
> >>> [  738.358763]  ? find_held_lock+0x34/0xa0
> >>> [  738.358780]  do_group_exit+0x47/0xb0
> >>> [  738.358791]  get_signal+0x15b/0xc50
> >>> [  738.358807]  arch_do_signal_or_restart+0xaf/0x710
> >>> [  738.358816]  ? lockdep_hardirqs_on_prepare+0xd4/0x180
> >>> [  738.358822]  ? _raw_spin_unlock_irqrestore+0x39/0x40
> >>> [  738.358831]  ? ktime_get_mono_fast_ns+0x50/0xa0
> >>> [  738.358844]  ? amdgpu_drm_ioctl+0x6b/0x80 [amdgpu]
> >>> [  738.359044]  exit_to_user_mode_prepare+0xf2/0x1b0
> >>> [  738.359054]  syscall_exit_to_user_mode+0x19/0x60
> >>> [  738.359062]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
> >>> [  738.359069] RIP: 0033:0x7f6b89a51887
> >>> [  738.359076] RSP: 002b:00007f6b82b54b18 EFLAGS: 00000246 ORIG_RAX:
> >>> 0000000000000010
> >>> [  738.359086] RAX: fffffffffffffe00 RBX: 00007f6b82b54b50 RCX: 00007f6b89a51887
> >>> [  738.359091] RDX: 00007f6b82b54b50 RSI: 00000000c02064c3 RDI: 0000000000000007
> >>> [  738.359096] RBP: 00000000c02064c3 R08: 0000000000000003 R09: 00007f6b82b54bbc
> >>> [  738.359101] R10: 0000000000000001 R11: 0000000000000246 R12: 0000000165a0bc00
> >>> [  738.359106] R13: 0000000000000007 R14: 0000000000000001 R15: 0000000000000000
> >>> [  738.359129]
> >>>                  Showing all locks held in the system:
> >>> [  738.359141] 1 lock held by khungtaskd/54:
> >>> [  738.359148]  #0: ffffffff829f6840 (rcu_read_lock){....}-{1:2}, at:
> >>> debug_show_all_locks+0x15/0x183
> >>> [  738.359187] 1 lock held by systemd-journal/174:
> >>> [  738.359202] 1 lock held by MatrixMultiplic/653:
> >>> [  738.359206]  #0: ffff88810e364fe0
> >>> (&adev->notifier_lock){+.+.}-{3:3}, at:
> >>> amdgpu_mn_invalidate_gfx+0x34/0xa0 [amdgpu]
> >>>
> >>> Daniel
> >> _______________________________________________
> >> dri-devel mailing list
> >> dri-devel@...ts.freedesktop.org
> >> https://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.freedesktop.org%2Fmailman%2Flistinfo%2Fdri-devel&amp;data=04%7C01%7Cchristian.koenig%40amd.com%7C81203e5bac5841b8e5a108d8c82087a9%7C3dd8961fe4884e608e11a82d994e183d%7C0%7C0%7C637479389339295622%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&amp;sdata=RDSxD6OqD8HaOA2VnNfbJwLnKzhCLgOr5SVLjLF91bA%3D&amp;reserved=0
> >
> >
>

Powered by blists - more mailing lists