linux-kernel - Re: [PATCH 3/3] drm/amdgpu: Switch to interrupted wait to recover from ring hang.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <7ea85f74-a52b-0e10-63ff-c0ccf2f63d8b@amd.com>
Date:   Thu, 26 Apr 2018 08:28:53 -0400
From:   Andrey Grodzovsky <Andrey.Grodzovsky@....com>
To:     "Eric W. Biederman" <ebiederm@...ssion.com>
Cc:     "Panariti, David" <David.Panariti@....com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "amd-gfx@...ts.freedesktop.org" <amd-gfx@...ts.freedesktop.org>,
        "Deucher, Alexander" <Alexander.Deucher@....com>,
        "Koenig, Christian" <Christian.Koenig@....com>,
        "oleg@...hat.com" <oleg@...hat.com>,
        "akpm@...ux-foundation.org" <akpm@...ux-foundation.org>
Subject: Re: [PATCH 3/3] drm/amdgpu: Switch to interrupted wait to recover
 from ring hang.



On 04/25/2018 04:55 PM, Eric W. Biederman wrote:
> Andrey Grodzovsky <Andrey.Grodzovsky@....com> writes:
>
>> On 04/24/2018 12:30 PM, Eric W. Biederman wrote:
>>> "Panariti, David" <David.Panariti@....com> writes:
>>>
>>>> Andrey Grodzovsky <andrey.grodzovsky@....com> writes:
>>>>> Kind of dma_fence_wait_killable, except that we don't have such API
>>>>> (maybe worth adding ?)
>>>> Depends on how many places it would be called, or think it might be called.  Can always factor on the 2nd time it's needed.
>>>> Factoring, IMO, rarely hurts.  The factored function can easily be visited using `M-.' ;->
>>>>
>>>> Also, if the wait could be very long, would a log message, something like "xxx has run for Y seconds."  help?
>>>> I personally hate hanging w/no info.
>>> Ugh.  This loop appears susceptible to loosing wake ups.  There are
>>> races between when a wake-up happens, when we clear the sleeping state,
>>> and when we test the stat to see if we should stat awake.  So yes
>>> implementing a dma_fence_wait_killable that handles of all that
>>> correctly sounds like an very good idea.
>> I am not clear here - could you be more specific about what races will happen
>> here, more bellow
>>> Eric
>>>
>>>
>>>>> If the ring is hanging for some reason allow to recover the waiting by sending fatal signal.
>>>>>
>>>>> Originally-by: David Panariti <David.Panariti@....com>
>>>>> Signed-off-by: Andrey Grodzovsky <andrey.grodzovsky@....com>
>>>>> ---
>>>>>     drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c | 14 ++++++++++----
>>>>>     1 file changed, 10 insertions(+), 4 deletions(-)
>>>>>
>>>>> diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
>>>>> index eb80edf..37a36af 100644
>>>>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
>>>>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
>>>>> @@ -421,10 +421,16 @@ int amdgpu_ctx_wait_prev_fence(struct amdgpu_ctx *ctx, unsigned ring_id)
>>>>>
>>>>>         if (other) {
>>>>>                 signed long r;
>>>>> -             r = dma_fence_wait_timeout(other, false, MAX_SCHEDULE_TIMEOUT);
>>>>> -             if (r < 0) {
>>>>> -                     DRM_ERROR("Error (%ld) waiting for fence!\n", r);
>>>>> -                     return r;
>>>>> +
>>>>> +             while (true) {
>>>>> +                     if ((r = dma_fence_wait_timeout(other, true,
>>>>> +                                     MAX_SCHEDULE_TIMEOUT)) >= 0)
>>>>> +                             return 0;
>>>>> +
>> Do you mean that by the time I reach here some other thread from my group
>> already might dequeued SIGKILL since it's a shared signal and hence
>> fatal_signal_pending will return false ? Or are you talking about the
>> dma_fence_wait_timeout implementation in dma_fence_default_wait with
>> schedule_timeout ?
> Given Oleg's earlier comment about the scheduler having special cases
> for signals I might be wrong.  But in general there is a pattern:
>
> 	for (;;) {
> 		set_current_state(TASK_UNINTERRUPTIBLE);
> 		if (loop_is_done())
> 			break;
> 		schedule();
> 	}
>          set_current_state(TASK_RUNNING);
>
> If you violate that pattern by testing for a condition without
> having first set your task as TASK_UNINTERRUPTIBLE (or whatever your
> sleep state is).  Then it is possible to miss a wake-up that
> tests the condidtion.
>
> Thus I am quite concerned that there is a subtle corner case where
> you can miss a wakeup and not retest fatal_signal_pending().


I see the general problem now. In this particular case 
dma_fence_default_wait
and the caller of wake_up_state use lock for protecting wake up delivery 
and wakeup condition
and also dma_fence_default_wait retests the wakeup condition on entry.
But obviously it's a bad practice to rely on API's internal 
implementation for assumptions
in client code.

>
> Given that there is is a timeout the worst case might have you sleep
> MAX_SCHEDULE_TIMEOUT instead of indefinitely.

It actually means never wake

>
> Without a comment why this is safe, or having fatal_signal_pending
> check integrated into dma_fence_wait_timeout I am not comfortable
> with this loop.

Agree, fatal_signal_pending should be part of the wait function.

Andrey

>
> Eric
>
>
>>>>> +                     if (fatal_signal_pending(current)) {
>>>>> +                             DRM_ERROR("Error (%ld) waiting for fence!\n", r);
>>>>> +                             return r;
>>>>> +                     }
>>>>>                 }
>>>>>         }
>>>>>
>>>>> --
>>>>> 2.7.4
>>>>>
>>> Eric