linux-kernel - Re: [PATCH] usb: dwc3: Add dwc3 lock for blocking interrupt storming

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a1b9752f-96e6-7abb-d6d3-ce4742742a0c@synopsys.com>
Date:   Fri, 11 Mar 2022 03:55:08 +0000
From:   Thinh Nguyen <Thinh.Nguyen@...opsys.com>
To:     정재훈 <jh0801.jung@...sung.com>,
        'Felipe Balbi' <balbi@...nel.org>,
        'Greg Kroah-Hartman' <gregkh@...uxfoundation.org>
CC:     "'open list:USB XHCI DRIVER'" <linux-usb@...r.kernel.org>,
        'open list' <linux-kernel@...r.kernel.org>,
        'Seungchull Suh' <sc.suh@...sung.com>,
        'Daehwan Jung' <dh10.jung@...sung.com>,
        "cpgs@...sung.com" <cpgs@...sung.com>,
        "cpgsproxy5@...sung.com" <cpgsproxy5@...sung.com>
Subject: Re: [PATCH] usb: dwc3: Add dwc3 lock for blocking interrupt storming

Thinh Nguyen wrote:
> 정재훈 wrote:
>>> -----Original Message-----
>>> From: Thinh Nguyen [mailto:Thinh.Nguyen@...opsys.com]
>>> Sent: Friday, March 11, 2022 10:57 AM
>>> To: 정재훈; Thinh Nguyen; 'Felipe Balbi'; 'Greg Kroah-Hartman'
>>> Cc: 'open list:USB XHCI DRIVER'; 'open list'; 'Seungchull Suh'; 'Daehwan
>>> Jung'; cpgs@...sung.com; cpgsproxy5@...sung.com
>>> Subject: Re: [PATCH] usb: dwc3: Add dwc3 lock for blocking interrupt
>>> storming
>>>
>>> 정재훈 wrote:
>>>> Hi.
>>>>
>>>>> -----Original Message-----
>>>>> From: Thinh Nguyen [mailto:Thinh.Nguyen@...opsys.com]
>>>>> Sent: Thursday, March 10, 2022 11:14 AM
>>>>> To: JaeHun Jung; Felipe Balbi; Greg Kroah-Hartman
>>>>> Cc: open list:USB XHCI DRIVER; open list; Seungchull Suh; Daehwan
>>>>> Jung
>>>>> Subject: Re: [PATCH] usb: dwc3: Add dwc3 lock for blocking interrupt
>>>>> storming
>>>>>
>>>>> Hi,
>>>>>
>>>>> JaeHun Jung wrote:
>>>>>> Interrupt Storming occurred with a very low probability of occurrence.
>>>>>> The occurrence of the problem is estimated to be caused by a race
>>>>>> condition between the top half and bottom half of the interrupt
>>>>>> service
>>>>> routine.
>>>>>> It was confirmed that variables have values that cannot be held when
>>>>>> ISR occurs through normal H / W irq.
>>>>>> ====================================================================
>>>>>> = (struct dwc3_event_buffer *) ev_buf = 0xFFFFFF88DE6A0380 (
>>>>>> 	(void *) buf = 0xFFFFFFC01594E000,
>>>>>> 	(void *) cache = 0xFFFFFF88DDC14080,
>>>>>> 	(unsigned int) length = 4096,
>>>>>> 	(unsigned int) lpos = 0,
>>>>>> 	(unsigned int) count = 0, <<
>>>>>> 	(unsigned int) flags = 1, <<
>>>>>> ====================================================================
>>>>>> = "evt->count=0" and "evt->flags=DWC3_EVENT_PENDING" cannot be set
>>>>>> at the same time.
>>>>>>
>>>>>> We estimate that a race condition occurred between dwc3_interrupt()
>>>>>> and dwc3_process_event_buf() called by
>>>>>> dwc3_gadget_process_pending_events().
>>>>>> So I try to block the race condition through spin_lock.
>>>>>
>>>>> This looks like it needs a memory barrier. Would this work for you?
>>>> Maybe it could be. But "evt->count = 0;" is updated on
>>> dwc3_process_event_buf().
>>>> So, I think spin_lock is more clear routine for this issue.
>>>>
>>>
>>> Not really. If problem is due to the evt->flags not updated in time, then
>>> the solution should be using the memory barrier. The spin_lock would
>>> obfuscate the issue. And we should avoid using spin_lock in the top-half.
>>
>> This issue was occurred by watchdog. The interrupt occurred in units of 4 to 5us and cannot be released until the bottom is executed.
>> If it is a problem with the memory barrier, the value should be updated after a few clocks and the TOP should run normally. Isn't it?
> 
> Can you guarantee that a value is stored after X amount of time, every time?
> 
>> And Could you explain me why we should avoid using spin_lock in the top-half.
>>
> 
> The top-half and bottom-half are serialized. While the bottom-half
> handler is running, the interrupt should be masked. If the top-half got
> called in the middle of the bottom-half handler, something else is
> wrong. There should not be a race that requires a spin_lock for this
> particular critical section.
> 
> The problem you're seeing is pointing toward a memory barrier issue.
> 
> Also you noted that there's an "interrupt storm", which doesn't indicate
> to me that it's due to PCIe legacy interrupt de-assertion delay response
> either.
> 
> Can you test it out and we can take a look further?
>
We want to avoid spin_lock because the top-half shouldn't stall for too
long, affecting performance. This can happen if some async call from the
upperlayer driver's holding the lock.

Thinh