netdev - Re: [ovs-dev] [PATCH net v2] openvswitch: fix a possible deadlock and lockdep warning

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 27 Mar 2014 14:18:29 -0700
From:	Pravin Shelar <pshelar@...ira.com>
To:	Jesse Gross <jesse@...ira.com>
Cc:	netdev <netdev@...r.kernel.org>, openvswitch <dev@...nvswitch.org>
Subject: Re: [ovs-dev] [PATCH net v2] openvswitch: fix a possible deadlock and
 lockdep warning

On Thu, Mar 27, 2014 at 1:01 PM, Jesse Gross <jesse@...ira.com> wrote:
> On Thu, Mar 27, 2014 at 11:40 AM, Pravin Shelar <pshelar@...ira.com> wrote:
>> On Thu, Mar 27, 2014 at 11:32 AM, Jesse Gross <jesse@...ira.com> wrote:
>>> On Thu, Mar 27, 2014 at 10:44 AM, Flavio Leitner <fbl@...hat.com> wrote:
>>>> On Thu, Mar 27, 2014 at 10:37:32AM -0700, Jesse Gross wrote:
>>>>> On Thu, Mar 27, 2014 at 10:33 AM, Flavio Leitner <fbl@...hat.com> wrote:
>>>>> > On Thu, Mar 27, 2014 at 10:19:23AM -0700, Pravin Shelar wrote:
>>>>> >> On Thu, Mar 27, 2014 at 7:05 AM, Flavio Leitner <fbl@...hat.com> wrote:
>>>>> >> > There are two problematic situations.
>>>>> >> >
>>>>> >> > A deadlock can happen when is_percpu is false because it can get
>>>>> >> > interrupted while holding the spinlock. Then it executes
>>>>> >> > ovs_flow_stats_update() in softirq context which tries to get
>>>>> >> > the same lock.
>>>>> >> >
>>>>> >> > The second sitation is that when is_percpu is true, the code
>>>>> >> > correctly disables BH but only for the local CPU, so the
>>>>> >> > following can happen when locking the remote CPU without
>>>>> >> > disabling BH:
>>>>> >> >
>>>>> >> >        CPU#0                            CPU#1
>>>>> >> >   ovs_flow_stats_get()
>>>>> >> >    stats_read()
>>>>> >> >  +->spin_lock remote CPU#1        ovs_flow_stats_get()
>>>>> >> >  |  <interrupted>                  stats_read()
>>>>> >> >  |  ...                       +-->  spin_lock remote CPU#0
>>>>> >> >  |                            |     <interrupted>
>>>>> >> >  |  ovs_flow_stats_update()   |     ...
>>>>> >> >  |   spin_lock local CPU#0 <--+     ovs_flow_stats_update()
>>>>> >> >  +---------------------------------- spin_lock local CPU#1
>>>>> >> >
>>>>> >> > This patch disables BH for both cases fixing the deadlocks.
>>>>> >>
>>>>> >> This bug is already fixed in OVS.
>>>>> >
>>>>> > Could you point me to the commit? I am not finding anything
>>>>> > recent.
>>>>>
>>>>> This is the commit:
>>>>>
>>>>> commit 9d73c9cac76ba557fdac4a89c1b7eafe132b85a3
>>>>> Author: Pravin B Shelar <pshelar@...ira.com>
>>>>> Date:   Tue Dec 17 15:43:30 2013 -0800
>>>>>
>>>>>     datapath: Fix deadlock during stats update.
>>>>>
>>>>> I thought that I had sent it in the most recent batch of changes for
>>>>> net but it looks like I missed it.
>>>>
>>>> That commit is incomplete. Look at the scenario #2 which I explain
>>>> why it is needed to disable bh for all cpus and not just local ones.
>>>
>>> OK, I understand the second problem now. OVS master (which I am
>>> currently working to cross-port to net-next) uses a different strategy
>>> that also always disables bottom halves for a different reason. Since
>>> I forgot to send the original patch, maybe we can just apply this one
>>> to net instead and use the new stuff directly everywhere else.
>>>
>>
>> The ovs patch fixes locking issue.
>>
>> local cpu check is optimization for better latency which can be
>> significant on large SMP system. I think we should fix lockdep rather
>> than adding latency for ovs packet processing. But all this is going
>> to change with NUMA stats anyways, so I do not have any problem
>> pushing this new patch.
>
> I think there is actually a possible deadlock here, not just a lockdep
> warning. If you look at Flavio's diagram, it's a rather complicated
> series of events that would be necessary but it seems theoretically
> possible.

Right, I missed v2 commit msg. This patch is required to fix all deadlock cases.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html