netdev - Re: [RFC] net/hsr: Add support for IEC 62439-3 High-availability Seamless Redundancy

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <4FBE6B5E.3060403@xdin.com>
Date:	Thu, 24 May 2012 17:09:53 +0000
From:	Arvid Brodin <Arvid.Brodin@...n.com>
To:	Stephen Hemminger <shemminger@...tta.com>
CC:	"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
	"David S. Miller" <davem@...emloft.net>,
	Bruno Ferreira <balferreira@...glemail.com>,
	Christian Borntraeger <borntraeger@...ibm.com>,
	Herbert Xu <herbert@...dor.apana.org.au>
Subject: Re: [RFC] net/hsr: Add support for IEC 62439-3 High-availability
 Seamless Redundancy

On 2012-05-14 20:28, Stephen Hemminger wrote:
> On Mon, 14 May 2012 18:11:44 +0000
> Arvid Brodin <Arvid.Brodin@...n.com> wrote:
> 
>> On 2012-03-27 15:20, Arvid Brodin wrote:
>>> Hi!
>>
>> *snip*
>>>
>>> 2) I have a locking problem that I haven't managed to figure out. This happens
>>>    the first time I send any packet (hsr_dev_xmit() in hsr_device.c:121, called
>>>    from hsr_device.c:147). It happens even if I set skb2 to NULL (i.e. only send
>>>    one copy):
>>>
>>> =============================================
>>> [ INFO: possible recursive locking detected ]
>>> 2.6.37 #118
>>> ---------------------------------------------
>>> swapper/0 is trying to acquire lock:
>>>  (_xmit_ETHER#2){+.-...}, at: [<901bf38e>] sch_direct_xmit+0x24/0x152
>>>
>>> but task is already holding lock:
>>>  (_xmit_ETHER#2){+.-...}, at: [<901b4d1a>] dev_queue_xmit+0x31e/0x3cc
>>>
>>> other info that might help us debug this:
>>> 4 locks held by swapper/0:
>>>  #0:  (&n->timer){+.-...}, at: [<9002bc20>] run_timer_softirq+0x98/0x184
>>>  #1:  (rcu_read_lock_bh){.+....}, at: [<901b49fc>] dev_queue_xmit+0x0/0x3cc
>>>  #2:  (_xmit_ETHER#2){+.-...}, at: [<901b4d1a>] dev_queue_xmit+0x31e/0x3cc
>>>  #3:  (rcu_read_lock_bh){.+....}, at: [<901b49fc>] dev_queue_xmit+0x0/0x3cc
>>>
>>> stack backtrace:
>>> Call trace:
>>>  [<9001c640>] dump_stack+0x18/0x20
>>>  [<90040eac>] validate_chain+0x40c/0x9ac
>>>  [<90041a58>] __lock_acquire+0x60c/0x670
>>>  [<90042f32>] lock_acquire+0x3a/0x48
>>>  [<902201a4>] _raw_spin_lock+0x20/0x44
>>>  [<901bf38e>] sch_direct_xmit+0x24/0x152
>>>  [<901b4c14>] dev_queue_xmit+0x218/0x3cc
>>>  [<9021c2e0>] slave_xmit+0x10/0x14
>>>  [<9021c540>] hsr_dev_xmit+0x88/0x8c
>>>  [<901b4942>] dev_hard_start_xmit+0x3c6/0x480
>>>  [<901b4d34>] dev_queue_xmit+0x338/0x3cc
>>>  [<901e3cd8>] arp_xmit+0x8/0xc
>>>  [<901e4436>] arp_send+0x2a/0x2c
>>>  [<901e4e74>] arp_solicit+0x15c/0x170
>>>  [<901bad0c>] neigh_timer_handler+0x1c0/0x204
>>>  [<9002bc8a>] run_timer_softirq+0x102/0x184
>>>  [<900287d8>] __do_softirq+0x64/0xe0
>>>  [<9002896a>] do_softirq+0x26/0x48
>>>  [<90028a66>] irq_exit+0x2e/0x64
>>>  [<90019f16>] do_IRQ+0x46/0x5c
>>>  [<90018428>] irq_level0+0x18/0x60
>>>  [<9021cc16>] rest_init+0x72/0x98
>>>  [<9000063c>] start_kernel+0x21c/0x258
>>>  [<00000000>] 0x0
>>>
>>>    Any idea why this happens? I need help!
>>
>>
>> I've spent a few days digging into this and the key apparently is NETIF_F_LLTX.
>>
>> The problem seems to be that HARD_TX_LOCK is called more than once, first for my virtual
>> hsr device and then, recursively, for each of the slaves in turn. (At least that's where
>> lockdep complains - at __netif_tx_lock(), that is.)
>>
>> At first I just could not understand why both the VLAN and the bonding code got away with
>> recursive calls to dev_queue_xmit() but I didn't. After some gooling (a lot, actually) I
>> found some references to the NETIF_F_LLTX flag (here's one:
>> http://lwn.net/Articles/121566/). I realised both VLAN and bonding code set this flag. And
>> sure enough, if I set it for my hsr device lockdep does not complain any more.
>>
>> But NETIF_F_LLTX is described as deprecated in both netdevice.h and in
>> Documentation/networking/netdevices.txt. Is there an alternative solution that I should
>> use instead?
>>
>> (To recap, a hsr device is a virtual device which uses two Ethernet devices as slaves.
>> This gives redundancy with instant failover, and since nodes are connected in a ring
>> topology, uses less cabling than duplication.)
>>
> 
> LLTX is deprecated (ie should not be used) for physical devices.
> 
> Also, for virtual devices, there should be non transmit queue, this
> causes mulit-queue lockless semantics to be preserved as the call passes
> through the virtual device.

(First: apologies for my late reply; your emails doesn't get through to me for some reason.)

So does this mean that it is OK to use LLTX for virtual devices? My virtual device has
zero queue length (no transmit queue), but since it calls dev_queue_xmit for its slaves, I
still get recursive locking if I don't set LLTX (just like vlan and bonding does).

-- 
Arvid Brodin | Consultant (Linux)
XDIN AB | Jan Stenbecks Torg 17 | SE-164 40 Kista | Sweden | xdin.com--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html