netdev - Fwd: [PATCH] bonding/802.3ad: fix slave initialization states race

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAJYOGF8LDwbZXXeEioKAtx=0rq9eZBxFYuRfF3jdFCDUGnJ-Rg@mail.gmail.com>
Date:   Wed, 18 Sep 2019 21:27:05 +0300
From:   Алексей Захаров 
        <zaharov@...ectel.ru>
To:     jay.vosburgh@...onical.com, netdev@...r.kernel.org
Subject: Fwd: [PATCH] bonding/802.3ad: fix slave initialization states race

> >Once a while, one of 802.3ad slaves fails to initialize and hangs in
> >BOND_LINK_FAIL state. Commit 334031219a84 ("bonding/802.3ad: fix slave
> >link initialization transition states") checks slave->last_link_up. But
> >link can still hang in weird state.
> >After physical link comes up it sends first two LACPDU messages and
> >doesn't work properly after that. It doesn't send or receive LACPDU.
> >Once it happens, the only message in dmesg is:
> >bond1: link status up again after 0 ms for interface eth2
>
>         I believe this message indicates that the slave entered
> BOND_LINK_FAIL state, but downdelay was not set.  The _FAIL state is
> really for managing the downdelay expiration, and a slave should not be
> in that state (outside of a brief transition entirely within
> bond_miimon_inspect) if downdelay is 0.
That's true, downdelay was set to 0, we only use updelay 500.
Does it mean, that the bonding driver shouldn't set slave to FAIL
state in this case?

>
> >This behavior can be reproduced (not every time):
> >1. Set slave link down
> >2. Wait for 1-3 seconds
> >3. Set slave link up
> >
> >The fix is to check slave->link before setting it to BOND_LINK_FAIL or
> >BOND_LINK_DOWN state. If got invalid Speed/Dupex values and link is in
> >BOND_LINK_UP state, mark it as BOND_LINK_FAIL; otherwise mark it as
> >BOND_LINK_DOWN.
> >
> >Fixes: 334031219a84 ("bonding/802.3ad: fix slave link initialization
> >transition states")
> >Signed-off-by: Aleksei Zakharov <zakharov.a.g@...dex.ru>
> >---
> > drivers/net/bonding/bond_main.c | 2 +-
> > 1 file changed, 1 insertion(+), 1 deletion(-)
> >
> >diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
> >index 931d9d935686..a28776d8f33f 100644
> >--- a/drivers/net/bonding/bond_main.c
> >+++ b/drivers/net/bonding/bond_main.c
> >@@ -3135,7 +3135,7 @@ static int bond_slave_netdev_event(unsigned long event,
> >                */
> >               if (bond_update_speed_duplex(slave) &&
> >                   BOND_MODE(bond) == BOND_MODE_8023AD) {
> >-                      if (slave->last_link_up)
> >+                      if (slave->link == BOND_LINK_UP)
> >                               slave->link = BOND_LINK_FAIL;
> >                       else
> >                               slave->link = BOND_LINK_DOWN;
>
>         Is the core problem here that slaves are reporting link up, but
> returning invalid values for speed and/or duplex?  If so, what network
> device are you testing with that is exhibiting this behavior?
That's true, because link becomes FAIL right in this block of code.
We use Mellanox ConnectX-3 Pro nic.

>
>         If I'm not mistaken, there have been several iterations of
> hackery on this block of code to work around this same problem, and each
> time there's some corner case that still doesn't work.
As i can see, commit 4d2c0cda0744 ("bonding: speed/duplex update at
NETDEV_UP event")
introduced BOND_LINK_DOWN state if update speed/duplex failed.

Commit ea53abfab960 ("bonding/802.3ad: fix link_failure_count tracking")
changed DOWN state to FAIL.

Commit 334031219a84 ("bonding/802.3ad: fix slave link initialization
transition states")
implemented different new state for different current states, but it
was based on slave->last_link_up.
In our case slave->last_link_up !=0 when this code runs. But, slave is
not in UP state at the moment. It becomes
FAIL and hangs in this state.
So, it looks like checking if slave is in UP mode is more appropriate
here. At least it works in our case.

There was one more commit 12185dfe4436 ("bonding: Force slave speed
check after link state recovery for 802.3ad")
but it doesn't help in our case.

>
>         As Davem asked last time around, is the real problem that device
> drivers report carrier up but supply invalid speed and duplex state?
Probably, but I'm not quite sure right now. We didn't face this issue
before 4d2c0cda0744 and ea53abfab960
commits.

>
>         -J
>
> ---
>         -Jay Vosburgh, jay.vosburgh@...onical.com



--
Best Regards,
Aleksei Zakharov
System administrator