linux-kernel - Re: bonding: link state question

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Sat, 7 Aug 2021 20:09:31 -0400
From:   Jonathan Toppins <jtoppins@...hat.com>
To:     Jay Vosburgh <jay.vosburgh@...onical.com>
Cc:     "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        Veaceslav Falico <vfalico@...il.com>,
        Andy Gospodarek <andy@...yhouse.net>,
        "David S. Miller" <davem@...emloft.net>,
        Jakub Kicinski <kuba@...nel.org>,
        LKML <linux-kernel@...r.kernel.org>
Subject: Re: bonding: link state question

On 8/7/21 6:42 PM, Jay Vosburgh wrote:
> Jonathan Toppins <jtoppins@...hat.com> wrote:
> 
>> Is there any reason why bonding should have an operstate of up when none
>> of its slaves are in an up state? In this particular scenario it seems
>> like the bonding device should at least assert NO-CARRIER, thoughts?
>>
>> $ ip -o -d link show | grep "bond5"
>> 2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc
>> fq_codel master bond5 state DOWN mode DEFAULT group default qlen 1000\
>> link/ether 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68
>> maxmtu 9000 \    bond_slave state ACTIVE mii_status UP link_failure_count
>> 0 perm_hwaddr 8c:8c:aa:f8:62:16 queue_id 0 numtxqueues 1 numrxqueues 1
>> gso_max_size 65536 gso_max_segs 65535
>> 41: bond5: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue
>> state UP mode DEFAULT group default qlen 1000\    link/ether
>> 8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu
>> 65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0
>> peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none
> 
> 	I'm going to speculate that your problem is that miimon and
> arp_interval are both 0, and the bond then doesn't have any active
> mechanism to monitor the link state of its interfaces.  There might be a
> warning in dmesg to this effect.
> 
> 	Do you see what you'd consider to be correct behavior if miimon
> is set to 100?
> 

setting miimon = 100 does appear to fix it.

It is interesting that there is no link monitor on by default. For 
example when I enslave enp0s31f6 to a new bond with miimon == 0, 
enp0s31f6 starts admin down and will never de-assert NO-CARRIER the bond 
always results in an operstate of up. It seems like miimon = 100 should 
be the default since some modes cannot use arpmon.

Thank you for the discussion, see below for the steps taken.

$ sudo ip link set dev enp0s31f6 nomaster
$ sudo ip link add dev bond6 type bond mode balance-xor
$ sudo ip -o -d link set dev bond6 up
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc 
noqueue state DOWN mode DEFAULT group default qlen 1000\    link/ether 
3e:12:01:8a:ed:b1 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535
$ ip -o -d link show dev enp0s31f6
2: enp0s31f6: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN 
mode DEFAULT group default qlen 1000\    link/ether 8c:8c:aa:f8:62:16 
brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 9000 numtxqueues 1 
numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
$ sudo ip -o -d link set dev enp0s31f6 master bond6
$ ip -o -d link show dev bond6
62: bond6: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc 
noqueue state UP mode DEFAULT group default qlen 1000\    link/ether 
8c:8c:aa:f8:62:16 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535
$ sudo ip link set dev enp0s31f6 nomaster
$ ip -o -d link show dev bond6
62: bond6: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc 
noqueue state DOWN mode DEFAULT group default qlen 1000\    link/ether 
ae:b8:6e:b3:ca:3f brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 
65535 \    bond mode balance-xor miimon 0 updelay 0 downdelay 0 
peer_notify_delay 0 use_carrier 1 arp_interval 0 arp_validate none 
arp_all_targets any primary_reselect always fail_over_mac none 
xmit_hash_policy layer2 resend_igmp 1 num_grat_arp 1 all_slaves_active 0 
min_links 0 lp_interval 1 packets_per_slave 1 lacp_rate slow ad_select 
stable tlb_dynamic_lb 1 numtxqueues 16 numrxqueues 16 gso_max_size 65536 
gso_max_segs 65535