Date:   Thu, 16 Aug 2018 15:29:56 +1000
From:   Daniel Axtens <dja@...ens.net>
To:     netdev@...r.kernel.org, jay.vosburgh@...onical.com,
        michael.chan@...adcom.com
Subject: bnxt: card intermittently hanging and dropping link

Hi Michael,

I have some user reports of issues with a Broadcom 57412 card with the
card intermittently hanging and dropping the link.

The problem has been observed on a Dell server with an Ubuntu 4.13
kernel (bnxt_en version 1.7.0) and with an Ubuntu 4.15 kernel (bnxt_en
version 1.8.0). It seems to occur while mounting an XFS volume over
iSCSI, although running blkid on the partition succeeds, and it seems to
be a different volume each time.

I've included an excerpt from the kernel log below.

It seems that other people have reported this with a RHEL7 kernel - I
have no idea what driver version that would be running, or what workload
they were operating, but the warnings and messages look the same:
https://www.dell.com/community/PowerEdge-Hardware-General/Critical-network-bnxt-en-module-crashes-on-14G-servers/td-p/6031769/highlight/true
The forum poster reported that disconnecting the card from power for 5
minutes was sufficient to get things working again and I have asked our
user to test that.

Is this a known issue?

Regards,
Daniel

[ 2662.526151] scsi host14: iSCSI Initiator over TCP/IP
[ 2662.538110] scsi host15: iSCSI Initiator over TCP/IP
[ 2662.547350] scsi host16: iSCSI Initiator over TCP/IP
[ 2662.554660] scsi host17: iSCSI Initiator over TCP/IP
[ 2662.813860] scsi 15:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
[ 2662.813888] scsi 14:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
[ 2662.813972] scsi 16:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
[ 2662.814322] sd 15:0:0:1: Attached scsi generic sg1 type 0
[ 2662.814553] sd 14:0:0:1: Attached scsi generic sg2 type 0
[ 2662.814554] scsi 17:0:0:1: Direct-Access     PURE     FlashArray       8888 PQ: 0 ANSI: 6
[ 2662.814612] sd 16:0:0:1: Attached scsi generic sg3 type 0
[ 2662.815081] sd 15:0:0:1: [sdb] 10737418240 512-byte logical blocks: (5.50 TB/5.00 TiB)
[ 2662.815195] sd 15:0:0:1: [sdb] Write Protect is off
[ 2662.815197] sd 15:0:0:1: [sdb] Mode Sense: 43 00 00 08
[ 2662.815229] sd 14:0:0:1: [sdc] 10737418240 512-byte logical blocks: (5.50 TB/5.00 TiB)
[ 2662.815292] sd 17:0:0:1: Attached scsi generic sg4 type 0
[ 2662.815342] sd 14:0:0:1: [sdc] Write Protect is off
[ 2662.815343] sd 14:0:0:1: [sdc] Mode Sense: 43 00 00 08
[ 2662.815419] sd 15:0:0:1: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 2662.815447] sd 16:0:0:1: [sdd] 10737418240 512-byte logical blocks: (5.50 TB/5.00 TiB)
[ 2662.815544] sd 14:0:0:1: [sdc] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 2662.815614] sd 16:0:0:1: [sdd] Write Protect is off
[ 2662.815615] sd 16:0:0:1: [sdd] Mode Sense: 43 00 00 08
[ 2662.815882] sd 16:0:0:1: [sdd] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 2662.816188] sd 17:0:0:1: [sde] 10737418240 512-byte logical blocks: (5.50 TB/5.00 TiB)
[ 2662.816298] sd 17:0:0:1: [sde] Write Protect is off
[ 2662.816300] sd 17:0:0:1: [sde] Mode Sense: 43 00 00 08
[ 2662.816502] sd 17:0:0:1: [sde] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[ 2662.820080] sd 15:0:0:1: [sdb] Attached SCSI disk
[ 2662.820594] sd 14:0:0:1: [sdc] Attached SCSI disk
[ 2662.820995] sd 17:0:0:1: [sde] Attached SCSI disk
[ 2662.821176] sd 16:0:0:1: [sdd] Attached SCSI disk
[ 2662.913642] device-mapper: multipath round-robin: version 1.2.0 loaded
[ 2663.954001] XFS (dm-2): Mounting V5 Filesystem
[ 2673.186083]  connection3:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295558209, last ping 4295559460, now 4295560768
[ 2673.186135]  connection3:0: detected conn error (1022)
[ 2673.186137]  connection2:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295558209, last ping 4295559460, now 4295560768
[ 2673.186168]  connection2:0: detected conn error (1022)
[ 2673.186170]  connection1:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295558209, last ping 4295559460, now 4295560768
[ 2673.186211]  connection1:0: detected conn error (1022)
[ 2674.209870]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295558463, last ping 4295559720, now 4295561024
[ 2674.209924]  connection4:0: detected conn error (1022)
[ 2678.560630]  session1: session recovery timed out after 5 secs
[ 2678.560641]  session2: session recovery timed out after 5 secs
[ 2678.560647]  session3: session recovery timed out after 5 secs
[ 2678.951453] device-mapper: multipath: Failing path 8:32.
[ 2678.951509] device-mapper: multipath: Failing path 8:48.
[ 2678.951548] device-mapper: multipath: Failing path 8:16.
[ 2679.584302]  session4: session recovery timed out after 5 secs
[ 2679.584313] sd 17:0:0:1: rejecting I/O to offline device
[ 2679.584356] sd 17:0:0:1: [sde] killing request
[ 2679.584362] sd 17:0:0:1: rejecting I/O to offline device
[ 2679.584392] sd 17:0:0:1: [sde] killing request
[ 2679.584401] sd 17:0:0:1: [sde] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[ 2679.584402] sd 17:0:0:1: [sde] CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 00 08 00 00
[ 2679.584404] print_req_error: I/O error, dev sde, sector 0
[ 2679.584436] sd 17:0:0:1: [sde] FAILED Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[ 2679.584438] sd 17:0:0:1: [sde] CDB: Write(16) 8a 00 00 00 00 00 80 10 7f 1e 00 00 10 00 00 00
[ 2679.584439] print_req_error: I/O error, dev sde, sector 2148564766
[ 2679.584464] device-mapper: multipath: Failing path 8:64.
[ 2679.584472] print_req_error: I/O error, dev dm-2, sector 2148564766
[ 2679.584562] XFS (dm-2): xfs_do_force_shutdown(0x1) called from line 1217 of file /build/linux-hwe-eGX82t/linux-hwe-4.15.0/fs/xfs/xfs_buf.c.  Return address = 0x00000000d95bdaac
[ 2679.584576] XFS (dm-2): I/O Error Detected. Shutting down filesystem
[ 2679.584595] XFS (dm-2): Please umount the filesystem and rectify the problem(s)
[ 2679.584613] XFS (dm-2): metadata I/O error: block 0x80107f1e ("xlog_bwrite") error 5 numblks 8192
[ 2679.584796] XFS (dm-2): failed to locate log tail
[ 2679.584798] XFS (dm-2): log mount/recovery failed: error -5
[ 2679.584910] XFS (dm-2): log mount failed
[ 2680.589182] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2680.589232] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2680.589250] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2680.589584] print_req_error: I/O error, dev dm-2, sector 2
[ 2680.589611] EXT4-fs (dm-2): unable to read superblock
[ 2680.589781] print_req_error: I/O error, dev dm-2, sector 2
[ 2680.589804] EXT4-fs (dm-2): unable to read superblock
[ 2680.590019] print_req_error: I/O error, dev dm-2, sector 2
[ 2680.590078] EXT4-fs (dm-2): unable to read superblock
[ 2680.590298] print_req_error: I/O error, dev dm-2, sector 0
[ 2680.590358] SQUASHFS error: squashfs_read_data failed to read block 0x0
[ 2680.590390] squashfs: SQUASHFS error: unable to read squashfs_super_block
[ 2681.592134] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2681.592217] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2681.592437] EXT4-fs (dm-2): unable to read superblock
[ 2681.592576] EXT4-fs (dm-2): unable to read superblock
[ 2681.592707] EXT4-fs (dm-2): unable to read superblock
[ 2681.592839] SQUASHFS error: squashfs_read_data failed to read block 0x0
[ 2681.592858] squashfs: SQUASHFS error: unable to read squashfs_super_block
[ 2682.661952] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2682.667509] Buffer I/O error on dev dm-2, logical block 10737418112, async page read
[ 2682.667533] Buffer I/O error on dev dm-2, logical block 10737418113, async page read
[ 2682.667552] Buffer I/O error on dev dm-2, logical block 10737418114, async page read
[ 2682.667571] Buffer I/O error on dev dm-2, logical block 10737418115, async page read
[ 2682.667612] Buffer I/O error on dev dm-2, logical block 10737418116, async page read
[ 2682.667632] Buffer I/O error on dev dm-2, logical block 10737418117, async page read
[ 2682.667653] Buffer I/O error on dev dm-2, logical block 10737418118, async page read
[ 2682.911295] ------------[ cut here ]------------
[ 2682.911319] NETDEV WATCHDOG: eth4 (bnxt_en): transmit queue 0 timed out
[ 2682.911350] WARNING: CPU: 23 PID: 0 at /build/linux-hwe-eGX82t/linux-hwe-4.15.0/net/sched/sch_generic.c:323 dev_watchdog+0x222/0x230
[ 2682.911352] Modules linked in: dm_round_robin binfmt_misc cfg80211 xt_set xt_multiport iptable_raw iptable_mangle ip_set_hash_ip ip_set_hash_net ip_set ipip tunnel4 ip_tunnel veth xt_statistic xt_nat xt_recent ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_addrtype ip_vs ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_comment ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables xt_mark iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables xt_conntrack x_tables nf_nat nf_conntrack_netlink nfnetlink nf_conntrack xfrm_user xfrm_algo aufs bonding intel_rapl skx_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel ipmi_ssif kvm dcdbas irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd glue_helper cryptd intel_cstate
[ 2682.911430]  intel_rapl_perf mei_me lpc_ich shpchp mei ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter mac_hid ib_iser rdma_cm iw_cm ib_cm ib_core iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi br_netfilter bridge stp llc dm_multipath autofs4 xfs bnx2x ptp mgag200 i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops pps_core bnxt_en mdio drm devlink libcrc32c ahci libahci
[ 2682.911482] CPU: 23 PID: 0 Comm: swapper/23 Not tainted 4.15.0-24-generic #26~16.04.1-Ubuntu
[ 2682.911483] Hardware name: Dell Inc. PowerEdge R640/0W23H8, BIOS 1.3.7 02/08/2018
[ 2682.911487] RIP: 0010:dev_watchdog+0x222/0x230
[ 2682.911489] RSP: 0018:ffff91eaeeec3e68 EFLAGS: 00010282
[ 2682.911492] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[ 2682.911493] RDX: 0000000000040400 RSI: 00000000000000f6 RDI: 0000000000000300
[ 2682.911495] RBP: ffff91eaeeec3e98 R08: 0000000000000001 R09: 000000000000071b
[ 2682.911497] R10: 0000000000000000 R11: 000000000000071b R12: 000000000000004a
[ 2682.911499] R13: ffff91aae1522000 R14: ffff91aae1522478 R15: ffff91aae066dbc0
[ 2682.911505] FS:  0000000000000000(0000) GS:ffff91eaeeec0000(0000) knlGS:0000000000000000
[ 2682.911507] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2682.911509] CR2: 00007f86f4058384 CR3: 00000033bd20a005 CR4: 00000000007606e0
[ 2682.911511] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2682.911513] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2682.911514] PKRU: 55555554
[ 2682.911515] Call Trace:
[ 2682.911518]  <IRQ>
[ 2682.911524]  ? dev_deactivate_queue.constprop.33+0x60/0x60
[ 2682.911534]  call_timer_fn+0x32/0x140
[ 2682.911539]  run_timer_softirq+0x1ed/0x440
[ 2682.911543]  ? ktime_get+0x3e/0xa0
[ 2682.911550]  ? lapic_next_deadline+0x26/0x30
[ 2682.911559]  __do_softirq+0xf2/0x288
[ 2682.911565]  irq_exit+0xb6/0xc0
[ 2682.911569]  smp_apic_timer_interrupt+0x71/0x140
[ 2682.911574]  apic_timer_interrupt+0x84/0x90
[ 2682.911575]  </IRQ>
[ 2682.911583] RIP: 0010:cpuidle_enter_state+0xa7/0x300
[ 2682.911585] RSP: 0018:ffffbac30c6efe60 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff11
[ 2682.911587] RAX: ffff91eaeeee2880 RBX: 0000000000000003 RCX: 000000000000001f
[ 2682.911589] RDX: 0000000000000000 RSI: 000000002f84dd9f RDI: 0000000000000000
[ 2682.911591] RBP: ffffbac30c6efe98 R08: ffff91eaeeee16a4 R09: 0000000000000018
[ 2682.911592] R10: ffffbac30c6efe30 R11: 000000000000266e R12: 0000000000000003
[ 2682.911594] R13: ffffdac2ff6c3138 R14: ffffffff85171d58 R15: 00000270a5abd299
[ 2682.911599]  cpuidle_enter+0x17/0x20
[ 2682.911606]  call_cpuidle+0x23/0x40
[ 2682.911610]  do_idle+0x197/0x200
[ 2682.911614]  cpu_startup_entry+0x73/0x80
[ 2682.911618]  start_secondary+0x1ab/0x200
[ 2682.911623]  secondary_startup_64+0xa5/0xb0
[ 2682.911626] Code: 38 00 49 63 4e e8 eb 92 4c 89 ef c6 05 76 f3 d8 00 01 e8 42 37 fd ff 89 d9 48 89 c2 4c 89 ee 48 c7 c7 78 9c d9 84 e8 1e b3 80 ff <0f> 0b eb c0 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 55 48 
[ 2682.911687] ---[ end trace a09e66b1b45bbbba ]---
[ 2682.911693] bnxt_en 0000:3b:00.0 eth4: TX timeout detected, starting reset task!
[ 2683.782496] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2683.783061] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free tx failed. rc:-1
[ 2684.634557] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2684.635120] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free tx failed. rc:-1
[ 2685.474679] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2685.475188] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free tx failed. rc:-1
[ 2686.323579] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2686.324072] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free tx failed. rc:-1
[ 2686.691126] print_req_error: 27 callbacks suppressed
[ 2686.691127] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2686.691604] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2686.692001] buffer_io_error: 10 callbacks suppressed
[ 2686.692002] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2686.693881] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2686.694399] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2686.694856] Buffer I/O error on dev dm-2, logical block 10737418112, async page read
[ 2686.695298] print_req_error: I/O error, dev dm-2, sector 10737418113
[ 2686.695720] Buffer I/O error on dev dm-2, logical block 10737418113, async page read
[ 2686.696135] print_req_error: I/O error, dev dm-2, sector 10737418114
[ 2686.696545] Buffer I/O error on dev dm-2, logical block 10737418114, async page read
[ 2686.696962] print_req_error: I/O error, dev dm-2, sector 10737418115
[ 2686.697379] Buffer I/O error on dev dm-2, logical block 10737418115, async page read
[ 2686.697795] print_req_error: I/O error, dev dm-2, sector 10737418116
[ 2686.698223] Buffer I/O error on dev dm-2, logical block 10737418116, async page read
[ 2686.698637] print_req_error: I/O error, dev dm-2, sector 10737418117
[ 2686.699044] Buffer I/O error on dev dm-2, logical block 10737418117, async page read
[ 2686.699456] print_req_error: I/O error, dev dm-2, sector 10737418118
[ 2686.699869] Buffer I/O error on dev dm-2, logical block 10737418118, async page read
[ 2686.700291] Buffer I/O error on dev dm-2, logical block 10737418119, async page read
[ 2687.162049] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2687.162552] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free tx failed. rc:-1
[ 2688.001911] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2688.002335] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2688.836936] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2688.837357] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2689.670237] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2689.670617] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2690.519733] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2690.520109] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2691.142178] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2691.358094] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2691.358503] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2692.210710] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2692.211091] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2693.052607] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2693.053029] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2693.907808] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2693.908233] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2694.761187] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2694.761626] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2695.609620] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2695.610055] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2696.450508] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2696.450939] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2697.293569] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2697.294007] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2698.138408] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2698.138854] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2698.992525] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2698.992979] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2699.605100] print_req_error: 12 callbacks suppressed
[ 2699.605111] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2699.605680] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2699.606212] buffer_io_error: 8 callbacks suppressed
[ 2699.606213] Buffer I/O error on dev dm-2, logical block 1342177264, async page read
[ 2699.608144] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2699.608722] print_req_error: I/O error, dev dm-2, sector 10737418112
[ 2699.609168] Buffer I/O error on dev dm-2, logical block 10737418112, async page read
[ 2699.609581] print_req_error: I/O error, dev dm-2, sector 10737418113
[ 2699.609987] Buffer I/O error on dev dm-2, logical block 10737418113, async page read
[ 2699.610422] print_req_error: I/O error, dev dm-2, sector 10737418114
[ 2699.610855] Buffer I/O error on dev dm-2, logical block 10737418114, async page read
[ 2699.611294] print_req_error: I/O error, dev dm-2, sector 10737418115
[ 2699.611727] Buffer I/O error on dev dm-2, logical block 10737418115, async page read
[ 2699.612133] print_req_error: I/O error, dev dm-2, sector 10737418116
[ 2699.612572] Buffer I/O error on dev dm-2, logical block 10737418116, async page read
[ 2699.612991] print_req_error: I/O error, dev dm-2, sector 10737418117
[ 2699.613407] Buffer I/O error on dev dm-2, logical block 10737418117, async page read
[ 2699.613836] print_req_error: I/O error, dev dm-2, sector 10737418118
[ 2699.614261] Buffer I/O error on dev dm-2, logical block 10737418118, async page read
[ 2699.614803] Buffer I/O error on dev dm-2, logical block 10737418119, async page read
[ 2699.845546] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2699.846018] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2700.688086] bnxt_en 0000:3b:00.0 eth4: Resp cmpl intr err msg: 0x51
[ 2700.688563] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free rx failed. rc:-1
[ 2701.523722] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb6b} len:0
[ 2701.524174] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2702.368502] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb6c} len:0
[ 2702.368952] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2703.222505] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb6d} len:0
[ 2703.222945] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2704.078981] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb6e} len:0
[ 2704.079748] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2704.913958] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb6f} len:0
[ 2704.914393] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2705.764961] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb70} len:0
[ 2705.765400] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2706.613411] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb71} len:0
[ 2706.613839] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2707.454748] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x51 0xb72} len:0
[ 2707.455187] bnxt_en 0000:3b:00.0 eth4: hwrm_ring_free cp failed. rc:-1
[ 2708.306362] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x61 0xb73} len:0
[ 2709.150419] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x50 0xb74} len:0
[ 2709.150880] bnxt_en 0000:3b:00.0 eth4: Invalid ring
[ 2709.151318] bnxt_en 0000:3b:00.0 eth4: hwrm ring alloc failure rc: ffffffff
[ 2709.989131] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x44 0xb75} len:0
[ 2709.989635] bnxt_en 0000:3b:00.0 eth4: hwrm vnic set tpa failure rc for vnic 0: ffffffff
[ 2710.826136] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0xb1 0xb76} len:0
[ 2710.826671] bnxt_en 0000:3b:00.0 eth4: bnxt_init_nic err: ffffffff
[ 2710.831139] bnxt_en 0000:3b:00.0 eth4: nic open fail (rc: ffffffff)
[ 2711.680497] bnxt_en 0000:3b:00.0 eth4: Error (timeout: 500) msg {0x20 0xb77} len:0
[ 2711.694743] bond1: link status definitely down for interface eth4, disabling it
[ 2711.694745] bond1: making interface eth5 the new active one
[ 2712.521953] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa2} len:0
[ 2713.363445] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x93 0xaa3} len:0
[ 2713.364096] bnxt_en 0000:3b:00.1 eth5: HWRM cfa l2 rx mask failure rc: ffffffff
[ 2714.213610] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa4} len:0
[ 2714.946281] device-mapper: multipath: Reinstating path 8:32.
[ 2714.946353] device-mapper: multipath: Reinstating path 8:48.
[ 2714.946409] device-mapper: multipath: Reinstating path 8:16.
[ 2715.052045] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa5} len:0
[ 2715.892140] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa6} len:0
[ 2715.987424] XFS (dm-2): Mounting V5 Filesystem
[ 2716.003187] XFS (dm-2): Starting recovery (logdev: internal)
[ 2716.018293] XFS (dm-2): Ending recovery (logdev: internal)
[ 2716.735471] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa7} len:0
[ 2717.575403] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa8} len:0
[ 2718.459295] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaa9} len:0
[ 2719.487382] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaaa} len:0
[ 2720.517072] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaab} len:0
[ 2721.525248] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaac} len:0
[ 2722.548765] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaad} len:0
[ 2723.572190] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaae} len:0
[ 2724.609428] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xaaf} len:0
[ 2724.882745]  connection4:0: ping timeout of 5 secs expired, recv timeout 5, last rx 4295571130, last ping 4295572416, now 4295573696
[ 2724.883488]  connection4:0: detected conn error (1022)
[ 2725.629310] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xab0} len:0
[ 2726.657752] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xab1} len:0
[ 2727.680069] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xab2} len:0
[ 2728.705973] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xab3} len:0
[ 2729.489387] bnxt_en 0000:3b:00.1 eth5: TX timeout detected, starting reset task!
[ 2729.726626] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x23 0xab4} len:0
...

[ 2744.875420] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2744.875775] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free tx failed. rc:-1
[ 2745.726024] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2745.726346] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free tx failed. rc:-1
[ 2746.579434] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2746.579709] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free tx failed. rc:-1
[ 2747.430915] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2747.431197] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free tx failed. rc:-1
[ 2748.285855] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2748.286132] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2749.141919] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2749.142170] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2749.986850] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2749.987190] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2750.821519] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2750.821848] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2751.657391] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2751.657709] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2752.512205] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2752.512543] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2753.352293] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2753.352613] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2754.187621] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2754.187946] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2755.022193] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2755.022524] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2755.856013] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2755.856329] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2756.694497] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2756.694840] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2757.527374] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2757.527732] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2758.363738] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2758.364115] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2759.215324] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2759.215701] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2760.061181] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2760.061552] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2760.898309] bnxt_en 0000:3b:00.1 eth5: Resp cmpl intr err msg: 0x51
[ 2760.898673] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free rx failed. rc:-1
[ 2761.732935] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xada} len:0
[ 2761.733303] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2762.573293] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xadb} len:0
[ 2762.573700] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2763.407002] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xadc} len:0
[ 2763.407411] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2764.255762] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xadd} len:0
[ 2764.256162] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2765.100554] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xade} len:0
[ 2765.100947] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2765.941308] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xadf} len:0
[ 2765.941712] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2766.793534] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xae0} len:0
[ 2766.794015] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2767.642247] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x51 0xae1} len:0
[ 2767.642630] bnxt_en 0000:3b:00.1 eth5: hwrm_ring_free cp failed. rc:-1
[ 2768.496229] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x61 0xae2} len:0
[ 2769.356587] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x50 0xae3} len:0
[ 2769.356939] bnxt_en 0000:3b:00.1 eth5: Invalid ring
[ 2769.357277] bnxt_en 0000:3b:00.1 eth5: hwrm ring alloc failure rc: ffffffff
[ 2770.215233] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x44 0xae4} len:0
[ 2770.215619] bnxt_en 0000:3b:00.1 eth5: hwrm vnic set tpa failure rc for vnic 0: ffffffff
[ 2771.073430] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0xb1 0xae5} len:0
[ 2771.073803] bnxt_en 0000:3b:00.1 eth5: bnxt_init_nic err: ffffffff
[ 2771.076758] bnxt_en 0000:3b:00.1 eth5: nic open fail (rc: ffffffff)
[ 2771.925191] bnxt_en 0000:3b:00.1 eth5: Error (timeout: 500) msg {0x20 0xae6} len:0
[ 2771.936019] bond1: link status definitely down for interface eth5, disabling it
[ 2771.936023] bond1: now running without any active interface!

Powered by blists - more mailing lists