netdev - Re: Problem with ixgbe and TX locked on one cpu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAEuXFEy89F_=J_PJMSbQ1fDcOEkkpz2vucq3LcToi7rTcZHcDA@mail.gmail.com>
Date:	Tue, 18 Oct 2011 11:57:33 -0700
From:	Jesse Brandeburg <jesse.brandeburg@...il.com>
To:	Paweł Staszewski <pstaszewski@...are.pl>
Cc:	Linux Network Development list <netdev@...r.kernel.org>,
	e1000-devel@...ts.sourceforge.net
Subject: Re: Problem with ixgbe and TX locked on one cpu

CC: e1000-devel

2011/10/14 Paweł Staszewski <pstaszewski@...are.pl>:
> Hello
>
> I have weird problem with ixgbe and irq affinity / rx-tx queue assignment

what application are you running, how are you using ixgbe? looks like
a router.  is something changing the skb->rx_queue entry (like
netfilter) or is there a layered device above ixgbe (bonding or ...) ?

why do your interrupts move after a long period? did you do it by
hand? we recommend disabling irqbalance and hand tuning interrupts
possibly with the set_irq_affinity.sh script.

> Statistics for my ethernet - ixgbe driver:
> ethtool -S eth4
> NIC statistics:
>     rx_packets: 5815535848808
>     tx_packets: 5811202378421
>     rx_bytes: 4791001750842200
>     tx_bytes: 4781190419358301
>     rx_pkts_nic: 5815535848827
>     tx_pkts_nic: 5811202378510
>     rx_bytes_nic: 4837563124411799
>     tx_bytes_nic: 4829987507084013
>     lsc_int: 8
>     tx_busy: 0
>     non_eop_descs: 0
>     rx_errors: 0
>     tx_errors: 0
>     rx_dropped: 0
>     tx_dropped: 0
>     multicast: 92494273
>     broadcast: 268718206
>     rx_no_buffer_count: 28829
>     collisions: 0
>     rx_over_errors: 0
>     rx_crc_errors: 0
>     rx_frame_errors: 0
>     hw_rsc_aggregated: 0
>     hw_rsc_flushed: 0
>     fdir_match: 0
>     fdir_miss: 0
>     rx_fifo_errors: 0
>     rx_missed_errors: 307051074
>     tx_aborted_errors: 0
>     tx_carrier_errors: 0
>     tx_fifo_errors: 0
>     tx_heartbeat_errors: 0
>     tx_timeout_count: 0
>     tx_restart_queue: 15926219
>     rx_long_length_errors: 298
>     rx_short_length_errors: 0
>     tx_flow_control_xon: 0
>     rx_flow_control_xon: 0
>     tx_flow_control_xoff: 0
>     rx_flow_control_xoff: 0
>     rx_csum_offload_errors: 54173917
>     alloc_rx_page_failed: 0
>     alloc_rx_buff_failed: 0
>     rx_no_dma_resources: 0
>     tx_queue_0_packets: 68694825
>     tx_queue_0_bytes: 9443750332
>     tx_queue_1_packets: 8410961
>     tx_queue_1_bytes: 2527763233
>     tx_queue_2_packets: 14411252
>     tx_queue_2_bytes: 1317132394
>     tx_queue_3_packets: 15013508147
>     tx_queue_3_bytes: 17364767277348
>     tx_queue_4_packets: 62779891
>     tx_queue_4_bytes: 63476596221
>     tx_queue_5_packets: 11176001
>     tx_queue_5_bytes: 2763600253
>     tx_queue_6_packets: 4416357
>     tx_queue_6_bytes: 611874984
>     tx_queue_7_packets: 8933405
>     tx_queue_7_bytes: 1837198524
>     tx_queue_8_packets: 13292669
>     tx_queue_8_bytes: 3241333510
>     tx_queue_9_packets: 10747236
>     tx_queue_9_bytes: 1805109931
>     tx_queue_10_packets: 5795935258380
>     tx_queue_10_bytes: 4763725304722245
>     tx_queue_11_packets: 12073934
>     tx_queue_11_bytes: 2982743045
>     tx_queue_12_packets: 10523764
>     tx_queue_12_bytes: 2637451199
>     tx_queue_13_packets: 12480552
>     tx_queue_13_bytes: 2434827407
>     tx_queue_14_packets: 7401777
>     tx_queue_14_bytes: 2413618099
>     tx_queue_15_packets: 8269270
>     tx_queue_15_bytes: 2854359576
>     rx_queue_0_packets: 361373769507
>     rx_queue_0_bytes: 298565751248279
>     rx_queue_1_packets: 369901571908
>     rx_queue_1_bytes: 303414679798160
>     rx_queue_2_packets: 362508961738
>     rx_queue_2_bytes: 299852439447157
>     rx_queue_3_packets: 363449272013
>     rx_queue_3_bytes: 299738390792515
>     rx_queue_4_packets: 361876234461
>     rx_queue_4_bytes: 297483366939732
>     rx_queue_5_packets: 361402926316
>     rx_queue_5_bytes: 297633876486533
>     rx_queue_6_packets: 362261522767
>     rx_queue_6_bytes: 298026696344647
>     rx_queue_7_packets: 361248593301
>     rx_queue_7_bytes: 296756459279986
>     rx_queue_8_packets: 361654143416
>     rx_queue_8_bytes: 298272433659520
>     rx_queue_9_packets: 362781764710
>     rx_queue_9_bytes: 298804803191595
>     rx_queue_10_packets: 361386593064
>     rx_queue_10_bytes: 297434987797644
>     rx_queue_11_packets: 369886597895
>     rx_queue_11_bytes: 302353350171712
>     rx_queue_12_packets: 361582732276
>     rx_queue_12_bytes: 298670408005971
>     rx_queue_13_packets: 365248093536
>     rx_queue_13_bytes: 302573023878287
>     rx_queue_14_packets: 366571142073
>     rx_queue_14_bytes: 302396739276514
>     rx_queue_15_packets: 362401929830
>     rx_queue_15_bytes: 299024344526029
>
> The problem is with queue 10
>     tx_queue_10_packets: 5795935258380
>     tx_queue_10_bytes: 4763725304722245
>
> as you can see most of the queue processing is used in queue 10
> Average difference is 1,854271229903958e-6  - compared to other queues
>
> and the problem is that almost all TX packet processing is on one CPU
> cat /proc/interrupts - in attached file
>
> Is this driver or kernel problem ?
>
> Kernel is: 2.6.38.2
>
> ixgbe driver is:
> ethtool -i eth4
> driver: ixgbe
> version: 3.2.9-k2
> firmware-version: 1.12-2
> bus-info: 0000:04:00.0
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html