| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-Id: <20090819135900.9cc50e73.billfink@mindspring.com>
Date: Wed, 19 Aug 2009 13:59:00 -0400
From: Bill Fink <billfink@...dspring.com>
To: Andrew Gallatin <gallatin@...i.com>
Cc: Jesse Barnes <jbarnes@...tuousgeek.org>,
"Brandeburg, Jesse" <jesse.brandeburg@...el.com>,
Neil Horman <nhorman@...driver.com>,
Brice Goglin <Brice.Goglin@...ia.fr>,
Linux Network Developers <netdev@...r.kernel.org>,
Yinghai Lu <yhlu.kernel@...il.com>
Subject: Re: Receive side performance issue with multi-10-GigE and NUMA
On Tue, 18 Aug 2009, Andrew Gallatin wrote:
> Bill Fink wrote:
>
> > < Latency: 0, Cache Line Size: 64 bytes
>
> <...>
>
> >> Latency: 0, Cache Line Size: 256 bytes
>
>
> A cache line size of 256 clearly seems wrong for a Xeon. I assume all
> devices on the SuperMicro show the same value?
I forgot to check that.
> > Interestingly, the "WC Enabled" is only indicated on the first two
>
> The WC is probably a red herring.
>
> What does ethtool -S show for the DMA write bandwidth of the
> NICs on the SuperMicro?
I've attached the full "ethtool -S" output from both the Asus and
SuperMicro systems. Here's just the bandwidth info:
Asus eth2:
[root@...est1 ~]# ethtool -S eth2
NIC statistics:
...
read_dma_bw_MBs: 1625
write_dma_bw_MBs: 1599
read_write_dma_bw_MBs: 3192
SuperMicro eth2 (on 5520 connected to NUMA node 1):
[root@...ntest1 ~]# ethtool -S eth2
NIC statistics:
...
read_dma_bw_MBs: 1624
write_dma_bw_MBs: 1605
read_write_dma_bw_MBs: 1323
SuperMicro eth8 (on 5520 connected to NUMA node 0):
[root@...ntest1 ~]# ethtool -S eth8
NIC statistics:
...
read_dma_bw_MBs: 1572
write_dma_bw_MBs: 1605
read_write_dma_bw_MBs: 2113
> These values are obtained serially, as the driver resets
> the NIC (reset happens at load time, and ifconfig up),
> so they could easily sum to more than the memory bandwidth
> of the system. But it would be good to check for any anomalies.
>
> I can send you a pointer to a tool we use internally, which loads
> some custom firmware on the NIC, and can exercise the DMA engines
> on all the NICs in parallel. This would give an idea of the
> aggregate DMA bandwidth available on the system. Let me know
> if you're interested.
Yes, I'd be interested.
-Thanks
-Bill
Full ethtool output:
--------------------------------------------------------------------------------
Asus eth2:
[root@...est1 ~]# ethtool -S eth2
NIC statistics:
rx_packets: 4
tx_packets: 10
rx_bytes: 240
tx_bytes: 708
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_boundary: 4096
WC: 0
irq: 2282
MSI: 1
MSIX: 0
read_dma_bw_MBs: 1625
write_dma_bw_MBs: 1599
read_write_dma_bw_MBs: 3192
serial_number: 356055
watchdog_resets: 0
link_changes: 6
link_up: 1
dropped_link_overflow: 0
dropped_link_error_or_filtered: 631516
dropped_pause: 631516
dropped_bad_phy: 0
dropped_bad_crc32: 0
dropped_unicast_filtered: 0
dropped_multicast_filtered: 11
dropped_runt: 0
dropped_overrun: 0
dropped_no_small_buffer: 0
dropped_no_big_buffer: 0
----------- slice ---------: 0
tx_pkt_start: 421736
tx_pkt_done: 421736
tx_req: 2866189
tx_done: 2866189
rx_small_cnt: 257731
rx_big_cnt: 3830824
wake_queue: 5698
stop_queue: 5698
tx_linearized: 0
LRO aggregated: 1276950
LRO flushed: 264545
LRO avg aggr: 4
LRO no_desc: 0
SuperMicro eth2 (on 5520 connected to NUMA node 1):
[root@...ntest1 ~]# ethtool -S eth2
NIC statistics:
rx_packets: 0
tx_packets: 10
rx_bytes: 0
tx_bytes: 708
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_boundary: 4096
WC: 0
irq: 112
MSI: 1
MSIX: 0
read_dma_bw_MBs: 1624
write_dma_bw_MBs: 1605
read_write_dma_bw_MBs: 1323
serial_number: 363134
watchdog_resets: 0
dca_capable_firmware: 1
dca_device_present: 0
link_changes: 2
link_up: 1
dropped_link_overflow: 0
dropped_link_error_or_filtered: 200
dropped_pause: 200
dropped_bad_phy: 0
dropped_bad_crc32: 0
dropped_unicast_filtered: 0
dropped_multicast_filtered: 0
dropped_runt: 0
dropped_overrun: 0
dropped_no_small_buffer: 0
dropped_no_big_buffer: 0
----------- slice ---------: 0
tx_pkt_start: 440223
tx_pkt_done: 440223
tx_req: 3412102
tx_done: 3412102
rx_small_cnt: 213976
rx_big_cnt: 3071854
wake_queue: 1846
stop_queue: 1846
tx_linearized: 0
LRO aggregated: 1024029
LRO flushed: 269709
LRO avg aggr: 3
LRO no_desc: 0
SuperMicro eth8 (on 5520 connected to NUMA node 0):
[root@...ntest1 ~]# ethtool -S eth8
NIC statistics:
rx_packets: 11
tx_packets: 16
rx_bytes: 864
tx_bytes: 1228
rx_errors: 0
tx_errors: 0
rx_dropped: 0
tx_dropped: 0
multicast: 0
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_fifo_errors: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_boundary: 4096
WC: 0
irq: 118
MSI: 1
MSIX: 0
read_dma_bw_MBs: 1572
write_dma_bw_MBs: 1605
read_write_dma_bw_MBs: 2113
serial_number: 361233
watchdog_resets: 0
dca_capable_firmware: 1
dca_device_present: 0
link_changes: 4
link_up: 1
dropped_link_overflow: 0
dropped_link_error_or_filtered: 224
dropped_pause: 224
dropped_bad_phy: 0
dropped_bad_crc32: 0
dropped_unicast_filtered: 0
dropped_multicast_filtered: 0
dropped_runt: 0
dropped_overrun: 0
dropped_no_small_buffer: 0
dropped_no_big_buffer: 0
----------- slice ---------: 0
tx_pkt_start: 575354
tx_pkt_done: 575354
tx_req: 3590761
tx_done: 3590761
rx_small_cnt: 227078
rx_big_cnt: 4733499
wake_queue: 2199
stop_queue: 2199
tx_linearized: 0
LRO aggregated: 1578229
LRO flushed: 404901
LRO avg aggr: 3
LRO no_desc: 0
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists