lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:	Wed, 13 Jan 2016 11:08:33 +0200
From:	Denys Fedoryshchenko <nuclearcat@...learcat.com>
To:	Netdev <netdev@...r.kernel.org>,
	Santosh Raspatur <santosh@...lsio.com>
Subject: cxgb3 Chelsio S310 tx drops and latency

Hi

I am trying to use Chelsio S310 on haproxy balancers and noticed 
following problems:

1)Latency on load 2.5+ Gbps is going in spikes 10ms+. Comparing with 
other vendors who is going to 7-8Gbps without problems.
2)I see a lot of drops in qdisc queues, while comparing with other 
vendors without drops on higher loads.

I suspect it might be or problem of card or drivers, because i am doing 
tests with similar cards from other vendor on same setup.
CPU resources are fine, i am monitoring them with with mpstat -P ALL, 
and cpufreq set to performance (always max freq on CPU).

Kernel 4.4 vanilla

HTTPS-VISP ~ # ethtool -i eth0
driver: cxgb3
version: 1.1.5-ko
firmware-version: T 7.12.0 TP 1.1.0
expansion-rom-version:
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no

Here is more details about my setup:

As far as i see i am having 2.5GT/s x8 PCI-Express established, i 
checked and tried perftune.sh, it made no difference.

05:00.0 Ethernet controller: Chelsio Communications Inc S310-CR 10GbE 
Single Port Adapter
         Subsystem: Chelsio Communications Inc Device 0001
         Physical Slot: 785
         Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- 
ParErr+ Stepping- SERR+ FastB2B- DisINTx+
         Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- 
<TAbort- <MAbort- >SERR- <PERR- INTx-
         Latency: 0, Cache Line Size: 32 bytes
         Interrupt: pin A routed to IRQ 0
         Region 0: Memory at 92800000 (64-bit, non-prefetchable) 
[size=4K]
         Region 2: Memory at 92000000 (64-bit, non-prefetchable) 
[size=8M]
         Region 4: Memory at 92801000 (64-bit, non-prefetchable) 
[size=4K]
         Expansion ROM at 92880000 [disabled] [size=512K]
         Capabilities: [40] Power Management version 3
                 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA 
PME(D0+,D1-,D2-,D3hot+,D3cold-)
                 Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
         Capabilities: [48] MSI: Enable- Count=1/32 Maskable- 64bit+
                 Address: 0000000000000000  Data: 0000
         Capabilities: [58] Express (v2) Endpoint, MSI 00
                 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s 
<64ns, L1 <1us
                         ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                 DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ 
Unsupported-
                         RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                         MaxPayload 256 bytes, MaxReadReq 512 bytes
                 DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- 
TransPend-
                 LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, 
Exit Latency L0s unlimited, L1 unlimited
                         ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                 LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                         ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                 LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ 
DLActive- BWMgmt- ABWMgmt-
                 DevCap2: Completion Timeout: Range ABC, TimeoutDis-, 
LTR-, OBFF Not Supported
                 DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, 
LTR-, OBFF Disabled
                 LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- 
SpeedDis-
                          Transmit Margin: Normal Operating Range, 
EnterModifiedCompliance- ComplianceSOS-
                          Compliance De-emphasis: -6dB
                 LnkSta2: Current De-emphasis Level: -6dB, 
EqualizationComplete-, EqualizationPhase1-
                          EqualizationPhase2-, EqualizationPhase3-, 
LinkEqualizationRequest-
         Capabilities: [94] Vital Product Data
                 Unknown small resource type 00, will not decode more.
         Capabilities: [9c] MSI-X: Enable+ Count=32 Masked-
                 Vector table: BAR=4 offset=00000000
                 PBA: BAR=4 offset=00000800
         Capabilities: [100 v1] Device Serial Number 
00-00-00-01-00-00-00-01
         Capabilities: [300 v1] Advanced Error Reporting
                 UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                 UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ 
UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                 UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ 
UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                 CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
NonFatalErr-
                 CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- 
NonFatalErr+
                 AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ 
ChkEn-
         Kernel driver in use: cxgb3

With uptime just 10 hours, over lowest load time:

BALANCER ~ # tc -s -d qdisc
qdisc mq 0: dev eth0 root
  Sent 3445096608552 bytes 2777441550 pkt (dropped 6509380, overlimits 0 
requeues 1156)
  backlog 0b 0p requeues 1156
qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 428721781866 bytes 880919921 pkt (dropped 1403007, overlimits 0 
requeues 243)
  backlog 0b 0p requeues 243
qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 429526790584 bytes 879237430 pkt (dropped 560836, overlimits 0 
requeues 102)
  backlog 0b 0p requeues 102
qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 427233606632 bytes 882026985 pkt (dropped 1255127, overlimits 0 
requeues 221)
  backlog 0b 0p requeues 221
qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 429138959430 bytes 882420516 pkt (dropped 914917, overlimits 0 
requeues 164)
  backlog 0b 0p requeues 164
qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 435498527047 bytes 888437887 pkt (dropped 325962, overlimits 0 
requeues 60)
  backlog 0b 0p requeues 60
qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 427458674980 bytes 881585256 pkt (dropped 610681, overlimits 0 
requeues 113)
  backlog 0b 0p requeues 113
qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 435164189156 bytes 891730352 pkt (dropped 891579, overlimits 0 
requeues 156)
  backlog 0b 0p requeues 156
qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap  1 2 2 2 1 2 0 0 
1 1 1 1 1 1 1 1
  Sent 432354080111 bytes 886050500 pkt (dropped 547271, overlimits 0 
requeues 97)


Powered by blists - more mailing lists