lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
Open Source and information security mailing list archives
| ||
|
Date: Wed, 13 Jan 2016 11:08:33 +0200 From: Denys Fedoryshchenko <nuclearcat@...learcat.com> To: Netdev <netdev@...r.kernel.org>, Santosh Raspatur <santosh@...lsio.com> Subject: cxgb3 Chelsio S310 tx drops and latency Hi I am trying to use Chelsio S310 on haproxy balancers and noticed following problems: 1)Latency on load 2.5+ Gbps is going in spikes 10ms+. Comparing with other vendors who is going to 7-8Gbps without problems. 2)I see a lot of drops in qdisc queues, while comparing with other vendors without drops on higher loads. I suspect it might be or problem of card or drivers, because i am doing tests with similar cards from other vendor on same setup. CPU resources are fine, i am monitoring them with with mpstat -P ALL, and cpufreq set to performance (always max freq on CPU). Kernel 4.4 vanilla HTTPS-VISP ~ # ethtool -i eth0 driver: cxgb3 version: 1.1.5-ko firmware-version: T 7.12.0 TP 1.1.0 expansion-rom-version: bus-info: 0000:05:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: yes supports-register-dump: yes supports-priv-flags: no Here is more details about my setup: As far as i see i am having 2.5GT/s x8 PCI-Express established, i checked and tried perftune.sh, it made no difference. 05:00.0 Ethernet controller: Chelsio Communications Inc S310-CR 10GbE Single Port Adapter Subsystem: Chelsio Communications Inc Device 0001 Physical Slot: 785 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 32 bytes Interrupt: pin A routed to IRQ 0 Region 0: Memory at 92800000 (64-bit, non-prefetchable) [size=4K] Region 2: Memory at 92000000 (64-bit, non-prefetchable) [size=8M] Region 4: Memory at 92801000 (64-bit, non-prefetchable) [size=4K] Expansion ROM at 92880000 [disabled] [size=512K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [48] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [58] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 4096 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 2.5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABC, TimeoutDis-, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [94] Vital Product Data Unknown small resource type 00, will not decode more. Capabilities: [9c] MSI-X: Enable+ Count=32 Masked- Vector table: BAR=4 offset=00000000 PBA: BAR=4 offset=00000800 Capabilities: [100 v1] Device Serial Number 00-00-00-01-00-00-00-01 Capabilities: [300 v1] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP+ FCP+ CmpltTO+ CmpltAbrt+ UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ AERCap: First Error Pointer: 14, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: cxgb3 With uptime just 10 hours, over lowest load time: BALANCER ~ # tc -s -d qdisc qdisc mq 0: dev eth0 root Sent 3445096608552 bytes 2777441550 pkt (dropped 6509380, overlimits 0 requeues 1156) backlog 0b 0p requeues 1156 qdisc pfifo_fast 0: dev eth0 parent :1 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 428721781866 bytes 880919921 pkt (dropped 1403007, overlimits 0 requeues 243) backlog 0b 0p requeues 243 qdisc pfifo_fast 0: dev eth0 parent :2 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 429526790584 bytes 879237430 pkt (dropped 560836, overlimits 0 requeues 102) backlog 0b 0p requeues 102 qdisc pfifo_fast 0: dev eth0 parent :3 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 427233606632 bytes 882026985 pkt (dropped 1255127, overlimits 0 requeues 221) backlog 0b 0p requeues 221 qdisc pfifo_fast 0: dev eth0 parent :4 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 429138959430 bytes 882420516 pkt (dropped 914917, overlimits 0 requeues 164) backlog 0b 0p requeues 164 qdisc pfifo_fast 0: dev eth0 parent :5 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 435498527047 bytes 888437887 pkt (dropped 325962, overlimits 0 requeues 60) backlog 0b 0p requeues 60 qdisc pfifo_fast 0: dev eth0 parent :6 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 427458674980 bytes 881585256 pkt (dropped 610681, overlimits 0 requeues 113) backlog 0b 0p requeues 113 qdisc pfifo_fast 0: dev eth0 parent :7 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 435164189156 bytes 891730352 pkt (dropped 891579, overlimits 0 requeues 156) backlog 0b 0p requeues 156 qdisc pfifo_fast 0: dev eth0 parent :8 bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1 Sent 432354080111 bytes 886050500 pkt (dropped 547271, overlimits 0 requeues 97)
Powered by blists - more mailing lists