netdev - Re: Kernel Performance Tuning for High Volume SCTP traffic

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20171016133456.GA19009@hmswarspite.think-freely.org>
Date:   Mon, 16 Oct 2017 09:34:56 -0400
From:   Neil Horman <nhorman@...driver.com>
To:     Traiano Welcome <traiano@...il.com>
Cc:     David Laight <David.Laight@...lab.com>,
        "linux-sctp@...r.kernel.org" <linux-sctp@...r.kernel.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: Kernel Performance Tuning for High Volume SCTP traffic

On Sat, Oct 14, 2017 at 10:29:53PM +0800, Traiano Welcome wrote:
> I've upped the value of the following sctp and udp related parameters,
> in the hope that this would help:
> 
> sysctl -w net.core.rmem_max=900000000
> sysctl -w net.core.wmem_max=900000000
> 
> sysctl -w net.sctp.sctp_mem="2100000000 2100000000 2100000000"
> sysctl -w net.sctp.sctp_rmem="2100000000 2100000000 2100000000"
> sysctl -w net.sctp.sctp_wmem="2100000000 2100000000 2100000000"
> 
> sysctl -w net.ipv4.udp_mem="5000000000 5000000000 5000000000"
> sysctl -w net.ipv4.udp_mem="10000000000 10000000000 10000000000"
> 
> However, I'm still seeing rapidly incrementing rx discards reported on the NIC:
> 
> :~# ethtool -S ens4f1 | egrep -i rx_discards
>      [0]: rx_discards: 6390805462
>      [1]: rx_discards: 6659315919
>      [2]: rx_discards: 6542570026
>      [3]: rx_discards: 6431513008
>      [4]: rx_discards: 6436779078
>      [5]: rx_discards: 6665897051
>      [6]: rx_discards: 6167985560
>      [7]: rx_discards: 11340068788
>      rx_discards: 56634934892
> 
If you're getting drops in the hardware and nothing in the higher layers is
overflowing, then your problem is likely due to one of two things:

1) The NIC is discarding frames for reasons orthogonal to provisioning.  That is
to say you are getting a large number of frames in that are being purposely
discarded. Check the other stats for the nic in ethtool to try give you some
additional visibilty on what these frames might be.

2) You're not servicing the NIC fast enough to pull frames out prior to its
internal buffer overflowing.  Check the interrupt mitigation and flow
director/ntuple settings to make sure that you're seeing proper spreading of
packets to per-cpu queues, that interrupt mitigation is preventing undue cpu
load, and that irqbalance is properly distributing that interrupt rate to all
cpus in the system

Neil

> Despite the fact that I've set the NIC ring buffer on the Netextreme
> interface to he maximum:
> 
> :~# ethtool -g ens4f0
> Ring parameters for ens4f0:
> Pre-set maximums:
> RX:             4078
> RX Mini:        0
> RX Jumbo:       0
> TX:             4078
> Current hardware settings:
> RX:             4078
> RX Mini:        0
> RX Jumbo:       0
> TX:             4078
> 
> I see no ip errors at the physical interface:
> 
> ethtool -S ens4f0 | egrep phy_ip_err_discard| tail -1
>      rx_phy_ip_err_discards: 0
> 
> 
> Could anyone suggest alternative approaches I might take to optimising
> the system's handling of SCTP traffic?
> 
> 
> 
> On Sat, Oct 14, 2017 at 12:35 AM, David Laight <David.Laight@...lab.com> wrote:
> > From: Traiano Welcome
> >> Sent: 13 October 2017 17:04
> >> On Fri, Oct 13, 2017 at 11:56 PM, David Laight <David.Laight@...lab.com> wrote:
> >> > From: Traiano Welcome
> >> >
> >> > (copied to netdev)
> >> >> Sent: 13 October 2017 07:16
> >> >> To: linux-sctp@...r.kernel.org
> >> >> Subject: Kernel Performance Tuning for High Volume SCTP traffic
> >> >>
> >> >> Hi List
> >> >>
> >> >> I'm running a linux server processing high volumes of SCTP traffic and
> >> >> am seeing large numbers of packet overruns (ifconfig output).
> >> >
> >> > I'd guess that overruns indicate that the ethernet MAC is failing to
> >> > copy the receive frames into kernel memory.
> >> > It is probably running out of receive buffers, but might be
> >> > suffering from a lack of bus bandwidth.
> >> > MAC drivers usually discard receive frames if they can't get
> >> > a replacement buffer - so you shouldn't run out of rx buffers.
> >> >
> >> > This means the errors are probably below SCTP - so changing SCTP parameters
> >> > is unlikely to help.
> >>
> >> Does this mean that tuning UDP performance could help ? Or do you mean
> >> hardware (NIC) performance could be the issue?
> >
> > I'd certainly check UDP performance.
> >
> >         David
> >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-sctp" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>