| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <1517165975.3715.85.camel@gmail.com>
Date: Sun, 28 Jan 2018 10:59:35 -0800
From: Eric Dumazet <eric.dumazet@...il.com>
To: Paweł Staszewski <pstaszewski@...are.pl>,
"netde >> Linux Kernel Network Developers" <netdev@...r.kernel.org>
Subject: Re: kernel 4.15.0-rc9+ (net-next) high cpu load at 50Gbit/s - about
6Mpps
On Sun, 2018-01-28 at 19:26 +0100, Paweł Staszewski wrote:
>
> W dniu 27.01.2018 o 23:23, Paweł Staszewski pisze:
> > Hi
> >
> >
> > Today I made some real life traffic tests with kernel 4.15.0-rc9
> >
> > but when traffic reach 50Gbit/s and about 6Mpps cpou load rises fast
> > from 48% to 100% for all cpu cores.
> >
> > Here is some graph that presenting how cpu load rises when there was
> > more pps.
> >
> >
> > https://ibb.co/mhD5ob
> >
> >
> > here is perf record from that time:
> >
> > https://pastebin.com/3zqG1rvE
> >
> >
> > There is 8x 10G ixgbe 82599 interfaces teamed with teamd.
> >
> > No traffic queueing - only pfifo fast on all interfaces.
> >
> > No NAT or iptables forles other than INPUT (about 30rules)
> >
> > All nic's have same ethtool settings:
> >
> > ethtool -k eth0
> > Features for eth0:
> > Cannot get device udp-fragmentation-offload settings: Operation not
> > supported
> > rx-checksumming: on
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: on
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: off [fixed]
> > tx-tcp-mangleid-segmentation: off
> > tx-tcp6-segmentation: on
> > udp-fragmentation-offload: off
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off
> > rx-vlan-offload: on
> > tx-vlan-offload: on
> > ntuple-filters: on
> > receive-hashing: on
> > highdma: on [fixed]
> > rx-vlan-filter: on
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: off [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: on
> > tx-gre-csum-segmentation: on
> > tx-ipxip4-segmentation: on
> > tx-ipxip6-segmentation: on
> > tx-udp_tnl-segmentation: on
> > tx-udp_tnl-csum-segmentation: on
> > tx-gso-partial: on
> > tx-sctp-segmentation: off [fixed]
> > tx-esp-segmentation: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off
> > hw-tc-offload: off
> > esp-hw-offload: off [fixed]
> > esp-tx-csum-hw-offload: off [fixed]
> > rx-udp_tunnel-port-offload: on
> >
> >
> > ethtool -g eth0
> > Ring parameters for eth0:
> > Pre-set maximums:
> > RX: 4096
> > RX Mini: 0
> > RX Jumbo: 0
> > TX: 4096
> > Current hardware settings:
> > RX: 4096
> > RX Mini: 0
> > RX Jumbo: 0
> > TX: 2048
> >
> >
> > ethtool -c eth0
> > Coalesce parameters for eth0:
> > Adaptive RX: off TX: off
> > stats-block-usecs: 0
> > sample-interval: 0
> > pkt-rate-low: 0
> > pkt-rate-high: 0
> >
> > rx-usecs: 512
> > rx-frames: 0
> > rx-usecs-irq: 0
> > rx-frames-irq: 0
> >
> > tx-usecs: 0
> > tx-frames: 0
> > tx-usecs-irq: 0
> > tx-frames-irq: 0
> >
> > rx-usecs-low: 0
> > rx-frame-low: 0
> > tx-usecs-low: 0
> > tx-frame-low: 0
> >
> > rx-usecs-high: 0
> > rx-frame-high: 0
> > tx-usecs-high: 0
> > tx-frame-high: 0
> >
> >
> >
> >
>
>
>
> Peft top for kernel 4.15.0-rc9 below (all 40 cores 100% cpu load with
> 6.3Mpps)
>
> 20.96% [kernel] [k] queued_spin_lock_slowpath
> 5.51% [kernel] [k] ixgbe_poll
> 5.49% [kernel] [k] ixgbe_xmit_frame_ring
> 4.39% [kernel] [k] do_raw_spin_lock
> 4.29% [kernel] [k] sch_direct_xmit
> 4.11% [kernel] [k] fib_table_lookup
> 3.11% [team_mode_roundrobin] [k] rr_transmit
> 2.71% [kernel] [k] __dev_queue_xmit
> 2.62% [kernel] [k] __ptr_ring_peek
> 2.39% [kernel] [k] skb_release_data
> 2.18% [kernel] [k] dev_gro_receive
> 1.75% [kernel] [k] __qdisc_run
> 1.67% [kernel] [k] pfifo_fast_enqueue
> 1.57% [kernel] [k] netdev_pick_tx
> 1.56% [kernel] [k] page_frag_free
> 1.48% [kernel] [k] ip_finish_output2
> 1.38% [kernel] [k] __slab_free
> 1.36% [kernel] [k] skb_unref
> 1.34% [kernel] [k] ixgbe_maybe_stop_tx
> 1.30% [kernel] [k] vlan_do_receive
> 1.28% [kernel] [k] pfifo_fast_dequeue
> 1.23% [kernel] [k] virt_to_head_page
>
>
>
> Same configuration kernel 4.15.0-rc3 (50% cpu load on all 40 cores with
> 6.3Mpps)
>
> 7.81% [kernel] [k] ixgbe_xmit_frame_ring
> 7.61% [kernel] [k] ixgbe_poll
> 7.09% [kernel] [k] do_raw_spin_lock
> 5.63% [kernel] [k] fib_table_lookup
> 5.19% [kernel] [k] __dev_queue_xmit
> 4.38% [team_mode_roundrobin] [k] rr_transmit
> 3.10% [kernel] [k] netdev_pick_tx
> 2.79% [kernel] [k] skb_release_data
> 2.34% [kernel] [k] dev_gro_receive
> 1.99% [kernel] [k] page_frag_free
> 1.96% [kernel] [k] skb_unref
> 1.92% [kernel] [k] virt_to_head_page
> 1.90% [kernel] [k] ixgbe_maybe_stop_tx
> 1.82% [kernel] [k] vlan_do_receive
> 1.74% [kernel] [k] ip_finish_output2
> 1.73% [kernel] [k] __build_skb
> 1.68% [kernel] [k] __slab_free
> 1.67% [kernel] [k] __netif_receive_skb_core
> 1.60% [kernel] [k] inet_gro_receive
> 1.49% [kernel] [k] netif_skb_features
> 1.35% [kernel] [k] ip_rcv
> 1.33% [kernel] [k] ipt_do_table
> 1.30% [kernel] [k] compound_head
> 1.26% [kernel] [k] dev_hard_start_xmit
> 1.18% [kernel] [k] put_page
> 1.13% [kernel] [k] tcp_gro_receive
> 1.13% [kernel] [k] ip_forward
> 0.99% [kernel] [k] validate_xmit_skb
> 0.97% [kernel] [k] ip_route_input_rcu
> 0.88% [kernel] [k] inet_lookup_ifaddr_rcu
> 0.81% [kernel] [k] pfifo_fast_dequeue
> 0.77% [kernel] [k] vlan_dev_hard_start_xmit
> 0.64% [kernel] [k] ___slab_alloc
Please report :
1) "ethtool -l" information for your ethernet adapters.
2) IRQ configuration (grep eth /proc/interrupts)
3) tc -s qdisc show
Powered by blists - more mailing lists