netdev - Fw: [Bug 88161] New: High traffic causes a lot of softirqs

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20141113162124.5170b82d@uryu.home.lan>
Date:	Thu, 13 Nov 2014 16:21:24 -0500
From:	Stephen Hemminger <stephen@...workplumber.org>
To:	netdev@...r.kernel.org
Subject: Fw: [Bug 88161] New: High traffic causes a lot of softirqs



Begin forwarded message:

Date: Thu, 13 Nov 2014 06:18:28 -0800
From: "bugzilla-daemon@...zilla.kernel.org" <bugzilla-daemon@...zilla.kernel.org>
To: "stephen@...workplumber.org" <stephen@...workplumber.org>
Subject: [Bug 88161] New: High traffic causes a lot of softirqs


https://bugzilla.kernel.org/show_bug.cgi?id=88161

            Bug ID: 88161
           Summary: High traffic causes a lot of softirqs
           Product: Networking
           Version: 2.5
    Kernel Version: 3.17.2
          Hardware: Intel
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: high
          Priority: P1
         Component: Other
          Assignee: shemminger@...ux-foundation.org
          Reporter: mike@...ntric.com
        Regression: No

I'm using packaged rpms by centos and elrepo with the same results and I can
replicate this on any server in our cluster. 

I have tried installing 

kernel-3.10.56-11.el6.centos.alt.x86_64

Also currently running

[root@...125-east.domain.com /var/www/html]# uname -a
Linux web125-east.domain.com 3.17.2-1.el6.elrepo.x86_64 #1 SMP Fri Oct 31
10:37:44 EDT 2014 x86_64 x86_64 x86_64 GNU/Linux

from the centosplus repo to solve a problem where 2.6 was locking up process
tree on high cpu and it fixed it but it introduced another issue where we have
a lot of softirq requests when under a lot of traffic load. 

Here is a powertop from a 2.6 series server

Summary: 42492.1 wakeups/second, 0.0 GPU ops/seconds, 0.0 VFS ops/sec and
2422.0% CPU use

                Usage Events/s Category Description
            22613 ms/s 23637.4    Process php-fpm: pool www
            716.9 ms/s 15783.2    Process nginx: worker process
             21.3 ms/s 1096.1    Process /usr/bin/java -Xms200m -Xmx2000m
-Xss256k -XX:MaxDirectMemorySize=516m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-Dage
              5.8 ms/s 674.4 Process /usr/sbin/gmond
            130.0 ms/s 494.5 Process /usr/bin/redis-server 127.0.0.1:6379
             73.2 ms/s 487.4 Process python /usr/bin/statsd-relay.py
              3.8 ms/s 82.7 Process java -Xmx6g -server -Dfile.encoding=utf-8
-XX:OnOutOfMemoryError=kill -9 %p -XX:+HeapDumpOnOutOfMemoryError -XX:HeapD
            212.4 ms/s 0.00 Interrupt [3] net_rx(softirq)


Here it is from 3.10


                Usage Events/s Category Description
             10.2 ms/s 1033.6    Timer hrtimer_wakeup
              3.3 ms/s 932.7 Process /usr/bin/java -Xms200m -Xmx2000m -Xss256k
            591.1 ms/s 624.3 Process php-fpm: pool www
             41.5 ms/s 724.0 Interrupt [3] net_rx(softirq)

Load pretty much just keeps crawling up to the 500's

There also is a lot of CPU usage from

  116 root 20 0 0 0 0 R 75.0 0.0 0:04.57 kworker/u66:0

Which from my understanding handles a lot of the acpi calls that softirq is
doing. 

I've tried many other 3.x kernels above 3.10 with the same results.. so I'm
wondering if this is a known issue

-- 
You are receiving this mail because:
You are the assignee for the bug.
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html