linux-kernel - User space starved by kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <1339092355.77924.YahooMailNeo@web43504.mail.sp1.yahoo.com>
Date:	Thu, 7 Jun 2012 11:05:55 -0700 (PDT)
From:	Sunny Das <internet_everyone@...oo.com>
To:	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: User space starved by kernel 

Hi Folks,

In a single vCPU VM with Linux 3.2.14, running just IP routing using Ixia, what we see is that user space processes starve for CPU and are not scheduled for a very long time. We have a network heartbeat daemon running inside the VM which sends out packets every 5 seconds to the master in the cluster. Once this daemon fails 3 successive heartbeats, it commits suicide. Running this daemon with SCHED_RR policy did not help.

Just looking at the 'top' from the console (whenever it manages to refresh), at the peak traffic, its clear that CPU is consumed by kernel spending a huge (98-99% si) amount of time servicing software interrupts.

Now, here is the kicker: we patched the kernel with (Con's) BFS scheduler and reran our setup. The network HB daemon is able to send out HBs regularly and cluster stays sane under peak traffic. Throughput shown by Ixia is not affected either. Now, we don't want to use BFS for our production installations. So, here is the question: Are there any settings in CFS or cgroups which we can tune to alter this behavior? Any other things we can try?

Will really appreciate your help.

Thanks a lot.
Sunny

PS: The reason for using a user space daemon for HB was that if our user space control plane is not able to schedule in a timely manner, the node's status in the cluster is hosed for all practical purposes.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/