linux-kernel - CFS vs. cpufreq/cstates vs. latency

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Date:	Tue, 17 Jul 2012 10:23:33 -0400
From:	Rik van Riel <riel@...hat.com>
To:	Linux kernel Mailing List <linux-kernel@...r.kernel.org>
CC:	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>, Avi Kivity <avi@...hat.com>,
	Gleb Natapov <gleb@...hat.com>,
	"Michael S. Tsirkin" <mst@...hat.com>,
	Andi Kleen <ak@...ux.intel.com>
Subject: CFS vs. cpufreq/cstates vs. latency

While tracking down a latency issue with communication between
KVM guests, we ran into a very interesting issue, an interplay
of CFS and power saving code.

About 3/4 of the 230us latency came from CPUs waking up out of
C-states. Disabling C states reduced the latency to 60us...

The issue? The communication is between various threads and
processes, each of which last ran on a CPU that is now in a
deeper C state. The total latency from that is "CPU wakeup
latency * NR CPUs woken".

This problem could be common to many different multi-threaded
or multi-process applications. It looks like something that
would be fixable at the scheduler + cpufreq level.

Specifically, waking up some process requires that the CPU
which is running the wakeup is already in C0 state. If the
CPU on which the to-be-woken task ran last is in a deep C
state, it may make sense to simply run the woken up task
on the local CPU, not the CPU where it was originally.

I seem to remember some scheduling code that (for power
saving reasons) tried running all the tasks on one CPU,
until that CPU got busy, and then spilled over onto other
CPUs.

I do not seem to be able to find that code in recent kernels,
but I have the feeling that a policy like that (related to
WAKE_AFFINE scheduling?) could improve this issue.

As an additional benefit, it has the possibility of further
improving power saving.

What do the scheduler and cpufreq people think about this
problem?

Any preferred ways to solve the "N * cpu wakeup latency"
problem that is plaguing multi-process and multi-threaded
workloads?

-- 
All rights reversed
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/