linux-kernel - Re: CFS vs. cpufreq/cstates vs. latency

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Tue, 24 Jul 2012 12:08:06 -0600
From:	Chris Friesen <chris.friesen@...band.com>
To:	Avi Kivity <avi@...hat.com>
CC:	Rik van Riel <riel@...hat.com>,
	Linux kernel Mailing List <linux-kernel@...r.kernel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Ingo Molnar <mingo@...nel.org>, Gleb Natapov <gleb@...hat.com>,
	"Michael S. Tsirkin" <mst@...hat.com>,
	Andi Kleen <ak@...ux.intel.com>
Subject: Re: CFS vs. cpufreq/cstates vs. latency

> On 07/17/2012 05:23 PM, Rik van Riel wrote:
>> While tracking down a latency issue with communication between
>> KVM guests, we ran into a very interesting issue, an interplay
>> of CFS and power saving code.
>>
>> About 3/4 of the 230us latency came from CPUs waking up out of
>> C-states. Disabling C states reduced the latency to 60us...
>>
>> The issue? The communication is between various threads and
>> processes, each of which last ran on a CPU that is now in a
>> deeper C state. The total latency from that is "CPU wakeup
>> latency * NR CPUs woken".
>>
>> This problem could be common to many different multi-threaded
>> or multi-process applications. It looks like something that
>> would be fixable at the scheduler + cpufreq level.
>>
>> Specifically, waking up some process requires that the CPU
>> which is running the wakeup is already in C0 state. If the
>> CPU on which the to-be-woken task ran last is in a deep C
>> state, it may make sense to simply run the woken up task
>> on the local CPU, not the CPU where it was originally.
>>
>> I seem to remember some scheduling code that (for power
>> saving reasons) tried running all the tasks on one CPU,
>> until that CPU got busy, and then spilled over onto other
>> CPUs.
>>
>> I do not seem to be able to find that code in recent kernels,
>> but I have the feeling that a policy like that (related to
>> WAKE_AFFINE scheduling?) could improve this issue.
>>
>> As an additional benefit, it has the possibility of further
>> improving power saving.
>>
>> What do the scheduler and cpufreq people think about this
>> problem?
>>
>> Any preferred ways to solve the "N * cpu wakeup latency"
>> problem that is plaguing multi-process and multi-threaded
>> workloads?
> A few notes:
>
> - if you go into deep C-state, it may be worthwhile to migrate all the
> interrupts away from that cpu.  sysfs says C3 latency is 200 us on one
> of my machines, if we go there we should migrate anything important away.
>
> - I believe some of those C-states flush the cache, so executing on a
> cpu that is has awoken from one of these states will be slow for a
> while; needs to be taken into account.

On current Intel I think C3 flushes L1/L2 and when all cores on a socket 
are in C7 the last-level-cache is flushed.

Chris

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/