linux-kernel - Re: [PATCH v4 1/5] nohz_full: add support for "cpu

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 13 Jul 2015 14:45:19 -0700
From:	Andy Lutomirski <luto@...capital.net>
To:	Chris Metcalf <cmetcalf@...hip.com>
Cc:	Gilad Ben Yossef <giladb@...hip.com>,
	Steven Rostedt <rostedt@...dmis.org>,
	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <peterz@...radead.org>,
	Andrew Morton <akpm@...ux-foundation.org>,
	Rik van Riel <riel@...hat.com>, Tejun Heo <tj@...nel.org>,
	Frederic Weisbecker <fweisbec@...il.com>,
	Thomas Gleixner <tglx@...utronix.de>,
	"Paul E. McKenney" <paulmck@...ux.vnet.ibm.com>,
	Christoph Lameter <cl@...ux.com>,
	Viresh Kumar <viresh.kumar@...aro.org>,
	"linux-doc@...r.kernel.org" <linux-doc@...r.kernel.org>,
	Linux API <linux-api@...r.kernel.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH v4 1/5] nohz_full: add support for "cpu_isolated" mode

On Mon, Jul 13, 2015 at 2:01 PM, Chris Metcalf <cmetcalf@...hip.com> wrote:
> On 07/13/2015 04:40 PM, Andy Lutomirski wrote:
>>
>> On Mon, Jul 13, 2015 at 12:57 PM, Chris Metcalf <cmetcalf@...hip.com>
>> wrote:
>>>
>>> The existing nohz_full mode makes tradeoffs to minimize userspace
>>> interruptions while still attempting to avoid overheads in the
>>> kernel entry/exit path, to provide 100% kernel semantics, etc.
>>>
>>> However, some applications require a stronger commitment from the
>>> kernel to avoid interruptions, in particular userspace device
>>> driver style applications, such as high-speed networking code.
>>>
>>> This change introduces a framework to allow applications to elect
>>> to have the stronger semantics as needed, specifying
>>> prctl(PR_SET_CPU_ISOLATED, PR_CPU_ISOLATED_ENABLE) to do so.
>>> Subsequent commits will add additional flags and additional
>>> semantics.
>>
>> I thought the general consensus was that this should be the default
>> behavior and that any associated bugs should be fixed.
>
>
> I think it comes down to dividing the set of use cases in two:
>
> - "Regular" nohz_full, as used to improve performance and limit
>   interruptions, possibly for power benefits, etc.  But, stray
>   interrupts are not particularly bad, and you don't want to take
>   extreme measures to avoid them.
>
> - What I'm calling "cpu_isolated" mode where when you return to
>   userspace, you expect that by God, the kernel doesn't interrupt you
>   again, and if it does, it's a flat-out bug.
>
> There are a few things that cpu_isolated mode currently does to
> accomplish its goals that are pretty heavy-weight:
>
> Processes are held in kernel space until ticks are quiesced; this is
> not necessarily what every nohz_full task wants.  If a task makes a
> kernel call, there may well be arbitrary timer fallout, and having a
> way to select whether or not you are willing to take a timer tick after
> return to userspace is pretty important.

Then shouldn't deferred work be done immediately in nohz_full mode
regardless?  What is this delayed work that's being done?

>
> Likewise, there are things that you may want to do on return to
> userspace that are designed to prevent further interruptions in
> cpu_isolated mode, even at a possible future performance cost if and
> when you return to the kernel, such as flushing the per-cpu free page
> list so that you won't be interrupted by an IPI to flush it later.
>

Why not just kick the per-cpu free page over to whatever cpu is
monitoring your RCU state, etc?  That should be very quick.

> If you're arguing that the cpu_isolated semantic is really the only
> one that makes sense for nohz_full, my sense is that it might be
> surprising to many of the folks who do nohz_full work.  But, I'm happy
> to be wrong on this point, and maybe all the nohz_full community is
> interested in making the same tradeoffs for nohz_full generally that
> I've proposed in this patch series just for cpu_isolated?

nohz_full is currently dog slow for no particularly good reasons.  I
suspect that the interrupts you're seeing are also there for no
particularly good reasons as well.

Let's fix them instead of adding new ABIs to work around them.

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/