[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <ZFe1IY+TdzSC0RLZ@righiandr-XPS-13-7390>
Date: Sun, 7 May 2023 16:26:41 +0200
From: Andrea Righi <andrea.righi@...onical.com>
To: Steven Rostedt <rostedt@...dmis.org>
Cc: Frederic Weisbecker <frederic@...nel.org>,
Ingo Molnar <mingo@...nel.org>,
Thomas Gleixner <tglx@...utronix.de>,
Greg Kroah-Hartman <gregkh@...uxfoundation.org>,
"Rafael J. Wysocki" <rafael@...nel.org>,
Jonathan Corbet <corbet@....net>,
Salvatore Bonaccorso <carnil@...ian.org>,
Pawan Gupta <pawan.kumar.gupta@...ux.intel.com>,
Mauro Carvalho Chehab <mchehab@...nel.org>,
Daniel Lezcano <daniel.lezcano@...aro.org>,
linux-kernel@...r.kernel.org, linux-doc@...r.kernel.org,
Anna-Maria Behnsen <anna-maria@...utronix.de>
Subject: Re: [PATCH] timers/nohz: introduce nohz_full_aggressive
On Sun, May 07, 2023 at 10:08:52AM -0400, Steven Rostedt wrote:
>
> [ Added Anna-Maria who is doing some timer work as well ]
>
> On Sun, 7 May 2023 11:07:00 +0200
> Andrea Righi <andrea.righi@...onical.com> wrote:
>
> > Overview:
> >
> > nohz_full is a feature that allows to reduce the number of CPU tick
> > interrupts, thereby improving energy efficiency and reducing kernel
> > jitter.
>
> Hmm, I never thought of NOHZ_FULL used for energy efficiency, as the
> CPU is still running user space code, and there's really nothing
> inherently more power consuming with the tick.
The idea here was to try to reduce the tick also on the timekeeping CPU
to have more idle time (because at least 1 CPU is periodically ticking
with nohz_full=all).
But my patch was mostly a toy patch and the real purpose was really to
get some advices/guidance on the tick/nohz topic.
>
> >
> > This works by stopping the tick interrupts on the CPUs that are either
> > idle or that have only one runnable task on them (there is no reason to
> > periodically interrupt the execution of a single running task if none
> > else is waiting to acquire the same CPU).
> >
> > It is not possible to configure all the available CPUs to work in the
> > nohz_full mode, at least one non-adaptive-tick CPU must be periodically
> > interrupted to properly handle timekeeping tasks in the system (such as
> > the gettimeofday() syscall returning accurate values).
>
> Do we really need nohz_full, instead, I think you want to look at what
> Anna-Maria is doing with moving the timer "manager" around to make sure
> that the tick stays on busy CPUs.
>
> Again, nohz_full is not for power consumption savings, but instead to
> reduce kernel interruption in user space.
Will definitely look at Anna-Maria's work.
>
> >
> > However, under certain conditions, we may want to relax this constraint,
> > accepting potential time inaccuracies in the system, in order to provide
> > additional benefits in terms of power consumption, performance and/or
> > reduce kernel jitter even more.
> >
> > For this reason introduce the new parameter nohz_full_aggressive.
> >
> > This option allows to enforce nozh_full across all the CPUs (even the
> > timekeeping CPU) at the cost of having potential timer inaccuracies in
> > the system.
> >
> > Test:
> >
> > - Hardware: Dell XPS 13 7390 w/ 8 cores
> >
> > - Kernel is using CONFIG_HZ=1000 (worst case scenario in terms of
> > power consumption and kernel jitter) and nohz_full=all
> >
> > - Measure interrupts and power consumption when the system is idle and
> > with 2, 4 and 8 cpu hogs
> >
> > Result:
> >
> > The following numbers have been collected using turbostat and dstat
> > measuring the average over a 5min run for each test.
> >
> > irqs/sec idle 1 CPU hog 2 CPU hogs 4 CPU hogs 8 CPU hogs
> > ------------------------------------------------------
> > nohz_full 1036.679 1047.522 1046.203 1048.590 1074.867
> > nohz_full_aggressive 98.685 106.296 127.587 146.586 1062.277
> >
> > Power (Watt) idle 1 CPU hog 2 CPU hogs 4 CPU hogs 8 CPU hogs
> > ------------------------------------------------------
> > nohz_full 0.502 W 3.436 W 3.755 W 6.187 W 6.019 W
> > nohz_full_aggressive 0.301 W 2.372 W 2.372 W 6.005 W 6.016 W
> >
> > % power reduction 40.04% 30.97% 36.83% 2.94% 0.05%
> >
>
> Nice.
>
> Now I doubt this is acceptable considering the side effects that the
> timer inaccuracy can cause. I think this breaks some basic assumptions
> in both the kernel and user space.
I've been running this nohz_full_aggressive patch for some days on my
laptop without any evident side effect, but I'm pretty sure it can break
something, considering that timing potentially can become totally
unreliable.
I was also wondering if we could try to implement a kind of dynamic HZ
scaling (like scaling HZ up/down dynamically at runtime or even at boot
time), but it seems quite complicated (and scary, especially looking at
the code in jiffies / timers, i.e. all the constants in
./kernel/time/timeconst.bc).
I remember there used to be a dynamic-hz patch a long long time ago by
Andrea Arcangeli, but I couldn't find any recent work on this topic.
>
> Now, I think what is really happening here is that you are somewhat
> simulating the results that Anna-Maria has indirectly. That is, you
> just prevent an idle CPU from waking up to handle interrupts when not
> needed.
>
> Anna-Maria,
>
> Do you have some patches that Andrea could test with?
>
> Thanks,
>
> -- Steve
Thanks for looking at this (and I'm happy to help Anna-Maria with any
test).
-Andrea
Powered by blists - more mailing lists