Message-ID: <87fsqqu6by.fsf@riseup.net>
Date:   Fri, 17 Dec 2021 16:04:01 -0800
From:   Francisco Jerez <currojerez@...eup.net>
To:     Julia Lawall <julia.lawall@...ia.fr>
Cc:     Julia Lawall <julia.lawall@...ia.fr>,
        "Rafael J. Wysocki" <rafael@...nel.org>,
        Srinivas Pandruvada <srinivas.pandruvada@...ux.intel.com>,
        Len Brown <lenb@...nel.org>,
        Viresh Kumar <viresh.kumar@...aro.org>,
        Linux PM <linux-pm@...r.kernel.org>,
        Linux Kernel Mailing List <linux-kernel@...r.kernel.org>,
        Ingo Molnar <mingo@...hat.com>,
        Peter Zijlstra <peterz@...radead.org>,
        Juri Lelli <juri.lelli@...hat.com>,
        Vincent Guittot <vincent.guittot@...aro.org>
Subject: Re: cpufreq: intel_pstate: map utilization into the pstate range

Julia Lawall <julia.lawall@...ia.fr> writes:

> On Fri, 17 Dec 2021, Francisco Jerez wrote:
>
>> Julia Lawall <julia.lawall@...ia.fr> writes:
>>
>> > On Fri, 17 Dec 2021, Rafael J. Wysocki wrote:
>> >
>> >> On Mon, Dec 13, 2021 at 11:52 PM Julia Lawall <julia.lawall@...ia.fr> wrote:
>> >> >
>> >> > With HWP, intel_cpufreq_adjust_perf takes the utilization, scales it
>> >> > between 0 and the capacity, and then maps everything below min_pstate to
>> >> > the lowest frequency.
>> >>
>> >> Well, it is not just intel_pstate with HWP.  This is how schedutil
>> >> works in general; see get_next_freq() in there.
>> >>
>> >> > On my Intel Xeon Gold 6130 and Intel Xeon Gold
>> >> > 5218, this means that more than the bottom quarter of utilizations are all
>> >> > mapped to the lowest frequency.  Running slowly doesn't necessarily save
>> >> > energy, because it takes more time.
>> >>
>> >> This is true, but the layout of the available range of performance
>> >> values is a property of the processor, not a driver issue.
>> >>
>> >> Moreover, the role of the driver is not to decide how to respond to
>> >> the given utilization value, that is the role of the governor.  The
>> >> driver is expected to do what it is asked for by the governor.
>> >
>> > OK, but what exactly is the goal of schedutil?
>> >
>> > I would have expected that it was to give good performance while saving
>> > energy, but it's not doing either in many of these cases.
>> >
>> > Is it the intent of schedutil that the bottom quarter of utilizations
>> > should be mapped to the lowest frequency?
>> >
>>
>> If the lowest frequency provides more performance than needed to handle
>> the CPU utilization observed by schedutil, why would it want any other
>> frequency than the (theoretically most efficient) minimum P-state?
>>
>> Remember that whether running more slowly saves energy or not depends
>> among other things on whether your system is running beyond the
>> inflection point of its power curve (AKA frequency of maximum
>> efficiency).  Within the region of concavity below this most efficient
>> frequency, yes, running more slowly will waste energy, however, the
>> optimal behavior within that region is to fix your clock to the most
>> efficient frequency and then power-gate the CPU once it's run out of
>> work to do -- Which is precisely what the current code can be expected
>> to achieve by clamping its response to min_pstate, which is meant to
>> approximate the most efficient P-state of the CPU -- Though looking at
>> your results makes me think that that's not happening for you, possibly
>> because intel_pstate's notion of the most efficient frequency may be
>> fairly inaccurate in this case.
>
> I'm not sure to understand the concept of the min_pstate being the most
> efficient one.  The min_pstate appears to be just the minimum frequency
> advertised for the machine.  Is that somehow intended to be the most
> efficient one?
>

Yeah, that's what it should be ideally, since there is hardly any reason
to ever program the CPU clock to run below this most efficient
frequency, since the concavity region of the CPU power curve is
inherently inefficient and delivers lower performance than the most
efficient frequency.

As you can see in intel_pstate.c, min_pstate is initialized on core
platforms from MSR_PLATFORM_INFO[47:40], which is "Maximum Efficiency
Ratio (R/O)".  However that seems to deviate massively from the most
efficient ratio on your system, which may indicate a firmware bug, some
sort of clock gating problem, or an issue with the way that
intel_pstate.c processes this information.

> On the other hand, I noticed that by putting lower numbers than the min
> one, one seems to obtain lower frequencies than what is advertised for the
> machine.
>
>> Your energy usage results below seem to provide some evidence that we're
>> botching min_pstate in your system: Your energy figures scale pretty
>> much linearly with the runtime of each testcase, which suggests that
>> your energy usage is mostly dominated by leakage current, as would be
>> the case for workloads running far below the most efficient frequency of
>> the CPU.
>
> I also tried just always forcing various pstates for a few applications:
>
> avrora pstate10 4804.4830
> avrora pstate15 3520.0250
> avrora pstate20 2975.5300
> avrora pstate25 3605.5110
> avrora pstate30 3265.1520
> avrora pstate35 3142.0730
> avrora pstate37 3149.4060
>
> h2 pstate10 6100.5350
> h2 pstate15 4440.2950
> h2 pstate20 3731.1560
> h2 pstate25 4924.2250
> h2 pstate30 4375.3220
> h2 pstate35 4227.6440
> h2 pstate37 4181.9290
>
> xalan pstate10 1153.3680
> xalan pstate15 1027.7840
> xalan pstate20 998.0690
> xalan pstate25 1094.4020
> xalan pstate30 1098.2600
> xalan pstate35 1092.1510
> xalan pstate37 1098.5350
>

Nice, so this confirms that the most efficient CPU frequency is roughly
2x the one currently assumed by intel_pstate on your system.  It would
be trivial to work around this locally on your system by forcing
min_pstate to be ~20 via sysfs.  Though of course it would be better to
find the root cause of this deviation.

> For these three cases, the best pstate in terms of CPU energy consumption
> is always 20.  For RAM, faster is always better:
>
> avrora pstate10 2372.9950
> avrora pstate15 1706.6990
> avrora pstate20 1383.3360
> avrora pstate25 1406.3790
> avrora pstate30 1235.5450
> avrora pstate35 1139.7800
> avrora pstate37 1142.9890
>
> h2 pstate10 3239.6100
> h2 pstate15 2321.2250
> h2 pstate20 1886.2960
> h2 pstate25 2030.6580
> h2 pstate30 1731.8120
> h2 pstate35 1635.3940
> h2 pstate37 1607.1940
>
> xalan pstate10 662.1400
> xalan pstate15 556.7600
> xalan pstate20 479.3040
> xalan pstate25 429.1490
> xalan pstate30 407.0890
> xalan pstate35 405.5320
> xalan pstate37 406.9260
>

Yeah, the picture becomes more complicated as one tries to take into
account the energy consumption of the various peripherals your CPU is
talking to, which will typically give you a combined power curve with a
different maximum efficiency point.  Predicting that doesn't seem
possible without additional information not available to intel_pstate
currently, including the set of devices the application is interacting
with, and their respective power curves.

>
>>
>> Attempting to correct that by introducing an additive bias term into the
>> P-state calculation as done in this patch will inevitably pessimize
>> energy usage in the (also fairly common) scenario that the CPU
>> utilization is high enough to push the CPU frequency into the convex
>> region of the power curve, and doesn't really fix the underlying problem
>> that our knowledge about the most efficient P-state may have a
>> substantial error in your system.
>>
>> Judging from the performance improvement you're observing with this, I'd
>> bet that most of the test cases below are fairly latency-bound: They
>> seem like the kind of workloads where a thread may block on something
>> for a significant fraction of the time and then run a burst of CPU work
>> that's not designed to run in parallel with the tasks the same thread
>> will subsequently block on.  That would explain the fact that you're
>> getting low enough utilization values that your change affects the
>> P-state calculation significantly.
>
> The three applications all alternate running and blocking at various fast
> rates.  Small portions of the traces of each one are attached.

Yup, thanks for the traces, seems like the kind of workloads that
greatly underutilize the CPU resources.  It's not surprising to see
schedutil give a suboptimal response in these cases, since the limiting
factor for such latency-bound workloads that spend most of their time
waiting is how quickly the CPU can react to some event and complete a
short non-parallelizable computation, rather than the total amount of
computational resources available to it.

Do you get any better results while using HWP as actual governor
(i.e. when intel_pstate is in active mode) instead of relying on
schedutil?  With schedutil you may be able to get better results in
combination with the deadline scheduler, though that would also need
userspace collaboration.

>
> thanks,
> julia
>
>> As you've probably realized
>> yourself, in such a scenario the optimality assumptions of the current
>> schedutil heuristic break down, however it doesn't seem like
>> intel_pstate has enough information to make up for that problem, if that
>> requires introducing another heuristic which will itself cause us to
>> further deviate from optimality in a different set of scenarios.
>>
>> > julia
>> >
>>
>> Regards,
>> Francisco
>>
>> >
>> >>
>> >> > This patch scales the utilization
>> >> > (target_perf) between the min pstate and the cap pstate instead.
>> >> >
>> >> > On the DaCapo (Java) benchmarks and on a few exmples of kernel compilation
>> >> > (based on make defconfig), on two-socket machines with the above CPUs, the
>> >> > performance is always the same or better as Linux v5.15, and the CPU and
>> >> > RAM energy consumption is likewise always the same or better (one
>> >> > exception: zxing-eval on the 5128 uses a little more energy).
>> >> >
>> >> > 6130:
>> >> >
>> >> > Performance (sec):
>> >> >                 v5.15           with this patch (improvement)
>> >> > avrora          77.5773         56.4090 (1.38)
>> >> > batik-eval      113.1173        112.4135 (1.01)
>> >> > biojava-eval    196.6533        196.7943 (1.00)
>> >> > cassandra-eval  62.6638         59.2800 (1.06)
>> >> > eclipse-eval    218.5988        210.0139 (1.04)
>> >> > fop             3.5537          3.4281 (1.04)
>> >> > graphchi-evalN  13.8668         10.3411 (1.34)
>> >> > h2              75.5018         62.2993 (1.21)
>> >> > jme-eval        94.9531         89.5722 (1.06)
>> >> > jython          23.5789         23.0603 (1.02)
>> >> > kafka-eval      60.2784         59.2057 (1.02)
>> >> > luindex         5.3537          5.1190 (1.05)
>> >> > lusearch-fix    3.5956          3.3628 (1.07)
>> >> > lusearch        3.5396          3.5204 (1.01)
>> >> > pmd             13.3505         10.8795 (1.23)
>> >> > sunflow         7.5932          7.4899 (1.01)
>> >> > tomcat-eval     39.6568         31.4844 (1.26)
>> >> > tradebeans      118.9918        99.3932 (1.20)
>> >> > tradesoap-eval  56.9113         54.7567 (1.04)
>> >> > tradesoap       50.7779         44.5169 (1.14)
>> >> > xalan           5.0711          4.8879 (1.04)
>> >> > zxing-eval      10.5532         10.2435 (1.03)
>> >> >
>> >> > make            45.5977         45.3454 (1.01)
>> >> > make sched      3.4318          3.3450 (1.03)
>> >> > make fair.o     2.9611          2.8464 (1.04)
>> >> >
>> >> > CPU energy consumption (J):
>> >> >
>> >> > avrora          4740.4813       3585.5843 (1.32)
>> >> > batik-eval      13361.34        13278.74 (1.01)
>> >> > biojava-eval    21608.70        21652.94 (1.00)
>> >> > cassandra-eval  3037.6907       2891.8117 (1.05)
>> >> > eclipse-eval    23528.15        23198.36 (1.01)
>> >> > fop             455.7363        441.6443 (1.03)
>> >> > graphchi-eval   999.9220        971.5633 (1.03)
>> >> > h2              5451.3093       4929.8383 (1.11)
>> >> > jme-eval        5343.7790       5143.8463 (1.04)
>> >> > jython          2685.3790       2623.1950 (1.02)
>> >> > kafka-eval      2715.6047       2548.7220 (1.07)
>> >> > luindex         597.7587        571.0387 (1.05)
>> >> > lusearch-fix    714.0340        692.4727 (1.03)
>> >> > lusearch        718.4863        704.3650 (1.02)
>> >> > pmd             1627.6377       1497.5437 (1.09)
>> >> > sunflow         1563.5173       1514.6013 (1.03)
>> >> > tomcat-eval     4740.1603       4539.1503 (1.04)
>> >> > tradebeans      8331.2260       7482.3737 (1.11)
>> >> > tradesoap-eval  6610.1040       6426.7077 (1.03)
>> >> > tradesoap       5641.9300       5544.3517 (1.02)
>> >> > xalan           1072.0363       1065.7957 (1.01)
>> >> > zxing-eval      2200.1883       2174.1137 (1.01)
>> >> >
>> >> > make            9788.9290       9777.5823 (1.00)
>> >> > make sched      501.0770        495.0600 (1.01)
>> >> > make fair.o     363.4570        352.8670 (1.03)
>> >> >
>> >> > RAM energy consumption (J):
>> >> >
>> >> > avrora          2508.5553       1844.5977 (1.36)
>> >> > batik-eval      5627.3327       5603.1820 (1.00)
>> >> > biojava-eval    9371.1417       9351.1543 (1.00)
>> >> > cassandra-eval  1398.0567       1289.8317 (1.08)
>> >> > eclipse-eval    10193.28        9952.3543 (1.02)
>> >> > fop             189.1927        184.0620 (1.03)
>> >> > graphchi-eval   539.3947        447.4557 (1.21)
>> >> > h2              2771.0573       2432.2587 (1.14)
>> >> > jme-eval        2702.4030       2504.0783 (1.08)
>> >> > jython          1135.7317       1114.5190 (1.02)
>> >> > kafka-eval      1320.6840       1220.6867 (1.08)
>> >> > luindex         246.6597        237.1593 (1.04)
>> >> > lusearch-fix    294.4317        282.2193 (1.04)
>> >> > lusearch        295.5400        284.3890 (1.04)
>> >> > pmd             721.7020        643.1280 (1.12)
>> >> > sunflow         568.6710        549.3780 (1.04)
>> >> > tomcat-eval     2305.8857       1995.8843 (1.16)
>> >> > tradebeans      4323.5243       3749.7033 (1.15)
>> >> > tradesoap-eval  2862.8047       2783.5733 (1.03)
>> >> > tradesoap       2717.3900       2519.9567 (1.08)
>> >> > xalan           430.6100        418.5797 (1.03)
>> >> > zxing-eval      732.2507        710.9423 (1.03)
>> >> >
>> >> > make            3362.8837       3356.2587 (1.00)
>> >> > make sched      191.7917        188.8863 (1.02)
>> >> > make fair.o     149.6850        145.8273 (1.03)
>> >> >
>> >> > 5128:
>> >> >
>> >> > Performance (sec):
>> >> >
>> >> > avrora          62.0511         43.9240 (1.41)
>> >> > batik-eval      111.6393        110.1999 (1.01)
>> >> > biojava-eval    241.4400        238.7388 (1.01)
>> >> > cassandra-eval  62.0185         58.9052 (1.05)
>> >> > eclipse-eval    240.9488        232.8944 (1.03)
>> >> > fop             3.8318          3.6408 (1.05)
>> >> > graphchi-eval   13.3911         10.4670 (1.28)
>> >> > h2              75.3658         62.8218 (1.20)
>> >> > jme-eval        95.0131         89.5635 (1.06)
>> >> > jython          28.1397         27.6802 (1.02)
>> >> > kafka-eval      60.4817         59.4780 (1.02)
>> >> > luindex         5.1994          4.9587 (1.05)
>> >> > lusearch-fix    3.8448          3.6519 (1.05)
>> >> > lusearch        3.8928          3.7068 (1.05)
>> >> > pmd             13.0990         10.8008 (1.21)
>> >> > sunflow         7.7983          7.8569 (0.99)
>> >> > tomcat-eval     39.2064         31.7629 (1.23)
>> >> > tradebeans      120.8676        100.9113 (1.20)
>> >> > tradesoap-eval  65.5552         63.3493 (1.03)
>> >> > xalan           5.4463          5.3576 (1.02)
>> >> > zxing-eval      9.8611          9.9692 (0.99)
>> >> >
>> >> > make            43.1852         43.1285 (1.00)
>> >> > make sched      3.2181          3.1706 (1.01)
>> >> > make fair.o     2.7584          2.6615 (1.04)
>> >> >
>> >> > CPU energy consumption (J):
>> >> >
>> >> > avrora          3979.5297       3049.3347 (1.31)
>> >> > batik-eval      12339.59        12413.41 (0.99)
>> >> > biojava-eval    23935.18        23931.61 (1.00)
>> >> > cassandra-eval  3552.2753       3380.4860 (1.05)
>> >> > eclipse-eval    24186.38        24076.57 (1.00)
>> >> > fop             441.0607        442.9647 (1.00)
>> >> > graphchi-eval   1021.1323       964.4800 (1.06)
>> >> > h2              5484.9667       4901.9067 (1.12)
>> >> > jme-eval        6167.5287       5909.5767 (1.04)
>> >> > jython          2956.7150       2986.3680 (0.99)
>> >> > kafka-eval      3229.9333       3197.7743 (1.01)
>> >> > luindex         537.0007        533.9980 (1.01)
>> >> > lusearch-fix    720.1830        699.2343 (1.03)
>> >> > lusearch        708.8190        700.7023 (1.01)
>> >> > pmd             1539.7463       1398.1850 (1.10)
>> >> > sunflow         1533.3367       1497.2863 (1.02)
>> >> > tomcat-eval     4551.9333       4289.2553 (1.06)
>> >> > tradebeans      8527.2623       7570.2933 (1.13)
>> >> > tradesoap-eval  6849.3213       6750.9687 (1.01)
>> >> > xalan           1013.2747       1019.1217 (0.99)
>> >> > zxing-eval      1852.9077       1943.1753 (0.95)
>> >> >
>> >> > make            9257.5547       9262.5993 (1.00)
>> >> > make sched      438.7123        435.9133 (1.01)
>> >> > make fair.o     315.6550        312.2280 (1.01)
>> >> >
>> >> > RAM energy consumption (J):
>> >> >
>> >> > avrora          16309.86        11458.08 (1.42)
>> >> > batik-eval      30107.11        29891.58 (1.01)
>> >> > biojava-eval    64290.01        63941.71 (1.01)
>> >> > cassandra-eval  13240.04        12403.19 (1.07)
>> >> > eclipse-eval    64188.41        62008.35 (1.04)
>> >> > fop             1052.2457       996.0907 (1.06)
>> >> > graphchi-eval   3622.5130       2856.1983 (1.27)
>> >> > h2              19965.58        16624.08 (1.20)
>> >> > jme-eval        21777.02        20211.06 (1.08)
>> >> > jython          7515.3843       7396.6437 (1.02)
>> >> > kafka-eval      12868.39        12577.32 (1.02)
>> >> > luindex         1387.7263       1328.8073 (1.04)
>> >> > lusearch-fix    1313.1220       1238.8813 (1.06)
>> >> > lusearch        1303.5597       1245.4130 (1.05)
>> >> > pmd             3650.6697       3049.8567 (1.20)
>> >> > sunflow         2460.8907       2380.3773 (1.03)
>> >> > tomcat-eval     11199.61        9232.8367 (1.21)
>> >> > tradebeans      32385.99        26901.40 (1.20)
>> >> > tradesoap-eval  17691.01        17006.95 (1.04)
>> >> > xalan           1783.7290       1735.1937 (1.03)
>> >> > zxing-eval      2812.9710       2952.2933 (0.95)
>> >> >
>> >> > make            13247.47        13258.64 (1.00)
>> >> > make sched      885.7790        877.1667 (1.01)
>> >> > make fair.o     741.2473        723.6313 (1.02)
>> >>
>> >> So the number look better after the change, because it makes the
>> >> driver ask the hardware for slightly more performance than it is asked
>> >> for by the governor.
>> >>
>> >> >
>> >> > Signed-off-by: Julia Lawall <julia.lawall@...ia.fr>
>> >> >
>> >> > ---
>> >> >
>> >> > min_pstate is defined in terms of cpu->pstate.min_pstate and
>> >> > cpu->min_perf_ratio.  Maybe one of these values should be used instead.
>> >> > Likewise, perhaps cap_pstate should be max_pstate?
>> >>
>> >> I'm not sure if I understand this remark.  cap_pstate is the max
>> >> performance level of the CPU and max_pstate is the current limit
>> >> imposed by the framework.  They are different things.
>> >>
>> >> >
>> >> > diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c
>> >> > index 8c176b7dae41..ba6a48959754 100644
>> >> > --- a/drivers/cpufreq/intel_pstate.c
>> >> > +++ b/drivers/cpufreq/intel_pstate.c
>> >> > @@ -2789,10 +2789,6 @@ static void intel_cpufreq_adjust_perf(unsigned int cpunum,
>> >> >
>> >> >         /* Optimization: Avoid unnecessary divisions. */
>> >> >
>> >> > -       target_pstate = cap_pstate;
>> >> > -       if (target_perf < capacity)
>> >> > -               target_pstate = DIV_ROUND_UP(cap_pstate * target_perf, capacity);
>> >> > -
>> >> >         min_pstate = cap_pstate;
>> >> >         if (min_perf < capacity)
>> >> >                 min_pstate = DIV_ROUND_UP(cap_pstate * min_perf, capacity);
>> >> > @@ -2807,6 +2803,10 @@ static void intel_cpufreq_adjust_perf(unsigned int cpunum,
>> >> >         if (max_pstate < min_pstate)
>> >> >                 max_pstate = min_pstate;
>> >> >
>> >> > +       target_pstate = cap_pstate;
>> >> > +       if (target_perf < capacity)
>> >> > +               target_pstate = DIV_ROUND_UP((cap_pstate - min_pstate) * target_perf, capacity) + min_pstate;
>> >>
>> >> So the driver is asked by the governor to deliver the fraction of the
>> >> max performance (cap_pstate) given by the target_perf / capacity ratio
>> >> with the floor given by min_perf / capacity.  It cannot turn around
>> >> and do something else, because it thinks it knows better.
>> >>
>> >> > +
>> >> >         target_pstate = clamp_t(int, target_pstate, min_pstate, max_pstate);
>> >> >
>> >> >         intel_cpufreq_hwp_update(cpu, min_pstate, max_pstate, target_pstate, true);
>> >>
>>

Powered by blists - more mailing lists