lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <hpyrujaoqd3qpd46ttrzpcj4hsbxeubfk55zw2k4tvezljxbvq@jeadvbhwdirf>
Date: Sun, 31 Aug 2025 18:29:18 -0400
From: Aaron Tomlin <atomlin@...mlin.com>
To: Mel Gorman <mgorman@...hsingularity.net>, 
	Peter Zijlstra <peterz@...radead.org>
Cc: tglx@...utronix.de, mingo@...hat.com, bp@...en8.de, 
	dave.hansen@...ux.intel.com, x86@...nel.org, juri.lelli@...hat.com, 
	vincent.guittot@...aro.org, hpa@...or.com, oleg@...hat.com, dietmar.eggemann@....com, 
	rostedt@...dmis.org, bsegall@...gle.com, mgorman@...e.de, vschneid@...hat.com, 
	linux-kernel@...r.kernel.org, atomlin@...mlin.com
Subject: Re: [RFC PATCH] sched: idle: Introduce CPU-specific idle=poll

On Mon, Jun 23, 2025 at 10:49:59PM +0100, Mel Gorman wrote:
> On Mon, Jun 23, 2025 at 12:23:34PM +0200, Peter Zijlstra wrote:
> > On Sat, Jun 21, 2025 at 07:57:45PM -0400, Aaron Tomlin wrote:
> > > Currently, the idle=poll kernel boot parameter applies globally, forcing
> > > all CPUs into a shallow polling idle state to ensure ultra-low latency
> > > responsiveness. While this is beneficial for extremely latency-sensitive
> > > workloads, this global application lacks flexibility and can lead to
> > > significant power inefficiency. This is particularly evident in systems
> > > with a high CPU count, such as those utilising the
> > > Full Dynticks/Adaptive Tick feature (i.e., nohz_full). In such
> > > environments, only a subset of CPUs might genuinely require
> > > sub-microsecond responsiveness, while others, though active, could
> > > benefit from entering deeper idle states to conserve power.
> > 
> > Can't we already do this at runtime with pmqos? If you set your latency
> > demand very low, it should end up picking the poll state, no? And you
> > can do this per-cpu.
> 
> Yes, we can. idle=poll can be hazardous in weird ways and it's not like
> pmqos is hard to use. For example, lets say you had a RT application with
> latency constraints running on isolated CPUs while leaving housekeeping
> CPUs alone then it's simply a case of;
> 
>         for CPU in $ISOLATED_CPUS; do
>                 SYSFS_PARAM="/sys/devices/system/cpu/cpu$CPU/power/pm_qos_resume_latency_us"
>                 if [ ! -e $SYSFS_PARAM ]; then
>                         echo "WARNING: Unable to set PM QOS max latency for CPU $CPU\n"
>                         continue
>                 fi
>                 echo $MAX_EXIT_LATENCY > $SYSFS_PARAM
>                 echo "Set PM QOS maximum resume latency on CPU $CPU to ${MAX_EXIT_LATENCY}us"
>         done
>  
> 
> In too many cases I've seen idle=poll being used when the user didn't know
> PM QOS existed. The most common response I've received is that the latency
> requirements were unknown resulting in much headbanging off the table.
> Don't get me started on the hazards of limiting c-states by index without
> checking that the c-states are or splitting isolated/housekeeping across
> SMT siblings.

Mel, Peter,

Yes, I can confirm the PM QoS subsystem allows one to set constraints on a
per-CPU basis or for the entire system. One can specify a maximum allowed
latency constraint. As per function pm_qos_resume_latency_us_store() a
value of "n/a" will prevent the specified CPU from entering the shallowest
CPU idle-state (namely "C1") given its exit latency constraint.
Indeed using "idle=poll" to prevent a CPU from entering idle C-states is
problematic given its crude, all-or-nothing approach.

Thank you for the suggestion.



  ...
    do_idle
      cpuidle_idle_call
      {
        next_state = cpuidle_select(drv, dev, &stop_tick)
                     // cpuidle_curr_governor->select(drv, dev, stop_tick)
                     menu_select(drv, dev, stop_tick)
                     {
                       latency_req = cpuidle_governor_latency_req(dev->cpu)
                                     {
                                       *device = get_cpu_device(cpu)
                                       device_req = dev_pm_qos_raw_resume_latency(device)
                                       global_req = cpu_latency_qos_limit()

                                       if (device_req > global_req)
                                         device_req = global_req

                                       return (s64)device_req * NSEC_PER_USEC
                                     }

                       if (unlikely(drv->state_count <= 1 || latency_req == 0) || ...
                            ... ) {
                         // A CPU idle driver which more than one C-state and a
                         // latency requirement of 0 will force C0 
                         *stop_tick = !(drv->states[0].flags & CPUIDLE_FLAG_POLLING)
                         return 0
                       }
                     }

        entered_state = call_cpuidle(drv, dev, next_state)
      }

crash> p cpuidle_curr_governor 
cpuidle_curr_governor = $1 = (struct cpuidle_governor *) 0xffffffff9ab913e0 <menu_governor>

crash> p cpuidle_curr_governor.select
$2 = (int (*)(struct cpuidle_driver *, struct cpuidle_device *, bool *)) 0xffffffff99157ed0 <menu_select>

crash> p cpuidle_curr_driver 
cpuidle_curr_driver = $3 = (struct cpuidle_driver *) 0xffffffff9ab04dc0 <intel_idle_driver>

crash> p ((struct cpuidle_driver *)0xffffffff9ab04dc0)->states[0].enter
$4 = (int (*)(struct cpuidle_device *, struct cpuidle_driver *, int)) 0xffffffff994d32a0 <poll_idle>

crash> p -d ((struct cpuidle_driver *)0xffffffff9ab04dc0)->states[1].exit_latency_ns
$5 = 2000

crash> p -d ((struct cpuidle_driver *)0xffffffff9ab04dc0)->states[0].exit_latency_ns
$6 = 0

# cat /sys/devices/system/cpu/cpuidle/current_driver
intel_idle

# cat /sys/devices/system/cpu/cpu7/cpuidle/state0/name 
POLL

# cat /sys/devices/system/cpu/cpu7/cpuidle/state0/latency 
0

# echo "n/a" > /sys/devices/system/cpu/cpu7/power/pm_qos_resume_latency_us

#
# Samples: 2K of event 'cpu-cycles:k'
# Event count (approx.): 89401819821
#
# Children      Self  Command          Shared Object                                   Symbol                                            
# ........  ........  ...............  ..............................................  ..................................................
#
    99.80%     0.05%  swapper          [kernel.kallsyms]                               [k] do_idle
    99.80%     0.00%  swapper          [kernel.kallsyms]                               [k] common_startup_64
    99.80%     0.00%  swapper          [kernel.kallsyms]                               [k] cpu_startup_entry
    99.80%     0.00%  swapper          [kernel.kallsyms]                               [k] start_secondary
    99.75%     0.00%  swapper          [kernel.kallsyms]                               [k] cpuidle_idle_call
    99.34%     0.05%  swapper          [kernel.kallsyms]                               [k] cpuidle_enter_state
    99.34%     0.00%  swapper          [kernel.kallsyms]                               [k] cpuidle_enter
    98.97%    98.56%  swapper          [kernel.kallsyms]                               [k] poll_idle


swapper       0 [007] 1203865.059685:    6998090 cpu-cycles:k: 
        ffffffff994d32f8 poll_idle+0x58 ([kernel.kallsyms])
        ffffffff994d16b4 cpuidle_enter_state+0x84 ([kernel.kallsyms])
        ffffffff99156241 cpuidle_enter+0x31 ([kernel.kallsyms])
        ffffffff9844ab77 cpuidle_idle_call+0xf7 ([kernel.kallsyms])
        ffffffff9844ac68 do_idle+0x78 ([kernel.kallsyms])
        ffffffff9844aec9 cpu_startup_entry+0x29 ([kernel.kallsyms])
        ffffffff983663db start_secondary+0x12b ([kernel.kallsyms])
        ffffffff9831452d common_startup_64+0x13e ([kernel.kallsyms])

-- 
Aaron Tomlin

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ