linux-kernel - Re: [PATCH] tick/broadcast: Do not set oneshot_mask except was

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Mon, 3 Apr 2023 12:26:27 +0200
From:   Frederic Weisbecker <frederic@...nel.org>
To:     Victor Hassan <victor@...winnertech.com>
Cc:     fweisbec@...il.com, tglx@...utronix.de, mingo@...nel.org,
        jindong.yue@....com, linux-kernel@...r.kernel.org
Subject: Re: [PATCH] tick/broadcast: Do not set oneshot_mask except
 was_periodic was true

On Tue, Mar 28, 2023 at 02:36:29PM +0800, Victor Hassan wrote:
> If a broadcast timer is registered after the system switched to oneshot
> mode, a hang_task err could occur like that:
> 
> INFO: task kworker/u15:0:7 blocked for more than 120 seconds.
>       Tainted: G            E     5.15.41-android13-8-00002-xxx #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:kworker/u16:0   state:D stack: 9808 pid:  7 ppid: 2 flags:0x00000008
> Workqueue: events_unbound deferred_probe_work_func.cfi_jt
> Call trace:
>  __switch_to+0y240/0x490
>  __schedule+0x620/0xafc
>  schedule+0x110/0x204
>  schedule_hrtimeout_range_clock+0x9c/0x118
>  usleep_range_state+0x150/0x1ac
>  _regulator_do_enable+0x528/0x878
>  set_machine_constraints+0x6a0/0xf2c
>  regulator_register+0x3ac/0x7ac
>  devm_regulator_register+0xbc/0x120
>  pmu_ext_regulator_probe+0xb0/0x1b4 [pmu_ext_regulator]
>  platform_probe+0x70/0x194
>  really_proe+0x320/0x68c
>  __driver_probe_device+0x204/0x260
>  driver_probe_device+0x48/0x1e0
> 
> When the new broadcast timer was registered after the system switched
> to oneshot mode, the broadcast timer was not used as periodic. If the
> oneshot mask was set incorrectly, all cores which did not enter cpu_idle
> state can't enter cpu_idle normally, causing the hrtimer mechanism to
> break.
> 
> This patch fixes the issue by moving the update action about oneshot
> mask to a more strict conditions. The tick_broadcast_setup_oneshot would
> be called in two typical condition, and they all will work.
> 
> 1. tick_handle_periodic -> tick_broadcast_setup_oneshot
> 
> The origin broadcast was periodic, so it can set the oneshot_mask bits
> for those waiting for periodic broadcast and program the broadcast timer
> to fire.
> 
> 2. tick_install_broadcast_device -> tick_broadcast_setup_oneshot
> 
> The origin broadcast was oneshot, so the cores which enter the cpu_idle
> already used the oneshot_mask bits. It is unnecessary to update the
> oneshot_mask.
> 
> Fixes: 9c336c9935cf ("tick/broadcast: Allow late registered device to enter oneshot mode")
> 
> Signed-off-by: Victor Hassan <victor@...winnertech.com>
> ---
>  kernel/time/tick-broadcast.c | 5 +++--
>  1 file changed, 3 insertions(+), 2 deletions(-)
> 
> diff --git a/kernel/time/tick-broadcast.c b/kernel/time/tick-broadcast.c
> index 93bf2b4e47e5..fdbbba487978 100644
> --- a/kernel/time/tick-broadcast.c
> +++ b/kernel/time/tick-broadcast.c
> @@ -1041,12 +1041,13 @@ static void tick_broadcast_setup_oneshot(struct clock_event_device *bc)
>  		 */
>  		cpumask_copy(tmpmask, tick_broadcast_mask);
>  		cpumask_clear_cpu(cpu, tmpmask);
> -		cpumask_or(tick_broadcast_oneshot_mask,
> -			   tick_broadcast_oneshot_mask, tmpmask);
>  
>  		if (was_periodic && !cpumask_empty(tmpmask)) {
>  			ktime_t nextevt = tick_get_next_period();
>  
> +			cpumask_or(tick_broadcast_oneshot_mask,
> +				   tick_broadcast_oneshot_mask, tmpmask);
> +

Good catch, it looks like one issue that can trigger is due to the resulting
ignored calls to tick_broadcast_exit(). Indeed if the cpu is already in
tick_broadcast_oneshot_mask then cpuidle won't call the exit.

Leading to such race:

* CPU 1 stop its tick, next event is in one hour
* CPU 0 registers new broadcast and sets CPU 1 in tick_broadcast_oneshot_mask
* CPU 1 runs into cpuidle_enter_state(), and tick_broadcast_enter() is ignored because
  the CPU is already in tick_broadcast_oneshot_mask
* CPU 1 goes to sleep
* CPU 0 runs the broadcast callback, sees that the next timer for CPU 1
  is in one hour, program the broadcast to that deadline
* CPU 1 gets an interrupt that enqueues a new timer expiring in the next jiffy
* CPU 1 don't call tick_broadcast_exit and thus don't remove itself from
  tick_broadcast_oneshot_mask
* CPU 1 re-enters in cpuidle_enter_state(), tick_broadcast_enter() is again
  ignored so the new timer isn't propagated to the broadcast.
* CPU 1 goes to sleep and won't be woken before one hour.


  Reviewed-by: Frederic Weisbecker <frederic@...nel.org>


Thanks.




>  			clockevents_switch_state(bc, CLOCK_EVT_STATE_ONESHOT);
>  			tick_broadcast_init_next_event(tmpmask, nextevt);
>  			tick_broadcast_set_event(bc, cpu, nextevt);
> -- 
> 2.29.0
>