linux-kernel - sched/fair: Long delays starting RT processes on idle cpu.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <13797bbe87b64f34877b89a5bbdb6d03@AcuMS.aculab.com>
Date:   Mon, 27 Jan 2020 15:39:24 +0000
From:   David Laight <David.Laight@...LAB.COM>
To:     'Steven Rostedt' <rostedt@...dmis.org>
CC:     'linux-kernel' <linux-kernel@...r.kernel.org>
Subject: sched/fair: Long delays starting RT processes on idle cpu.

I'm seeing long delays (eg up to 90us) between a process being scheduled on an idle cpu
and the process actually executing.

This is an ivy bridge cpu:
model name      : Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz

[    0.583178] intel_idle: MWAIT substates: 0x1120
[    0.583178] intel_idle: v0.4.1 model 0x3A
[    0.583293] intel_idle: lapic_timer_reliable_states 0xffffffff

I picked one wakeup out of some ftrace data.
A short extract:
Process 10318 (the one that is woken) goes to sleep.
...
          <idle>-0     [003] d... 945687.207448: sched_idle_set_state <-cpuidle_enter_state
          <idle>-0     [003] d... 945687.207449: intel_idle <-cpuidle_enter_state
          <idle>-0     [003] d... 945687.207449: leave_mm <-intel_idle
          <idle>-0     [003] d... 945687.207449: switch_mm <-intel_idle
          <idle>-0     [003] d... 945687.207449: switch_mm_irqs_off <-switch_mm

I think cpu-3 is now executing the MWAIT from mwait_idle_with_hints()
called from intel_idle() in drivers/idle/intel_idle.c.

Not much happens on any of the cpu for 10ms until a process wakes up from a timed delay
and then tries to wake up the other threads.

    ProsodySServ-10319 [000] d... 945687.217414: sched_wakeup: comm=ProsodySServ pid=10318 prio=3 target_cpu=003
    ProsodySServ-10319 [000] d... 945687.217414: task_woken_rt <-ttwu_do_wakeup
    ProsodySServ-10319 [000] d... 945687.217414: _raw_spin_unlock_irqrestore <-try_to_wake_up
    ProsodySServ-10319 [000] .... 945687.217414: drop_futex_key_refs.isra.14 <-futex_wake

I presume the above is supposed to have done a memory write that wakes the MWAIT.

There is now a long delay (26us here) before anything happens.
          <idle>-0     [003] dN.. 945687.217440: sched_idle_set_state <-cpuidle_enter_state
...
          <idle>-0     [003] d... 945687.217447: sched_switch: prev_comm=swapper/3 prev_pid=0 prev_prio=120 prev_state=R ==> next_comm=ProsodySServ next_pid=10318 next_prio=3

I'd have thought that the processor should wake up much faster than that.
I can't see the memory write that is paired with the monitor/mwait.
Does it need a strong barrier?

	David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
Registration No: 1397386 (Wales)