linux-kernel - [PATCH 09/25] kcpustat: Track running task following vtime sequences

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-Id: <1542163569-20047-10-git-send-email-frederic@kernel.org>
Date:   Wed, 14 Nov 2018 03:45:53 +0100
From:   Frederic Weisbecker <frederic@...nel.org>
To:     LKML <linux-kernel@...r.kernel.org>
Cc:     Frederic Weisbecker <frederic@...nel.org>,
        Peter Zijlstra <peterz@...radead.org>,
        Wanpeng Li <wanpengli@...cent.com>,
        Thomas Gleixner <tglx@...utronix.de>,
        Yauheni Kaliuta <yauheni.kaliuta@...hat.com>,
        Ingo Molnar <mingo@...nel.org>, Rik van Riel <riel@...hat.com>
Subject: [PATCH 09/25] kcpustat: Track running task following vtime sequences

In order to make kcpustat vtime aware (ie: work on nohz_full without
freezing), we need to track the task running on the CPU in order to
fetch its vtime delta and add it to the relevant kcpustat field.

The most efficient way to track this task is to use RCU. The task is
assigned on context switch right after we flush the vtime of the previous
task and the next task has been set on vtime.

Things are then prepared to be ordered that way:

             WRITER (ctx switch)                READER
             ------------------            -----------------------
        vtime_seqcount_write_lock(prev)     rcu_read_lock()
        //flush prev vtime                  curr = rcu_dereference(kcpustat->curr)
        vtime_seqcount_write_unlock(prev)   vtime_seqcount_read_start(curr)
                                            //fetch curr vtime
        vtime_seqcount_lock(next)           vtime_seqcount_read_end(curr)
        //Init vtime                        rcu_read_unlock()
        vtime_seqcount_unlock(next)

        rcu_assign_pointer(kcpustat->curr, next)

With this ordering layout, we are sure that we get a sequence with a
coherent couple (task cputime, kcpustat).

Signed-off-by: Frederic Weisbecker <frederic@...nel.org>
Cc: Yauheni Kaliuta <yauheni.kaliuta@...hat.com>
Cc: Thomas Gleixner <tglx@...utronix.de>
Cc: Rik van Riel <riel@...hat.com>
Cc: Peter Zijlstra <peterz@...radead.org>
Cc: Wanpeng Li <wanpengli@...cent.com>
Cc: Ingo Molnar <mingo@...nel.org>
---
 include/linux/kernel_stat.h |  1 +
 kernel/sched/cputime.c      | 11 ++++++++++-
 2 files changed, 11 insertions(+), 1 deletion(-)

diff --git a/include/linux/kernel_stat.h b/include/linux/kernel_stat.h
index 7ee2bb4..86fdbce 100644
--- a/include/linux/kernel_stat.h
+++ b/include/linux/kernel_stat.h
@@ -32,6 +32,7 @@ enum cpu_usage_stat {
 };
 
 struct kernel_cpustat {
+	struct task_struct __rcu *curr;
 	u64 cpustat[NR_STATS];
 };
 
diff --git a/kernel/sched/cputime.c b/kernel/sched/cputime.c
index a0c3a82..2eb313a 100644
--- a/kernel/sched/cputime.c
+++ b/kernel/sched/cputime.c
@@ -812,6 +812,7 @@ void vtime_account_idle(struct task_struct *tsk)
 void vtime_task_switch_generic(struct task_struct *prev)
 {
 	struct vtime *vtime = &prev->vtime;
+	struct kernel_cpustat *kcpustat = kcpustat_this_cpu;
 
 	/*
 	 * Flush the prev task vtime, unless it has passed
@@ -835,8 +836,10 @@ void vtime_task_switch_generic(struct task_struct *prev)
 	 * Ignore the next task if it has been preempted after
 	 * vtime_exit_task().
 	 */
-	if (vtime->state == VTIME_DEAD)
+	if (vtime->state == VTIME_DEAD) {
+		rcu_assign_pointer(kcpustat->curr, NULL);
 		return;
+	}
 
 	write_seqcount_begin(&vtime->seqcount);
 	if (is_idle_task(current))
@@ -848,10 +851,13 @@ void vtime_task_switch_generic(struct task_struct *prev)
 	vtime->starttime = sched_clock();
 	vtime->cpu = smp_processor_id();
 	write_seqcount_end(&vtime->seqcount);
+
+	rcu_assign_pointer(kcpustat->curr, current);
 }
 
 void vtime_init_idle(struct task_struct *t, int cpu)
 {
+	struct kernel_cpustat *kcpustat = &kcpustat_cpu(cpu);
 	struct vtime *vtime = &t->vtime;
 	unsigned long flags;
 
@@ -862,6 +868,8 @@ void vtime_init_idle(struct task_struct *t, int cpu)
 	vtime->cpu = cpu;
 	write_seqcount_end(&vtime->seqcount);
 	local_irq_restore(flags);
+
+	rcu_assign_pointer(kcpustat->curr, t);
 }
 
 /*
@@ -885,6 +893,7 @@ void vtime_exit_task(struct task_struct *t)
 	vtime->state = VTIME_DEAD;
 	vtime->cpu = -1;
 	write_seqcount_end(&vtime->seqcount);
+	rcu_assign_pointer(kcpustat_this_cpu->curr, NULL);
 	local_irq_restore(flags);
 }
 
-- 
2.7.4