Message-ID: <20221129144834.GW4001@paulmck-ThinkPad-P17-Gen-1>
Date:   Tue, 29 Nov 2022 06:48:34 -0800
From:   "Paul E. McKenney" <paulmck@...nel.org>
To:     Frederic Weisbecker <frederic@...nel.org>
Cc:     LKML <linux-kernel@...r.kernel.org>,
        "Eric W . Biederman" <ebiederm@...ssion.com>,
        Neeraj Upadhyay <quic_neeraju@...cinc.com>,
        Oleg Nesterov <oleg@...hat.com>,
        Pengfei Xu <pengfei.xu@...el.com>,
        Boqun Feng <boqun.feng@...il.com>,
        Lai Jiangshan <jiangshanlai@...il.com>, rcu@...r.kernel.org
Subject: Re: [PATCH 0/3] rcu-tasks: Fix race against exiting pid_ns

On Tue, Nov 29, 2022 at 10:55:00AM +0100, Frederic Weisbecker wrote:
> On Mon, Nov 28, 2022 at 04:22:40PM -0800, Paul E. McKenney wrote:
> > On Fri, Nov 25, 2022 at 02:54:57PM +0100, Frederic Weisbecker wrote:
> > > Pengfei Xu has reported a deadlock involving calls to unshare(),
> > > perf_event_open() and clone3() calls. It requires CAP_SYS_ADMIN
> > > to reproduce (at least I don't see a way for a non privilege process to
> > > reproduce).
> > > 
> > > See this thread for details: https://lore.kernel.org/all/Y3sOgrOmMQqPMItu@xpf.sh.intel.com/
> > > And this document for the collaborative analysis with Boqun, Paul and Neeraj:
> > > https://docs.google.com/document/d/1hJxgiZ5TMZ4YJkdJPLAkRvq7sYQ-A7svgA8no6i-v8k
> > > 
> > > The two first patches are small improvements. The fix is in the last patch.
> > > 
> > > git://git.kernel.org/pub/scm/linux/kernel/git/frederic/linux-dynticks.git
> > > 	rcu/dev
> > > 
> > > HEAD: 45ef5a0a4be4e0db9eadcc86e8f346d34c62e744
> > 
> > Hearing no objections, queued for further review and testing.
> > 
> > And thank you very much!  That race between synchronize_rcu_tasks() and
> > zap_pid_ns_processes() certainly was more than a bit on the non-trivial
> > side.  Good show!!!
> 
> Thanks!
> 
> Also please replace the last patch with the following to fix
> a !CONFIG_RCU_TASKS issue:

Like this?  ;-)

a0c355bbdfee ("squash! rcu-tasks: Fix synchronize_rcu_tasks() VS zap_pid_ns_processes()")

							Thanx, Paul

> ---
> From: Frederic Weisbecker <frederic@...nel.org>
> Date: Thu, 24 Nov 2022 18:15:46 +0100
> Subject: [PATCH] rcu-tasks: Fix synchronize_rcu_tasks() VS
>  zap_pid_ns_processes()
> 
> RCU Tasks and PID-namespace unshare can interact in do_exit() in a
> complicated circular dependency:
> 
> 1) TASK A calls unshare(CLONE_NEWPID), this creates a new PID namespace
>    that every subsequent child of TASK A will belong to. But TASK A
>    doesn't itself belong to that new PID namespace.
> 
> 2) TASK A forks() and creates TASK B. TASK A stays attached to its PID
>    namespace (let's say PID_NS1) and TASK B is the first task belonging
>    to the new PID namespace created by unshare()  (let's call it PID_NS2).
> 
> 3) Since TASK B is the first task attached to PID_NS2, it becomes the
>    PID_NS2 child reaper.
> 
> 4) TASK A forks() again and creates TASK C which get attached to PID_NS2.
>    Note how TASK C has TASK A as a parent (belonging to PID_NS1) but has
>    TASK B (belonging to PID_NS2) as a pid_namespace child_reaper.
> 
> 5) TASK B exits and since it is the child reaper for PID_NS2, it has to
>    kill all other tasks attached to PID_NS2, and wait for all of them to
>    die before getting reaped itself (zap_pid_ns_process()).
> 
> 6) TASK A calls synchronize_rcu_tasks() which leads to
>    synchronize_srcu(&tasks_rcu_exit_srcu).
> 
> 7) TASK B is waiting for TASK C to get reaped. But TASK B is under a
>    tasks_rcu_exit_srcu SRCU critical section (exit_notify() is between
>    exit_tasks_rcu_start() and exit_tasks_rcu_finish()), blocking TASK A.
> 
> 8) TASK C exits and since TASK A is its parent, it waits for it to reap
>    TASK C, but it can't because TASK A waits for TASK B that waits for
>    TASK C.
> 
> Pid_namespace semantics can hardly be changed at this point. But the
> coverage of tasks_rcu_exit_srcu can be reduced instead.
> 
> The current task is assumed not to be concurrently reapable at this
> stage of exit_notify() and therefore tasks_rcu_exit_srcu can be
> temporarily relaxed without breaking its constraints, providing a way
> out of the deadlock scenario.
> 
> Fixes: 3f95aa81d265 ("rcu: Make TASKS_RCU handle tasks that are almost done exiting")
> Reported-by: Pengfei Xu <pengfei.xu@...el.com>
> Suggested-by: Boqun Feng <boqun.feng@...il.com>
> Suggested-by: Neeraj Upadhyay <quic_neeraju@...cinc.com>
> Suggested-by: Paul E. McKenney <paulmck@...nel.org>
> Cc: Oleg Nesterov <oleg@...hat.com>
> Cc: Lai Jiangshan <jiangshanlai@...il.com>
> Cc: Eric W . Biederman <ebiederm@...ssion.com>
> Signed-off-by: Frederic Weisbecker <frederic@...nel.org>
> ---
>  include/linux/rcupdate.h |  2 ++
>  kernel/pid_namespace.c   | 17 +++++++++++++++++
>  kernel/rcu/tasks.h       | 15 +++++++++++++--
>  3 files changed, 32 insertions(+), 2 deletions(-)
> 
> diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h
> index 89b3036746d2..a19d91d5461c 100644
> --- a/include/linux/rcupdate.h
> +++ b/include/linux/rcupdate.h
> @@ -238,6 +238,7 @@ void synchronize_rcu_tasks_rude(void);
>  
>  #define rcu_note_voluntary_context_switch(t) rcu_tasks_qs(t, false)
>  void exit_tasks_rcu_start(void);
> +void exit_tasks_rcu_stop(void);
>  void exit_tasks_rcu_finish(void);
>  #else /* #ifdef CONFIG_TASKS_RCU_GENERIC */
>  #define rcu_tasks_classic_qs(t, preempt) do { } while (0)
> @@ -246,6 +247,7 @@ void exit_tasks_rcu_finish(void);
>  #define call_rcu_tasks call_rcu
>  #define synchronize_rcu_tasks synchronize_rcu
>  static inline void exit_tasks_rcu_start(void) { }
> +static inline void exit_tasks_rcu_stop(void) { }
>  static inline void exit_tasks_rcu_finish(void) { }
>  #endif /* #else #ifdef CONFIG_TASKS_RCU_GENERIC */
>  
> diff --git a/kernel/pid_namespace.c b/kernel/pid_namespace.c
> index f4f8cb0435b4..fc21c5d5fd5d 100644
> --- a/kernel/pid_namespace.c
> +++ b/kernel/pid_namespace.c
> @@ -244,7 +244,24 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns)
>  		set_current_state(TASK_INTERRUPTIBLE);
>  		if (pid_ns->pid_allocated == init_pids)
>  			break;
> +		/*
> +		 * Release tasks_rcu_exit_srcu to avoid following deadlock:
> +		 *
> +		 * 1) TASK A unshare(CLONE_NEWPID)
> +		 * 2) TASK A fork() twice -> TASK B (child reaper for new ns)
> +		 *    and TASK C
> +		 * 3) TASK B exits, kills TASK C, waits for TASK A to reap it
> +		 * 4) TASK A calls synchronize_rcu_tasks()
> +		 *                   -> synchronize_srcu(tasks_rcu_exit_srcu)
> +		 * 5) *DEADLOCK*
> +		 *
> +		 * It is considered safe to release tasks_rcu_exit_srcu here
> +		 * because we assume the current task can not be concurrently
> +		 * reaped at this point.
> +		 */
> +		exit_tasks_rcu_stop();
>  		schedule();
> +		exit_tasks_rcu_start();
>  	}
>  	__set_current_state(TASK_RUNNING);
>  
> diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
> index 9a8114114b48..4dda8e6e5707 100644
> --- a/kernel/rcu/tasks.h
> +++ b/kernel/rcu/tasks.h
> @@ -1016,16 +1016,27 @@ void exit_tasks_rcu_start(void) __acquires(&tasks_rcu_exit_srcu)
>   * task is exiting and may be removed from the tasklist. See
>   * corresponding synchronize_srcu() for further details.
>   */
> -void exit_tasks_rcu_finish(void) __releases(&tasks_rcu_exit_srcu)
> +void exit_tasks_rcu_stop(void) __releases(&tasks_rcu_exit_srcu)
>  {
>  	struct task_struct *t = current;
>  
>  	__srcu_read_unlock(&tasks_rcu_exit_srcu, t->rcu_tasks_idx);
> -	exit_tasks_rcu_finish_trace(t);
> +}
> +
> +/*
> + * Contribute to protect against tasklist scan blind spot while the
> + * task is exiting and may be removed from the tasklist. See
> + * corresponding synchronize_srcu() for further details.
> + */
> +void exit_tasks_rcu_finish(void)
> +{
> +	exit_tasks_rcu_stop();
> +	exit_tasks_rcu_finish_trace(current);
>  }
>  
>  #else /* #ifdef CONFIG_TASKS_RCU */
>  void exit_tasks_rcu_start(void) { }
> +void exit_tasks_rcu_stop(void) { }
>  void exit_tasks_rcu_finish(void) { exit_tasks_rcu_finish_trace(current); }
>  #endif /* #else #ifdef CONFIG_TASKS_RCU */
>  
> -- 
> 2.25.1
> 

Powered by blists - more mailing lists