Message-ID: <20151203011129.GA510@swordfish>
Date:	Thu, 3 Dec 2015 10:11:29 +0900
From:	Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
To:	tj@...nel.org
Cc:	akpm@...ux-foundation.org, calvinowens@...com,
	davej@...emonkey.org.uk, jack@...e.com, kyle@...nel.org,
	stable@...r.kernel.org, mm-commits@...r.kernel.org,
	linux-kernel@...r.kernel.org
Subject: Re: +
 printk-do-cond_resched-between-lines-while-outputting-to-consoles.patch
 added to -mm tree

Hello,

On (12/02/15 15:57), akpm@...ux-foundation.org wrote:
[..]
> @console_may_schedule tracks whether console_sem was acquired through lock
> or trylock.  If the former, we're inside a sleepable context and
> console_conditional_schedule() performs cond_resched().  This allows
> console drivers which use console_lock for synchronization to yield while
> performing time-consuming operations such as scrolling.
> 
> However, the actual console outputting is performed while holding irq-safe
> logbuf_lock, so console_unlock() clears @console_may_schedule before
> starting outputting lines.  Also, only a few drivers call
> console_conditional_schedule() to begin with.  This means that when a lot
> of lines need to be output by console_unlock(), for example on a console
> registration, the task doing console_unlock() may not yield for a long
> time on a non-preemptible kernel.
> 
> If this happens with a slow console devices, for example a serial console,
> the outputting task may occupy the cpu for a very long time.  Long enough
> to trigger softlockup and/or RCU stall warnings, which in turn pile more
> messages, sometimes enough to trigger the next cycle of warnings
> incapacitating the system.
> 
> Fix it by making console_unlock() insert cond_resched() between lines if
> @console_may_schedule.

CPU2 still can cause lots of troubles. consider

CPU0		CPU1			CPU2
printk		
...		printk_deferred		
printk					wake_up_klogd
						wake_up_klogd_work_func
							console_trylock
								console_unlock

printk_deferred() may be issued by scheduler, for example.

	-ss

> Signed-off-by: Tejun Heo <tj@...nel.org>
> Reported-by: Calvin Owens <calvinowens@...com>
> Acked-by: Jan Kara <jack@...e.com>
> Cc: Dave Jones <davej@...emonkey.org.uk>
> Cc: Kyle McMartin <kyle@...nel.org>
> Cc: <stable@...r.kernel.org>
> Signed-off-by: Andrew Morton <akpm@...ux-foundation.org>
> ---
> 
>  kernel/printk/printk.c |   16 +++++++++++++++-
>  1 file changed, 15 insertions(+), 1 deletion(-)
> 
> diff -puN kernel/printk/printk.c~printk-do-cond_resched-between-lines-while-outputting-to-consoles kernel/printk/printk.c
> --- a/kernel/printk/printk.c~printk-do-cond_resched-between-lines-while-outputting-to-consoles
> +++ a/kernel/printk/printk.c
> @@ -2234,13 +2234,24 @@ void console_unlock(void)
>  	static u64 seen_seq;
>  	unsigned long flags;
>  	bool wake_klogd = false;
> -	bool retry;
> +	bool do_cond_resched, retry;
>  
>  	if (console_suspended) {
>  		up_console_sem();
>  		return;
>  	}
>  
> +	/*
> +	 * Console drivers are called under logbuf_lock, so
> +	 * @console_may_schedule should be cleared before; however, we may
> +	 * end up dumping a lot of lines, for example, if called from
> +	 * console registration path, and should invoke cond_resched()
> +	 * between lines if allowable.  Not doing so can cause a very long
> +	 * scheduling stall on a slow console leading to RCU stall and
> +	 * softlockup warnings which exacerbate the issue with more
> +	 * messages practically incapacitating the system.
> +	 */
> +	do_cond_resched = console_may_schedule;
>  	console_may_schedule = 0;
>  
>  	/* flush buffered message fragment immediately to console */
> @@ -2312,6 +2323,9 @@ skip:
>  		call_console_drivers(level, ext_text, ext_len, text, len);
>  		start_critical_timings();
>  		local_irq_restore(flags);
> +
> +		if (do_cond_resched)
> +			cond_resched();
>  	}
>  	console_locked = 0;
>  
> _
> 
> Patches currently in -mm which might be from tj@...nel.org are
> 
> printk-do-cond_resched-between-lines-while-outputting-to-consoles.patch
> 
> --
> To unsubscribe from this list: send the line "unsubscribe mm-commits" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists