Message-Id: <20170306124554.828-1-sergey.senozhatsky@gmail.com>
Date:   Mon,  6 Mar 2017 21:45:50 +0900
From:   Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
To:     Steven Rostedt <rostedt@...dmis.org>,
        Petr Mladek <pmladek@...e.com>
Cc:     Andrew Morton <akpm@...ux-foundation.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>,
        Peter Zijlstra <peterz@...radead.org>,
        "Rafael J . Wysocki" <rjw@...ysocki.net>,
        linux-kernel@...r.kernel.org,
        Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
        Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
Subject: [RFC][PATCH 0/4] printk: introduce printing kernel thread

Hello,

	RFC

	This patch set adds a printk() kernel thread which lets us to
print kernel messages to the console from a non-atomic/schedule-able
context, avoiding different sort of lockups, stalls, etc.


The patch set is in [RFC] stage - it may contain ugly and dumb ideas.
The reason I decided to take a step back is because we've been using
printk kthread in development/production for ~10 months now and there
are some observations from my side, which I want to list/outline.

 1) We had a `bool' flag to temporarily disable printk offloading (in suspend,
 hibernate, etc.) to address a bug reported by Viresh Kumar [1]. Turned out
 that our in-house kernel tweaks/ad hocs in PM code introduce nested sections
 that disable/enable printk offloading and thus `bool' flag can re-enable
 printk-kthread too early. (it wasn't me who added those PM 'tweaks' :)).

 Would there be any objections if we switch to `int' flag and allow nested
 printk-kthread enable/disable sections?


 2) Just letting you know,
 printk kthread changes the behavior of printk in one _corner case_.
 The corner case is quite interesting and actually consists of two corner
 cases. Suppose on SMP system there is only one CPU that printk()-s a lot,
 the rest of CPUs don't lock console_sem and don't printk(). Previously
 that printing CPU had been throttling itself (*) because of console drivers
 call for every printk():

          CPU0

       printk("a")
        console_unlock()
         call_console_drivers("a")

         ...

       printk("z")
        console_unlock()
         call_console_drivers("z")

 * Given that no other CPU locks the console_sem.

 With printk kthread the case turns into this one:

          CPU0                               CPU1

       printk("a")
       wake_up printk_kthread
       ...                               printk_kthread
       printk("k")                        console_unlock()
       ...                                 call_console_drivers("a")
       printk("z")                         call_console_drivers("b")
                                           call_console_drivers("c")
                                           ...


 The second 'corner case' part here is that CPU0 may be much faster
 than printing CPU, which may result in dropped printk messages.

 This all is absolutely possible even with out the printk-kthread.
 A single console_lock() call from CPUx will result in exactly the
 same condition. So it's not necessarily a regression. But there may
 be some scenarios in the kernel that may suffer from this change.
 From the top of my head -- sysrq backtrace dump, and, probably, OOM
 print out and backtrace dump.

 sysrq is potentially even trickier. can we always wake_up() kernel
 thread from sysrq? there probably might be cases when we can't rely
 on the scheduler.

 So may be we would want to export disable/enable printk-kthread offloading
 symbols and let people use them in appropriate places.

 What do you think?


 3) A minor one,
 'async printk' is, apparently, a confusing name. people assume that the
 opposite to 'asynchronous printk' is 'synchronous printk'. which is not
 the case. printk may accidentally be synchronous but in general there is
 no way to make it always synchronous on SMP system for all CPUs (well,
 except for wrapping every printk() in console_lock()/console_unlock(),
 probably), because it's up to the CPU that holds the console_sem to print
 kernel messages from console_unlock() at some point. so in some sense it's
 asynchronous by design. printk-kthread targets different aims anyway.
 but still I would probably prefer to avoid this confusion.



[1] this is from the days when printk-kthread had RT prio
    lkml.kernel.org/r/20160714221251.GE3057@...ntu



= Patches 0001/0002 just introduce the printk kthread, it's not eanbled yet.

= Patch 0003 is incomplete and a bit ugly. I need to think more. It exposes
some printk internals and in general I'm not really happy with it. PM parts,
probably, can be kept within the printk.c. For example, disable printk-kthread
in suspend_console() and enable it in resume_console(). unlike sysrq though.

= Patch 0004 enables printk-kthread.


Sergey Senozhatsky (4):
  printk: introduce printing kernel thread
  printk: offload printing from wake_up_klogd_work_func()
  kernel, power: disable printk_kthread in unsafe places
  printk: enable printk offloading

 include/linux/console.h  |   3 ++
 kernel/kexec_core.c      |   4 ++
 kernel/power/hibernate.c |   8 +++
 kernel/power/suspend.c   |   4 ++
 kernel/printk/printk.c   | 133 ++++++++++++++++++++++++++++++++++++++++++++---
 5 files changed, 144 insertions(+), 8 deletions(-)

-- 
2.12.0

Powered by blists - more mailing lists