[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20160129052838.GD4820@swordfish>
Date: Fri, 29 Jan 2016 14:28:38 +0900
From: Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
To: Peter Hurley <peter@...leysoftware.com>
Cc: Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>,
Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
Byungchul Park <byungchul.park@....com>,
akpm@...ux-foundation.org, mingo@...nel.org,
linux-kernel@...r.kernel.org, akinobu.mita@...il.com, jack@...e.cz,
torvalds@...ux-foundation.org
Subject: Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in
the debug code
On (01/28/16 20:32), Peter Hurley wrote:
[..]
> You're assuming that Byungchul's patch is relevant to the recursion
> he witnessed. There are several paths into spin_dump().
yes. I was speaking in the context of Byungchul's report.
> Here's one that doesn't wait at all:
>
> vprintk_emit()
> console_trylock()
> down_trylock()
> raw_spin_lock_irqsave()
> ...
> do_raw_spin_lock()
> debug_spin_lock_before()
> SPIN_BUG_ON()
> spin_bug()
> spin_dump()
> printk()
> ** RINSE AND REPEAT **
ah, yes, agree.
> >> Additionally, what if the console_sem is simply corrupted?
> >> A livelock with no output ever is not very helpful.
> >
> > if it's corrupted then this is not a spinlock debug problem.
> > at all.
>
> I don't follow you.
>
indeed very misleading, sorry, almost didn't sleep last nigh.
removing SPIN_BUG_ON entirely is not my logic, not all. printk locks are
special, I agree. in context of the proposed patch - we can't disable
spin_dump() for printk locks if they were corrupted. for printk locks it's
over, nothing will be printed anymore. the only thing that _may be_ will
help is zap_locks(), but not 100% guaranteed... we can panic the system,
probably, if printk locks are getting corrupted, but panic() will not do the
trick in general case, at this point -- console_unlock() takes the logbuf_lock,
which can be corrupted. apart from that console driver can be in a weird state.
I sort of proposed to update console_flush_on_panic() (called from panic())
function a while ago to do zap_locks(), so in this case declaring BUG() from
spinlock debug when we see 'bad' printk-related locks will have better
chances to work out (assuming that console driver(-s) is (are) not against
us).
[..]
> This was in reference to a problem with spin lock recursion that
> can't print. The spin lock recursion deadlocks, but you'll never
> see the diagnostic because the driver is already holding the lock
> (not from printk() but from some other code).
>
> The printk doesn't even need to be directly related to the
> console driver itself, but some other thing that the console driver
> depends on while holding the spin lock that it claims for console output.
aha, ok. yes, this is certainly possible.
> > this is not a case of printk recursion and it should be handled
> > just fine. console drivers are called under console_sem only.
> > logbuf lock is unlocked. vprintk_emit() adds message to the logbuf,
> > calls console_trylock() (which of course does not lock anything)
> > and returns back to console_driver code.
>
> Not if locks are zapped because printk() recognizes a recursion.
> Note console driver's locks are not zapped. For example,
yes, I proposed to add a ->reset callback to struct console
a while ago, and to do a console reset loop in zap_locks()
zap_locks:
...
for_each_console(con)
if (con->reset)
con->reset(con)
that would re-init console drivers (locks, etc.).
IOW, panic() does zap_locks(), zap_locks() zap the locks and
resets the console drivers. that's sort of what I have in my
private kernels.
[..]
> > the only case when we really have a printk recursion is when
> > someone calls printk() from within the vprintk_emit() logbuf_lock
> > area.
>
> Not true.
>
> A while back, Jan Kara reworked the call site around
> console_trylock_from_printk(). Hung with no output under unknown
> conditions [1].
>
> Never solved, but obviously means there are unhandled recursions.
aha, ok.
-ss
Powered by blists - more mailing lists