linux-kernel - Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the debug code

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 28 Jan 2016 21:48:49 -0800
From:	Peter Hurley <peter@...leysoftware.com>
To:	Sergey Senozhatsky <sergey.senozhatsky.work@...il.com>
Cc:	Sergey Senozhatsky <sergey.senozhatsky@...il.com>,
	Byungchul Park <byungchul.park@....com>,
	akpm@...ux-foundation.org, mingo@...nel.org,
	linux-kernel@...r.kernel.org, akinobu.mita@...il.com, jack@...e.cz,
	torvalds@...ux-foundation.org
Subject: Re: [PATCH v4] lib/spinlock_debug.c: prevent a recursive cycle in the
 debug code

On 01/28/2016 09:28 PM, Sergey Senozhatsky wrote:
> On (01/28/16 20:32), Peter Hurley wrote:
> [..]
>> You're assuming that Byungchul's patch is relevant to the recursion
>> he witnessed. There are several paths into spin_dump().
> 
> yes. I was speaking in the context of Byungchul's report.
> 
>> Here's one that doesn't wait at all:
>>
>> vprintk_emit()
>>   console_trylock()
>>     down_trylock()
>>       raw_spin_lock_irqsave()
>>         ...             
>>           do_raw_spin_lock()
>>             debug_spin_lock_before()
>>               SPIN_BUG_ON()
>>                 spin_bug()
>>                    spin_dump()
>>                      printk()
>>                        ** RINSE AND REPEAT **
> 
> ah, yes, agree.
> 
>>>> Additionally, what if the console_sem is simply corrupted?
>>>> A livelock with no output ever is not very helpful.
>>>
>>> if it's corrupted then this is not a spinlock debug problem.
>>> at all.
>>
>> I don't follow you.
>>
> 
> indeed very misleading, sorry, almost didn't sleep last nigh.
> removing SPIN_BUG_ON entirely is not my logic, not all. printk locks are
> special, I agree. in context of the proposed patch - we can't disable
> spin_dump() for printk locks if they were corrupted. for printk locks it's
> over, nothing will be printed anymore. the only thing that _may be_ will
> help is zap_locks(), but not 100% guaranteed... we can panic the system,
> probably, if printk locks are getting corrupted, but panic() will not do the
> trick in general case, at this point -- console_unlock() takes the logbuf_lock,
> which can be corrupted. apart from that console driver can be in a weird state.
> 
> I sort of proposed to update console_flush_on_panic()  (called from panic())
> function a while ago to do zap_locks(), so in this case declaring BUG() from
> spinlock debug when we see 'bad' printk-related locks will have better
> chances to work out (assuming that console driver(-s) is (are) not against
> us).

Yeah, exactly, something that improves the chances of successful output.


> [..]
>> This was in reference to a problem with spin lock recursion that
>> can't print. The spin lock recursion deadlocks, but you'll never
>> see the diagnostic because the driver is already holding the lock
>> (not from printk() but from some other code).
>>
>> The printk doesn't even need to be directly related to the
>> console driver itself, but some other thing that the console driver
>> depends on while holding the spin lock that it claims for console output.
> 
> aha, ok. yes, this is certainly possible.
> 
>>> this is not a case of printk recursion and it should be handled
>>> just fine. console drivers are called under console_sem only.
>>> logbuf lock is unlocked. vprintk_emit() adds message to the logbuf,
>>> calls console_trylock() (which of course does not lock anything)
>>> and returns back to console_driver code.
>>
>> Not if locks are zapped because printk() recognizes a recursion.
>> Note console driver's locks are not zapped. For example,
> 
> yes, I proposed to add a ->reset callback to struct console
> a while ago, and to do a console reset loop in zap_locks()

What was the patch series title? I'd like to review that.

That would solve the recursive deadlock from console driver as well
(at least with CONFIG_DEBUG_SPINLOCK) because the printk() recursion
would zap the locks including the console driver's lock and
at least get the last output so that we'd know there was a recursion,
and fix it.


> zap_locks:
> ...
> 	for_each_console(con)
> 		if (con->reset)
> 			con->reset(con)
> 
> that would re-init console drivers (locks, etc.).
> 
> 
> IOW, panic() does zap_locks(), zap_locks() zap the locks and
> resets the console drivers. that's sort of what I have in my
> private kernels.
> 
> [..]
>>> the only case when we really have a printk recursion is when
>>> someone calls printk() from within the vprintk_emit() logbuf_lock
>>> area.
>>
>> Not true.
>>
>> A while back, Jan Kara reworked the call site around
>> console_trylock_from_printk(). Hung with no output under unknown
>> conditions [1].
>>
>> Never solved, but obviously means there are unhandled recursions.

I'd still like to enable lockdep for console drivers, but I need a
better plan to debug unknown printk() recursions.


> aha, ok.
> 
> 	-ss
>