linux-kernel - Re: [feature] automatically detect hung TASK

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20071203121357.GB2986@one.firstfloor.org>
Date:	Mon, 3 Dec 2007 13:13:57 +0100
From:	Andi Kleen <andi@...stfloor.org>
To:	Ingo Molnar <mingo@...e.hu>
Cc:	Andi Kleen <andi@...stfloor.org>,
	Radoslaw Szkodzinski <lkml@...ralstorm.puszkin.org>,
	Arjan van de Ven <arjan@...radead.org>,
	linux-kernel@...r.kernel.org,
	Andrew Morton <akpm@...ux-foundation.org>,
	Thomas Gleixner <tglx@...utronix.de>
Subject: Re: [feature] automatically detect hung TASK_UNINTERRUPTIBLE tasks

On Mon, Dec 03, 2007 at 12:59:00PM +0100, Ingo Molnar wrote:
> no. (that's why i added the '(or a kill -9)' qualification above - if 
> NFS is mounted noninterruptible then standard signals (such as Ctrl-C) 
> should not have an interrupting effect.)

NFS is already interruptible with umount -f (I use that all the time...), 
but softlockup won't know that and throw the warning anyways.

> your syslet snide comment aside (which is quite incomprehensible - a 

For the record I have no principle problem with syslets, just I do
consider them roughly equivalent in end result to a explicit retry based
AIO implementation.

> retry based asynchonous IO model is clearly inferior even if it were 
> implemented everywhere), i do think that most if not all of these 
> supposedly "difficult to fix" codepaths are just on the backburner out 
> of lack of a clear blame vector.

Hmm. -ENOPARSE. Can you please clarify?

> 
> "audit thousands of callsites in 8 million lines of code first" is a 
> nice euphemism for hiding from the blame forever. We had 10 years for it 

Ok your approach is then to "let's warn about it and hope
it will go away" 

> and it didnt happen. As we've seen it again and again, getting a 
> non-fatal reminder in the dmesg about the suckage is quite efficient at 

It's not universal suckage I would say, but sometimes unavoidable
conditions. Now it is better of course to have these all TASK_KILLABLE,
but then fixing that all in the kernel will probably a long term
project. I'm not arguing against that, just forcing it through
backtraces before even starting all that is probably not the right
strategy to do that.

> getting people to fix crappy solutions, and gives users and exact blame 
> point of where to start. That will create pressure to fix these 
> problems.

After impacting the user base -- many of these conditions are infrequent
enough that we will likely only see them during real production. Throwing
warnings for lots of known cases is probably ok for a -mm kernel
(where users expect things lik that), but not a "release" (be it
Linus release or any kind of end user distribution) imho.

I don't think there is a real alternative to code audit first
(and someone doing all the work of fixing all these first) 

> 
> > > I think you are somehow confusing two issues: this patch in no way 
> > > declares that "long waits are bad" - if the user _choses_ to wait 
> > > for
> > 
> > Throwing a backtrace is the kernel's way to declare something as bad. 
> > The only more clear ways to that I know of would be BUG or panic().
> 
> there are various levels of declarig something bad, and you are quite 
> wrong to suggest that a BUG() would be the only recourse.

I didn't write that, please reread my sentence.. 

But we seem to agree that a backtrace is something "declared bad" anyways,
which was my point.

> 
> > > way to stop_ are quite likely bad".
> > 
> > The user will just see the backtraces and think the kernel has 
> > crashed.
> 
> i've just changed the message to:
> 
>   INFO: task keventd/5 blocked for more than 120 seconds.
>   "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message

That's better, but the backtrace is still there isn't it?

Anyways I think I could live with it a one liner warning (if it's
seriously rate limited etc.) and a sysctl to enable the backtraces;
off by default. Or if you prefer that record 
the backtrace always in a buffer and make it available somewhere in /proc
or /sys or /debug. Would that work for you? 

-Andi
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/