Date:	Wed, 11 Apr 2007 17:28:39 +1000
From:	Nick Piggin <nickpiggin@...oo.com.au>
To:	Jeff Garzik <jeff@...zik.org>
CC:	Linus Torvalds <torvalds@...ux-foundation.org>,
	Robin Holt <holt@....com>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	Ingo Molnar <mingo@...e.hu>, linux-kernel@...r.kernel.org,
	Jack Steiner <steiner@...ricas.sgi.com>
Subject: Re: init's children list is long and slows reaping children.

Jeff Garzik wrote:
> Linus Torvalds wrote:
> 
>>
>> On Fri, 6 Apr 2007, Jeff Garzik wrote:
>>
>>> I would rather change the implementation under the hood to start per-CPU
>>> threads on demand, similar to a thread-pool implementation.
>>>
>>> Boxes with $BigNum CPUs probably won't ever use half of those threads.
>>
>>
>> The counter-argument is that boxes with $BigNum CPU's really don't 
>> hurt from it either, and having per-process data structures is often 
>> simpler and more efficient than trying to have some thread pool.
> 
> 
> Two points here:
> 
> * A lot of the users in the current kernel tree don't rely on the 
> per-CPU qualities.  They just need multiple threads running.
> 
> * Even with per-CPU data structures and code, you don't necessarily have 
> to keep a thread alive and running for each CPU.  Reap the ones that 
> haven't been used in $TimeFrame, and add thread creation to the slow 
> path that already exists in the bowels of schedule_work().
> 
> Or if some kernel hacker is really motivated, all workqueue users in the 
> kernel would benefit from a "thread audit", looking at working 
> conditions to decide if the new kthread APIs are more appropriate.

spawn on demand would require heuristics and complexity though. And
I think there is barely any positive tradeoff to weigh it against.

>> IOW, once we get the processes off the global list, there just isn't 
>> any downside from them. Sure, they use some memory, but people who buy 
>> 1024-cpu machines won't care about a few kB per CPU..
>>
>> So the *only* downside is literally the process list, and one 
>> suggested patch already just removes kernel threads entirely from the 
>> parenthood lists.
>>
>> The other potential downside could be "ps is slow", but on the other 
>> hand, having the things stick around and have things like CPU-time 
>> accumulate is probably worth it - if there are some issues, they'd 
>> show up properly accounted for in a way that process pools would have 
>> a hard time doing.
> 
> 
> Regardless of how things are shuffled about internally, there will 
> always be annoying overhead /somewhere/ when you have a metric ton of 
> kernel threads.  I think that people should also be working on ways to 
> make the kernel threads a bit more manageable for the average human.

There are a few per CPU, but they should need no human management to
speak of.

Presumably if you have a 1024 CPU system, you'd generally want to be
running at least 1024 of your own processes there, so you already need
some tools to handle that magnitude of processes anyway.

>> So I really don't think this is worth changing things over, apart from 
>> literally removing them from process lists, which I think everybody 
>> agrees we should just do - it just never even came up before!
> 
> 
> I think there is a human downside.  For an admin you have to wade 
> through a ton of processes on your machine, if you are attempting to 
> evaluate the overall state of the machine.  Just google around for all 
> the admins complaining about the explosion of kernel threads on 
> production machines :)

User tools should be improved. It shouldn't be too hard to be able to
aggregate kernel thread stats into a single top entry, for example.

I'm not saying the number of threads couldn't be cut down, but there
is still be an order of magnitude problem there...

-- 
SUSE Labs, Novell Inc.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists