linux-kernel - Re: Linux scheduler capabilities for batch jobs.

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:	Wed, 17 Jun 2009 11:38:51 +0200
From:	Pavel Machek <pavel@....cz>
To:	J Louis <handstogether8@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Linux scheduler capabilities for batch jobs.

On Mon 2009-06-01 09:41:07, J Louis wrote:
> Hello All,
> 
> I have been writing user land server code and increasingly I find
> myself writing resource management code which I think would be better
> handled by the scheduler.  The problem can be described as "don't run
> this process if the machine is swapping."  I would think that this
> would be a common enough need that it was already in the kernel.  I'm
> hoping it is and I have simply overlooked it (part of the reason for
> this post,) but I've looked around a good bit and most of the
> scheduler enhancements have to do with real time and latency, not
> batch jobs.
> 
> My problem is analogous to a parallel make.  Say I have an 8 CPU
> machine, and I run "make -j8".  If the total memory of the 8 jobs
> throws the machine into swap, it begins to thrash and runtime is
> awful.  I believe this is aggravated by the scheduler trying to be

This seems to be racy by design.

make -j:
     launch gcc1
     gcc1 does preprocessing (10mb)  
     ok, still not swapping
     launch gcc2
     gcc2 does preprocessing (10mb)  
     ...
     gcc100 does preprocessing (10mb, 1GB total)
now gcc1..gcc100 start optimizing (100mb needed) and boom.

									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/