linux-kernel - Re: apache threads not balanced over all CPU cores, schedular problem or apache?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Thu, 13 Nov 2008 22:48:25 -0500
From:	Andy Lutomirski <luto@...ealbox.com>
To:	kyle <kylewong@...tha.com>
CC:	linux-kernel@...r.kernel.org
Subject: Re: apache threads not balanced over all CPU cores, schedular problem
 or apache?

kyle wrote:
> Hi,
> 
> Please cc me if possible.
> 
> I have a machine with Intel Q9300 4-core CPU, 8GB RAM. It runs apache 2.0.63
> with traditional prefork MPM.
> It's a high traffic site, most of the time there's over 100 httpd processes
> running and at peak hours around 200.
> 
> I found that most of the time, all httpd processes are running at 2 CPU
> cores only, sometime it's cpu0 and cpu1, sometime it's cpu1 and cpu3,
> sometime it's cpu0 and cpu2 ... etc. It may take several hours the processes
> "migrate" from 2 cores to another 2. To make it clear:
> 
> top - 12:44:42 up 8 days, 15:45,  7 users,  load average: 2.07, 1.86, 2.02
> Tasks: 251 total,   1 running, 249 sleeping,   0 stopped,   1 zombie
> Cpu0  :  0.5%us,  0.5%sy,  0.0%ni, 99.0%id,  0.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Cpu1  : 14.6%us,  4.4%sy,  0.0%ni, 62.6%id, 17.0%wa,  0.0%hi,  1.5%si,
> 0.0%st
> Cpu2  :  4.3%us,  1.4%sy,  0.0%ni, 93.8%id,  0.0%wa,  0.5%hi,  0.0%si,
> 0.0%st
> Cpu3  :  0.0%us,  0.0%sy,  0.0%ni, 98.0%id,  2.0%wa,  0.0%hi,  0.0%si,
> 0.0%st
> Mem:   8117172k total,  8060936k used,    56236k free,   357488k buffers
> Swap:   208824k total,     8436k used,   200388k free,  5765168k cached

Looks like you're mostly idle or waiting for I/O.  From this 
information, it looks like the kernel is probably doing the right thing: 
keeping a lot of processes that share the same memory and run for very 
short periods of time on the same cores to minimize cacheline bouncing.

> 
> All httpd are running on cpu1 and cpu2 only for several hours, after that,
> all httpd may run on cpu1 and cpu3, or cpu2 and cpu3, etc. again for several
> hours.
> 
> Since the apache is in prefork mode, every httpd instance should be quite
> independant, I suppose the CPU schedular should balance the processes over
> the avaiable cores. However never seen httpd processes evenly distributed to
> 3 cores, or 4 cores.

Is there any reason to expect it to run faster if it were balanced evenly?

--Andy
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/