linux-kernel - Re: Quad core CPUs loaded at only 50% when running a CPU and mmap intensive multi-threaded task

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:	Mon, 25 Aug 2008 14:00:55 +0300
From:	edwin <edwintorok@...il.com>
To:	Peter Zijlstra <peterz@...radead.org>
CC:	Ingo Molnar <mingo@...e.hu>, rml@...h9.net,
	Linux Kernel <linux-kernel@...r.kernel.org>,
	"Thomas Gleixner mingo@...hat.com" <tglx@...utronix.de>,
	"H. Peter Anvin" <hpa@...or.com>
Subject: Re: Quad core CPUs loaded at only 50% when running a CPU and mmap
 intensive multi-threaded task

Peter Zijlstra wrote:
> On Mon, 2008-08-25 at 13:22 +0300, Török Edwin wrote:
>
>   
>> Well, the real program (clamd) that this testprogram tries to simulate 
>> does an mmap for almost every file, and I have lots of small files.
>> 6.5G, 114122 files, average size 57k.
>>
>> I'll run latencytop again, last time it has showed 100ms - 500ms latency 
>>     

Latencytop output attached.
There is 4 - 60 ms latency for mmap/munmap, and the more threads there 
are the total latency gets higher (latencytop says sum was ~480ms).

Running with MaxThreads 4 gets me 300-400% CPU usage, but with 
MaxThreads 8 CPU usage drops to around 120-250%.
Now, maxthreads 4 looks like a good choice from a CPU usage point of 
view, but is actually bad because it means that threads gets stuck in 
iowait, and the CPU won't have anything to do. MaxThreads 8 looked like 
a good alternative to fill the iowait gaps, but we run into the mmap_sem 
issue.
In a real world environment MaxThreads influences how many mails you can 
process in parallel with your MTA, so generally it should be as high as 
possible.

On 2.6.27-rc4:

MaxThreads 4 time, empty database (all cached, almost no I/O):
1m9s

MaxThreads 4 time, after echo 3>/proc/sys/vm/drop_caches:
1m29s

MaxThreads 8 time, empty database (all cached, almost no I/O):
2m16s

MaxThreads 8 time, after echo 3>/proc/sys/vm/drop_caches:
2m15s

Of course running with a full database will give different results, so 
I'll do some timing with that too (will take a little longer though).

>> for clamd, and it was about mmap, I'll provide you with the exact output.
>>     
>
> Right - does it make sense to teach clamav about pread() ?

If it is preferred over mmap, then maybe yes.

Peter Zijlstra wrote:
> OK, I'll poke a little more at is later today to see if I can spot
> something

Thanks!

Best regards,
--Edwin

View attachment "latgather-1219660865" of type "text/plain" (27285 bytes)

View attachment "latgather-1219660915" of type "text/plain" (52120 bytes)