linux-kernel - Re: Slowdown due to threads bouncing between HT cores

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20141004141244.GB27832@sesse.net>
Date:	Sat, 4 Oct 2014 16:12:44 +0200
From:	"Steinar H. Gunderson" <sgunderson@...foot.com>
To:	Andi Kleen <andi@...stfloor.org>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Slowdown due to threads bouncing between HT cores

On Sat, Oct 04, 2014 at 06:41:15AM -0700, Andi Kleen wrote:
> - something else gets scheduled on these logical CPUs, so
> the scheduler tries to balance to run queue lengths
> 
> You could check that with perf timechart or perf sched record/map
> or kernelshark.

I've never read any of these maps before, but perf sched map really doesn't
indicate to me that there's a lot of other stuff going on. It seems to mainly
show a lot of Stockfish processes bouncing around seemingly randomly with not
much understanding of hyperthread pairs. Of course, there's the odd other
job, including ksoftirq or an RCU process.

I can send you a copy of the map if you want to, but it is of course rather
large.

> - there is some IO or communication which causes wakeup affinity.

There's a fair amount of communication between the threads; I don't know the
architecture very deeply (multithreading in chess is rather nontrivial),
but as far as I know, the worker threads access shared data through shm,
sometimes using pthread mutexes to lock some of it.

This also means, by the way, that occasionally they will sleep. They're not
by default going to hog the CPU 100% of the time, more like 90%.

> You could try disabling WAKEUP_PREEMPTION or NEXT_BUDDY in 
> /sys/kernel/debug/sched_features

NO_NEXT_BUDDY was already set. (Changing it to NEXT_BUDDY didn't seem to help
anything.) I tried setting NO_WAKEUP_PREEMPTION, and it didn't make a
difference that I could see; they still bounce around a lot.

/* Steinar */
-- 
Homepage: http://www.sesse.net/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/