linux-kernel - 2.6.25/26/27 scheduler regression on NUMA configurations

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <20080828142854.GA13512@localdomain>
Date:	Thu, 28 Aug 2008 07:28:54 -0700
From:	Ravikiran G Thirumalai <kiran@...lex86.org>
To:	linux-kernel@...r.kernel.org
Cc:	Ingo Molnar <mingo@...e.hu>
Subject: 2.6.25/26/27 scheduler regression on NUMA configurations

Hi,
We are noticing a significant regression in performance with some test workloads on
x86_64 NUMA machines.  The workload is  multiple  datatest processes
running in parallel.  Each process is basically performing  writes and
reads to disk backed files.  More information about datatest can be found at:
http://home.comcast.net/~SCSIguy/SCSI_FAQ/RMiller_Tools/dt.html

To give an idea about the regression, on a plain vanilla 2.6.21 kernel,  a test suite
running a bunch of datatest processes on different files takes about 900 seconds to
complete, and with  2.6.25/26/27-rc3 the exact same workload takes 1200 seconds to complete.
Two times slower!  This is on a 24 cpu 6 node NUMA box.

Looking at schedstats, I see that there are many more remote wakeups due to
passive load balancing with 27 than in 21, which seems to be causing this
performance difference (ttwu_wake_remote). In fact, there are twice as many
ttwu_wake_remote events in the NODE domain with 27 than on 21.  If I turn off wake
balancing on the NUMA domain, the same workload runs in 300-400s!  Obviously turning off wake
balancing is not the right  solution, but, the increased
remote wakeups due to 'passive' wake balancing is definitely a regression!

I am attaching the full schedstat snapshots with 2.6.21 and 2.6.27 vanilla, before starting
the workload and at the end of the workload.  This is on a 6 node, 24 core, 12 socket
woodcrest machine.

I was wondering if this regression has been noted elsewhere.

Thanks,
Kiran

View attachment "vanilla-21-schedstat-before" of type "text/plain" (10908 bytes)

View attachment "vanilla-21-schedstat-after" of type "text/plain" (12985 bytes)

View attachment "vanilla-27-rc3-schedstat-before" of type "text/plain" (11819 bytes)

View attachment "vanilla-27-rc3-schedstat-after" of type "text/plain" (13913 bytes)