Message-ID: <1183082664.11002.1602082242482.JavaMail.zimbra@efficios.com>
Date:   Wed, 7 Oct 2020 10:50:42 -0400 (EDT)
From:   Mathieu Desnoyers <mathieu.desnoyers@...icios.com>
To:     Rong Chen <rong.a.chen@...el.com>,
        Anton Blanchard <anton@...ibm.com>,
        Peter Zijlstra <peterz@...radead.org>
Cc:     Boqun Feng <boqun.feng@...il.com>,
        linux-kernel <linux-kernel@...r.kernel.org>,
        Will Deacon <will@...nel.org>, paulmck <paulmck@...nel.org>,
        Nicholas Piggin <npiggin@...il.com>,
        Andy Lutomirski <luto@...capital.net>,
        Andrew Morton <akpm@...ux-foundation.org>,
        0day robot <lkp@...el.com>, lkp@...ts.01.org,
        Huang Ying <ying.huang@...el.com>,
        feng tang <feng.tang@...el.com>,
        zhengjun xing <zhengjun.xing@...el.com>,
        aubrey li <aubrey.li@...ux.intel.com>,
        yu c chen <yu.c.chen@...el.com>
Subject: Re: [sched] bdfcae1140: will-it-scale.per_thread_ops -37.0%
 regression

----- On Oct 2, 2020, at 4:33 AM, Rong Chen rong.a.chen@...el.com wrote:

> Greeting,
> 
> FYI, we noticed a -37.0% regression of will-it-scale.per_thread_ops due to
> commit:
> 
> 
> commit: bdfcae11403e5099769a7c8dc3262e3c4193edef ("[RFC PATCH 2/3] sched:
> membarrier: cover kthread_use_mm (v3)")
> url:
> https://github.com/0day-ci/linux/commits/Mathieu-Desnoyers/Membarrier-updates/20200925-012549
> base: https://git.kernel.org/cgit/linux/kernel/git/tip/tip.git
> 848785df48835eefebe0c4eb5da7690690b0a8b7
> 
> in testcase: will-it-scale
> on test machine: 104 threads Skylake with 192G memory
> with following parameters:
> 
>	nr_task: 50%
>	mode: thread
>	test: context_switch1
>	cpufreq_governor: performance
>	ucode: 0x2006906
> 
> test-description: Will It Scale takes a testcase and runs it from 1 through to n
> parallel copies to see if the testcase will scale. It builds both a process and
> threads based test in order to see any differences between the two.
> test-url: https://github.com/antonblanchard/will-it-scale
>

Hi,

I would like to report what I suspect is a random thread placement issue in the
context_switch1 test used by the 0day bot when running on a machine with hyperthread
enabled.

AFAIU the test code uses hwloc for thread placement which should theoretically ensure
that each thread is placed on same processing unit, core and numa node between runs.

We can find the test code here:

https://github.com/antonblanchard/will-it-scale/blob/master/tests/context_switch1.c

And the main file containing thread setup is here:

https://github.com/antonblanchard/will-it-scale/blob/master/main.c

AFAIU, the test is started without the "-m" switch, which therefore affinitizes
tasks on cores rather than on processing units (SMT threads).

When testcase() creates the child thread with new_task(), it basically issues:

  pthread_create(&threads[nr_threads++], NULL, func, arg);

passing a NULL pthread_attr_t, and not executing any pre_trampoline on the child.
The pre_trampoline would have issued hwloc_set_thread_cpubind if it were executed on
the child, but it's not. Therefore, we expect the cpu affinity mask of the parent to
be copied on clone and used by the child.

A quick test on a machine with hyperthreading enabled shows that the cpu affinity mask
for the parent and child has two bits set:

taskset -p 1868607
pid 1868607's current affinity mask: 10001
taskset -p 1868606
pid 1868606's current affinity mask: 10001

So AFAIU the placement of the parent and child will be random on either the same
processing unit, or on separate processing units within the same core.

I suspect this randomness can significantly affect the performance number between
runs, and trigger unwarranted performance regression warnings.

Thanks,

Mathieu

-- 
Mathieu Desnoyers
EfficiOS Inc.
http://www.efficios.com

Powered by blists - more mailing lists