Date:	Mon, 21 Mar 2016 09:00:12 +0100 (CET)
From:	Thomas Gleixner <tglx@...utronix.de>
To:	"Huang, Ying" <ying.huang@...el.com>
cc:	Ingo Molnar <mingo@...nel.org>,
	Peter Zijlstra <a.p.zijlstra@...llo.nl>,
	Chris Mason <clm@...com>,
	Peter Zijlstra <peterz@...radead.org>, lkp@...org,
	Sebastian Andrzej Siewior <bigeasy@...utronix.de>,
	Davidlohr Bueso <dbueso@...e.de>,
	Hugh Dickins <hughd@...gle.com>,
	LKML <linux-kernel@...r.kernel.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Mel Gorman <mgorman@...e.de>,
	Mel Gorman <mgorman@...hsingularity.net>,
	Darren Hart <dvhart@...ux.intel.com>,
	Wu Fengguang <fengguang.wu@...el.com>,
	Xiaolong Ye <xiaolong.ye@...el.com>
Subject: Re: [LKP] [lkp] [futex] 65d8fc777f: +25.6%
 will-it-scale.per_process_ops

On Mon, 21 Mar 2016, Huang, Ying wrote:
> >  FYI, we noticed 25.6% performance improvement due to commit
> >
> >    65d8fc777f6d "futex: Remove requirement for lock_page() in get_futex_key()"
> >
> >  in the will-it-scale.per_process_ops test.
> >
> >  will-it-scale.per_process_ops tests the futex operations for process shared
> >  futexes (Or whatever that test really does).
> 
> There is a futex sub test case for will-it-scale test suite.  But I got your
> point, we need some description for the test case.  If email is not too
> limited for the full description, we will put it in some web site and
> include short description and link to the full description in email.

Ok. Just make sure the short description gives enough information for the
casual reader.
 
> >  The commit has no significant impact on any other test in the test suite.
> 
> Sorry, we have no enough machine power to test all test cases for each
> bisect result.  So we will have no such information until we find a way
> to do that.

Well, then I really have to ask how I should interpret the data here:

   5076304 ±  0%     +25.6%    6374220 ±  0%  will-it-scale.per_process_ops

   ^^^ That's the reason why you sent the mail in the first place

   1194117 ±  0%     +14.4%    1366153 ±  1%  will-it-scale.per_thread_ops
      0.58 ±  0%      -2.0%       0.57 ±  0%  will-it-scale.scalability
      6820 ±  0%     -19.6%       5483 ± 15%  meminfo.AnonHugePages
      2652 ±  5%     -10.4%       2375 ±  2%  vmstat.system.cs
      2848 ± 32%    +141.2%       6870 ± 65%  numa-meminfo.node1.Active(anon)
      2832 ± 31%     +57.6%       4465 ± 27%  numa-meminfo.node1.AnonPages
     15018 ± 12%     -23.3%      11515 ± 15%  numa-meminfo.node2.AnonPages
      1214 ± 14%     -22.8%     936.75 ± 20%  numa-meminfo.node3.PageTables
    712.25 ± 32%    +141.2%       1718 ± 65%  numa-vmstat.node1.nr_active_anon
    708.25 ± 31%     +57.7%       1116 ± 27%  numa-vmstat.node1.nr_anon_pages

How is this related and what should I do about this information? 
 
 If it's important then I have to admit, that I fail to understand why.

 If it's not important then I have to ask why is this included.

> > So that allows me to reproduce that test more or less with no effort. And
> > that's the really important part.
> 
> For reproducing, now we use lkp-tests tool, which includes scripts to
> build the test case, run the test, collect various information, compare
> the test result, with the job file attached with the report email.  That
> is not the easiest way, we will continuously improve it.

I know and lkp-tests is a pain to work with. So please look into a way to
extract the relevant binaries, so it's simple for developers to reproduce.
 
> > You can provide nice charts and full comparison tables for all tests on a web
> > site for those who are interested in large stats and pretty charts.
> >
> > Full results: http://wherever.you.store/your/results/test-nr/results
> 
> Before we have a website for detailed information, we will still put
> some details into report email.

Ok, but please make them understandable for mere mortals.
 
Thanks,

	tglx

Powered by blists - more mailing lists