lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20190823065606.GA22771@shao2-debian>
Date:   Fri, 23 Aug 2019 14:56:06 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Christoph Hellwig <hch@....de>
Cc:     "Darrick J. Wong" <darrick.wong@...cle.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [xfs] 79b54d9bfc: aim7.jobs-per-min -2.8% regression

Greeting,

FYI, we noticed a -2.8% regression of aim7.jobs-per-min due to commit:


commit: 79b54d9bfcdcd0a431bb3501013885f8f00cdb43 ("xfs: use bios directly to write log buffers")
https://kernel.googlesource.com/pub/scm/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID1
	fs: xfs
	test: sync_disk_rw
	load: 300
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/xfs/x86_64-rhel-7.6/300/RAID1/debian-x86_64-2019-05-14.cgz/lkp-ivb-ep01/sync_disk_rw/aim7

commit: 
  2d15d2c0e0 ("xfs: make use of the l_targ field in struct xlog")
  79b54d9bfc ("xfs: use bios directly to write log buffers")

2d15d2c0e0f7ecd6 79b54d9bfcdcd0a431bb3501013 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          3:4          -75%            :4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      3210            -2.8%       3120        aim7.jobs-per-min
    560.76            +2.9%     576.86        aim7.time.elapsed_time
    560.76            +2.9%     576.86        aim7.time.elapsed_time.max
   4185708           -36.5%    2656308        aim7.time.involuntary_context_switches
     44779           +13.8%      50970 ±  4%  aim7.time.minor_page_faults
      8008           +21.7%       9746        aim7.time.system_time
  54284549            +1.5%   55104076        aim7.time.voluntary_context_switches
   6546944 ±  8%     -20.1%    5228544 ±  8%  meminfo.DirectMap2M
      1203 ± 32%     +57.4%       1893        meminfo.Mlocked
    656.50 ± 39%     +55.1%       1018 ± 12%  numa-meminfo.node0.Mlocked
    545.00 ± 28%     +60.5%     874.75 ± 14%  numa-meminfo.node1.Mlocked
    163.75 ± 39%     +55.3%     254.25 ± 12%  numa-vmstat.node0.nr_mlock
    135.75 ± 28%     +60.8%     218.25 ± 13%  numa-vmstat.node1.nr_mlock
      2880 ±  3%      +8.9%       3135 ±  2%  slabinfo.Acpi-Namespace.active_objs
      2880 ±  3%      +8.9%       3135 ±  2%  slabinfo.Acpi-Namespace.num_objs
      7.87            +1.3        9.17 ±  8%  mpstat.cpu.all.idle%
     52.24            -7.8       44.43        mpstat.cpu.all.iowait%
     39.47            +6.5       45.98        mpstat.cpu.all.sys%
     51.50           -15.5%      43.50        vmstat.cpu.wa
    121621            -3.3%     117644        vmstat.io.bo
    152.00 ±  2%     -32.9%     102.00 ±  2%  vmstat.procs.b
    166223           -12.8%     144884        vmstat.system.cs
      8.17           +15.8%       9.46 ±  8%  iostat.cpu.idle
     52.06           -14.9%      44.28        iostat.cpu.iowait
     39.35           +16.5%      45.84        iostat.cpu.system
     56478            -3.2%      54670        iostat.md0.w/s
    121621            -3.3%     117591        iostat.md0.wkB/s
 7.705e+09           -33.2%  5.143e+09 ± 15%  cpuidle.C1.time
  37185864           -38.0%   23064840 ± 15%  cpuidle.C1.usage
 3.847e+09 ±  2%     +38.3%  5.319e+09 ± 11%  cpuidle.C1E.time
  13506915           +25.9%   16999001 ±  4%  cpuidle.C1E.usage
  6.22e+08 ± 16%     +42.8%  8.885e+08 ± 17%  cpuidle.C6.time
   1502731 ± 30%     +49.0%    2238626 ±  5%  cpuidle.C6.usage
     14436            -2.4%      14096        proc-vmstat.nr_kernel_stack
    300.00 ± 33%     +57.8%     473.25        proc-vmstat.nr_mlock
      8177            +2.1%       8350        proc-vmstat.nr_zone_write_pending
     28750           +21.4%      34895 ±  5%  proc-vmstat.numa_hint_faults
     19128           +18.0%      22570 ±  8%  proc-vmstat.numa_hint_faults_local
      5515 ±  8%     +40.7%       7758 ± 17%  proc-vmstat.numa_pages_migrated
     31200 ±  6%     +49.3%      46570 ± 31%  proc-vmstat.numa_pte_updates
   5829999            -3.9%    5599986        proc-vmstat.pgalloc_normal
   1453272            +3.7%    1506462        proc-vmstat.pgfault
   5598375            -4.3%    5355258        proc-vmstat.pgfree
      5515 ±  8%     +40.7%       7758 ± 17%  proc-vmstat.pgmigrate_success
    519.25           +13.1%     587.25        turbostat.Avg_MHz
     43.24            +5.6       48.87        turbostat.Busy%
  37178210           -38.0%   23057179 ± 15%  turbostat.C1
     34.26           -12.1       22.11 ± 15%  turbostat.C1%
  13506740           +25.9%   16998768 ±  4%  turbostat.C1E
     17.10            +5.8       22.91 ± 12%  turbostat.C1E%
   1498986 ± 31%     +49.1%    2234353 ±  5%  turbostat.C6
      2.76 ± 16%      +1.1        3.82 ± 18%  turbostat.C6%
     55.25           -11.1%      49.13        turbostat.CPU%c1
      0.94 ±  7%     +51.6%       1.42 ± 26%  turbostat.CPU%c6
     57.75 ±  2%      +3.9%      60.00 ±  3%  turbostat.PkgTmp
     15.66 ±  2%      -6.2%      14.69 ±  3%  perf-stat.i.MPKI
 1.312e+09            +7.3%  1.408e+09        perf-stat.i.branch-instructions
  85566446            -2.1%   83741795        perf-stat.i.cache-references
    166943           -12.9%     145469        perf-stat.i.context-switches
      3.62            +6.5%       3.85        perf-stat.i.cpi
 2.037e+10           +13.2%  2.306e+10        perf-stat.i.cpu-cycles
     14929           +13.3%      16911        perf-stat.i.cpu-migrations
      1151           +15.9%       1334        perf-stat.i.cycles-between-cache-misses
 1.516e+09            +6.1%  1.609e+09        perf-stat.i.dTLB-loads
 5.031e+08            -4.5%  4.807e+08        perf-stat.i.dTLB-stores
    446380            -7.5%     413106 ±  2%  perf-stat.i.iTLB-loads
 5.724e+09            +6.5%  6.098e+09        perf-stat.i.instructions
      1814            +8.6%       1970 ±  2%  perf-stat.i.instructions-per-iTLB-miss
      0.28            -5.6%       0.26        perf-stat.i.ipc
   6126990            -4.1%    5875615        perf-stat.i.node-load-misses
   7052743            -3.9%    6776675        perf-stat.i.node-loads
     14.95            -8.1%      13.73        perf-stat.overall.MPKI
      3.56            +6.2%       3.78        perf-stat.overall.cpi
      1151           +15.6%       1331        perf-stat.overall.cycles-between-cache-misses
      1812            +7.8%       1953 ±  2%  perf-stat.overall.instructions-per-iTLB-miss
      0.28            -5.9%       0.26        perf-stat.overall.ipc
  1.31e+09            +7.3%  1.405e+09        perf-stat.ps.branch-instructions
  85412963            -2.1%   83596915        perf-stat.ps.cache-references
    166644           -12.9%     145217        perf-stat.ps.context-switches
 2.033e+10           +13.2%  2.302e+10        perf-stat.ps.cpu-cycles
     14902           +13.3%      16882        perf-stat.ps.cpu-migrations
 1.513e+09            +6.1%  1.606e+09        perf-stat.ps.dTLB-loads
 5.022e+08            -4.5%  4.798e+08        perf-stat.ps.dTLB-stores
    445580            -7.4%     412389 ±  2%  perf-stat.ps.iTLB-loads
 5.714e+09            +6.6%  6.088e+09        perf-stat.ps.instructions
   6116017            -4.1%    5865466        perf-stat.ps.node-load-misses
   7040111            -3.9%    6764971        perf-stat.ps.node-loads
 3.208e+12           +10.1%   3.53e+12        perf-stat.total.instructions
    106009           +16.2%     123235        sched_debug.cfs_rq:/.exec_clock.avg
    107819           +17.6%     126771        sched_debug.cfs_rq:/.exec_clock.max
    105443           +16.4%     122706        sched_debug.cfs_rq:/.exec_clock.min
    406.56 ± 30%     +51.5%     616.03 ±  8%  sched_debug.cfs_rq:/.exec_clock.stddev
    667.46 ±  4%     +32.9%     887.07 ±  3%  sched_debug.cfs_rq:/.load_avg.avg
      5478 ± 12%     +30.0%       7120 ±  4%  sched_debug.cfs_rq:/.load_avg.max
      1231 ± 11%     +30.1%       1603 ±  4%  sched_debug.cfs_rq:/.load_avg.stddev
   2240608           +24.6%    2791499        sched_debug.cfs_rq:/.min_vruntime.avg
   2276805           +24.4%    2831917        sched_debug.cfs_rq:/.min_vruntime.max
   2211580           +25.0%    2763664        sched_debug.cfs_rq:/.min_vruntime.min
      0.35 ±  4%     +20.3%       0.42 ±  7%  sched_debug.cfs_rq:/.nr_running.avg
      0.65 ± 12%     +23.3%       0.80 ±  7%  sched_debug.cfs_rq:/.nr_spread_over.avg
      0.02 ± 82%     -88.9%       0.00 ±173%  sched_debug.cfs_rq:/.spread.avg
      0.65 ± 82%     -88.9%       0.07 ±173%  sched_debug.cfs_rq:/.spread.max
      0.10 ± 82%     -88.9%       0.01 ±173%  sched_debug.cfs_rq:/.spread.stddev
    453.33           +13.1%     512.68        sched_debug.cfs_rq:/.util_avg.avg
     54.96 ± 12%     +28.7%      70.71 ± 10%  sched_debug.cfs_rq:/.util_est_enqueued.avg
    133762 ± 27%     +67.3%     223845 ±  7%  sched_debug.cpu.avg_idle.min
    176517 ±  5%     -27.7%     127615 ±  6%  sched_debug.cpu.avg_idle.stddev
     45.82 ± 20%     +48.7%      68.15 ± 17%  sched_debug.cpu.cpu_load[1].avg
      0.08 ± 57%    +766.7%       0.65 ± 41%  sched_debug.cpu.cpu_load[1].min
    180.41 ± 21%     +42.2%     256.56 ± 19%  sched_debug.cpu.cpu_load[1].stddev
     50.85 ± 17%     +47.1%      74.78 ± 11%  sched_debug.cpu.cpu_load[2].avg
      1064 ± 15%     +40.8%       1499 ± 13%  sched_debug.cpu.cpu_load[2].max
      0.95 ± 36%    +286.8%       3.67 ± 12%  sched_debug.cpu.cpu_load[2].min
    182.31 ± 16%     +43.2%     261.03 ± 14%  sched_debug.cpu.cpu_load[2].stddev
     54.75 ± 14%     +37.4%      75.24 ±  8%  sched_debug.cpu.cpu_load[3].avg
      2.70 ±  2%    +147.2%       6.67        sched_debug.cpu.cpu_load[3].min
    172.59 ± 15%     +32.8%     229.19 ± 13%  sched_debug.cpu.cpu_load[3].stddev
     55.56 ± 11%     +34.3%      74.62 ±  5%  sched_debug.cpu.cpu_load[4].avg
    818.43 ± 15%     +25.5%       1027 ±  9%  sched_debug.cpu.cpu_load[4].max
      3.68 ± 16%    +149.0%       9.15 ±  4%  sched_debug.cpu.cpu_load[4].min
    149.46 ± 14%     +27.7%     190.91 ±  7%  sched_debug.cpu.cpu_load[4].stddev
    786.26 ±  6%     +22.5%     963.52 ±  6%  sched_debug.cpu.curr->pid.avg
      0.58 ±  9%     +36.8%       0.80 ± 10%  sched_debug.cpu.nr_running.avg
      1.07 ±  6%     +17.7%       1.26 ±  8%  sched_debug.cpu.nr_running.stddev
   1109136           -12.7%     968119        sched_debug.cpu.nr_switches.avg
   1136593           -12.5%     995035        sched_debug.cpu.nr_switches.max
   1084891           -13.2%     941961        sched_debug.cpu.nr_switches.min
   1108298           -12.7%     966993        sched_debug.cpu.sched_count.avg
   1134353           -12.7%     990115        sched_debug.cpu.sched_count.max
   1084183           -13.1%     941989        sched_debug.cpu.sched_count.min
    363169           -26.0%     268783        sched_debug.cpu.sched_goidle.avg
    370264           -25.8%     274814        sched_debug.cpu.sched_goidle.max
    356698           -26.4%     262531        sched_debug.cpu.sched_goidle.min
     54790           -31.1%      37767        sched_debug.cpu.ttwu_local.avg
     58890           -30.2%      41086        sched_debug.cpu.ttwu_local.max
     50991           -32.5%      34406        sched_debug.cpu.ttwu_local.min
      2893 ±  3%     -23.4%       2216 ±  7%  sched_debug.cpu.ttwu_local.stddev
    614.75 ±  5%     -33.4%     409.25 ± 24%  interrupts.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
    202670 ±  2%      +4.0%     210754 ±  2%  interrupts.CAL:Function_call_interrupts
      1180 ± 66%     -78.4%     255.25 ±167%  interrupts.CPU0.NMI:Non-maskable_interrupts
      1180 ± 66%     -78.4%     255.25 ±167%  interrupts.CPU0.PMI:Performance_monitoring_interrupts
    280939           +33.1%     373920        interrupts.CPU0.RES:Rescheduling_interrupts
    276566           +29.8%     358849        interrupts.CPU1.RES:Rescheduling_interrupts
      4780 ±  6%     +17.2%       5603 ±  3%  interrupts.CPU10.CAL:Function_call_interrupts
    280074           +33.1%     372754        interrupts.CPU10.RES:Rescheduling_interrupts
    275727           +29.6%     357377        interrupts.CPU11.RES:Rescheduling_interrupts
    279558           +33.0%     371728        interrupts.CPU12.RES:Rescheduling_interrupts
      4951 ±  5%     +15.8%       5733 ±  5%  interrupts.CPU13.CAL:Function_call_interrupts
    228.75 ±172%    +540.0%       1464 ± 32%  interrupts.CPU13.NMI:Non-maskable_interrupts
    228.75 ±172%    +540.0%       1464 ± 32%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
    274475           +29.7%     356099        interrupts.CPU13.RES:Rescheduling_interrupts
    279541           +33.3%     372530        interrupts.CPU14.RES:Rescheduling_interrupts
    274190           +30.1%     356589        interrupts.CPU15.RES:Rescheduling_interrupts
    161.00 ±  6%     +18.8%     191.25 ±  2%  interrupts.CPU15.TLB:TLB_shootdowns
    280230           +32.8%     372098        interrupts.CPU16.RES:Rescheduling_interrupts
    274564           +29.9%     356548        interrupts.CPU17.RES:Rescheduling_interrupts
      4726 ±  4%     +15.0%       5433 ±  7%  interrupts.CPU18.CAL:Function_call_interrupts
    281353           +32.5%     372837        interrupts.CPU18.RES:Rescheduling_interrupts
    275881           +29.8%     358162        interrupts.CPU19.RES:Rescheduling_interrupts
    279855           +33.4%     373407        interrupts.CPU2.RES:Rescheduling_interrupts
    279868           +33.2%     372868        interrupts.CPU20.RES:Rescheduling_interrupts
    275194           +29.9%     357364        interrupts.CPU21.RES:Rescheduling_interrupts
    279996           +33.1%     372750        interrupts.CPU22.RES:Rescheduling_interrupts
      4813 ±  3%     +11.0%       5343 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
    274267           +30.1%     356824        interrupts.CPU23.RES:Rescheduling_interrupts
    147.25 ±  6%     +23.1%     181.25 ± 12%  interrupts.CPU23.TLB:TLB_shootdowns
    280417           +32.8%     372408        interrupts.CPU24.RES:Rescheduling_interrupts
    274998           +29.7%     356596        interrupts.CPU25.RES:Rescheduling_interrupts
    279837           +33.1%     372572        interrupts.CPU26.RES:Rescheduling_interrupts
    274653           +29.9%     356815        interrupts.CPU27.RES:Rescheduling_interrupts
    280411           +33.5%     374295        interrupts.CPU28.RES:Rescheduling_interrupts
      1346 ± 32%     -81.5%     249.25 ±172%  interrupts.CPU29.NMI:Non-maskable_interrupts
      1346 ± 32%     -81.5%     249.25 ±172%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
    275415           +30.1%     358211        interrupts.CPU29.RES:Rescheduling_interrupts
    275415           +29.8%     357414        interrupts.CPU3.RES:Rescheduling_interrupts
    280591           +33.4%     374339        interrupts.CPU30.RES:Rescheduling_interrupts
    275173           +30.1%     357888        interrupts.CPU31.RES:Rescheduling_interrupts
    614.75 ±  5%     -33.4%     409.25 ± 24%  interrupts.CPU32.40:IR-PCI-MSI.524290-edge.eth0-TxRx-1
    279946           +33.2%     373008        interrupts.CPU32.RES:Rescheduling_interrupts
    183.50 ±  6%     -12.9%     159.75 ±  8%  interrupts.CPU32.TLB:TLB_shootdowns
    274555           +30.1%     357231        interrupts.CPU33.RES:Rescheduling_interrupts
    279591           +33.4%     372975        interrupts.CPU34.RES:Rescheduling_interrupts
    274826           +29.7%     356522        interrupts.CPU35.RES:Rescheduling_interrupts
    279964           +33.1%     372728        interrupts.CPU36.RES:Rescheduling_interrupts
    274751           +30.0%     357304        interrupts.CPU37.RES:Rescheduling_interrupts
    280687           +33.4%     374361        interrupts.CPU38.RES:Rescheduling_interrupts
    275349           +29.9%     357769        interrupts.CPU39.RES:Rescheduling_interrupts
    279879           +33.3%     372946        interrupts.CPU4.RES:Rescheduling_interrupts
    275474           +29.2%     356034        interrupts.CPU5.RES:Rescheduling_interrupts
    279870           +33.4%     373364        interrupts.CPU6.RES:Rescheduling_interrupts
    275249           +29.9%     357679        interrupts.CPU7.RES:Rescheduling_interrupts
    280669           +33.2%     373808        interrupts.CPU8.RES:Rescheduling_interrupts
      4951 ±  8%     +12.1%       5552 ±  7%  interrupts.CPU9.CAL:Function_call_interrupts
    232.25 ±173%    +423.8%       1216 ± 65%  interrupts.CPU9.NMI:Non-maskable_interrupts
    232.25 ±173%    +423.8%       1216 ± 65%  interrupts.CPU9.PMI:Performance_monitoring_interrupts
    275277           +30.0%     357736        interrupts.CPU9.RES:Rescheduling_interrupts
  11105289           +31.5%   14606722        interrupts.RES:Rescheduling_interrupts
     19.07            -9.6        9.44        perf-profile.calltrace.cycles-pp.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     17.30            -9.2        8.05        perf-profile.calltrace.cycles-pp.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     17.26            -9.2        8.02        perf-profile.calltrace.cycles-pp.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
     17.24            -9.2        8.01        perf-profile.calltrace.cycles-pp.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range.xfs_file_fsync
     34.15            -9.2       24.92        perf-profile.calltrace.cycles-pp.raid1_make_request.md_handle_request.md_make_request.generic_make_request.submit_bio
     16.31            -9.1        7.18        perf-profile.calltrace.cycles-pp.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range.file_write_and_wait_range
     16.18            -9.1        7.05        perf-profile.calltrace.cycles-pp.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages
     16.29            -9.1        7.16        perf-profile.calltrace.cycles-pp.generic_make_request.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages
     16.29            -9.1        7.17        perf-profile.calltrace.cycles-pp.submit_bio.xfs_submit_ioend.xfs_vm_writepages.do_writepages.__filemap_fdatawrite_range
     16.09            -9.1        6.98        perf-profile.calltrace.cycles-pp.md_handle_request.md_make_request.generic_make_request.submit_bio.xfs_submit_ioend
     15.94            -9.1        6.85        perf-profile.calltrace.cycles-pp.raid1_write_request.raid1_make_request.md_handle_request.md_make_request.generic_make_request
      7.56            -7.6        0.00        perf-profile.calltrace.cycles-pp.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
      7.40            -7.4        0.00        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request.md_handle_request
      7.38            -7.4        0.00        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.raid1_write_request.raid1_make_request
     22.34            -7.1       15.21 ±  3%  perf-profile.calltrace.cycles-pp.secondary_startup_64
     21.41 ±  3%      -6.4       15.06 ±  4%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
     21.41 ±  3%      -6.4       15.06 ±  4%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
     21.39 ±  3%      -6.4       15.04 ±  4%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     17.70 ±  3%      -5.5       12.23 ±  3%  perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
     17.55 ±  3%      -5.4       12.13 ±  4%  perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     15.22            -5.2       10.05        perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     17.28            -3.0       14.30        perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     17.62            -3.0       14.67        perf-profile.calltrace.cycles-pp.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     14.99            -2.5       12.45 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync
     14.95            -2.5       12.41 ±  2%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn
     15.04            -2.5       12.51 ±  2%  perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      3.66 ±  3%      -0.6        3.07 ±  6%  perf-profile.calltrace.cycles-pp.raid1d.md_thread.kthread.ret_from_fork
      3.65 ±  3%      -0.6        3.07 ±  6%  perf-profile.calltrace.cycles-pp.flush_pending_writes.raid1d.md_thread.kthread.ret_from_fork
      3.65 ±  3%      -0.6        3.07 ±  6%  perf-profile.calltrace.cycles-pp.flush_bio_list.flush_pending_writes.raid1d.md_thread.kthread
      3.66 ±  3%      -0.6        3.08 ±  6%  perf-profile.calltrace.cycles-pp.md_thread.kthread.ret_from_fork
      6.76 ±  4%      -0.6        6.20 ±  2%  perf-profile.calltrace.cycles-pp.ret_from_fork
      6.76 ±  4%      -0.6        6.20 ±  2%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      3.52 ±  3%      -0.6        2.97 ±  6%  perf-profile.calltrace.cycles-pp.generic_make_request.flush_bio_list.flush_pending_writes.raid1d.md_thread
      3.35 ±  3%      -0.5        2.81 ±  6%  perf-profile.calltrace.cycles-pp.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes.raid1d
      1.85 ±  5%      -0.5        1.34 ±  8%  perf-profile.calltrace.cycles-pp.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list.flush_pending_writes
      1.51 ±  6%      -0.5        1.03 ± 10%  perf-profile.calltrace.cycles-pp.raid_end_bio_io.raid1_end_write_request.brd_make_request.generic_make_request.flush_bio_list
      5.27 ±  2%      -0.5        4.79 ±  2%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request.md_make_request
      5.11 ±  2%      -0.4        4.66        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.raid1_write_request.raid1_make_request.md_handle_request
      1.69 ±  5%      -0.4        1.28 ± 10%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      1.68 ±  6%      -0.4        1.31 ±  4%  perf-profile.calltrace.cycles-pp.__filemap_fdatawait_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      1.55 ±  6%      -0.4        1.19 ±  9%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      1.54 ±  7%      -0.3        1.20 ±  4%  perf-profile.calltrace.cycles-pp.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range.xfs_file_fsync.xfs_file_buffered_aio_write
      1.11 ±  6%      -0.3        0.79 ±  8%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.42 ±  8%      -0.3        1.11 ±  5%  perf-profile.calltrace.cycles-pp.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range.xfs_file_fsync
      1.40 ±  8%      -0.3        1.10 ±  5%  perf-profile.calltrace.cycles-pp.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range.file_write_and_wait_range
      1.39 ±  8%      -0.3        1.09 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.io_schedule.wait_on_page_bit.__filemap_fdatawait_range
      1.28 ±  8%      -0.3        1.02 ±  4%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      1.12 ± 11%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      1.12 ± 11%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      1.12 ± 11%      -0.2        0.88 ±  4%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.94 ±  6%      -0.2        0.69 ± 14%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
      1.07 ± 11%      -0.2        0.83 ±  4%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.64 ±  2%      -0.2        0.41 ± 58%  perf-profile.calltrace.cycles-pp.try_to_wake_up.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.md_submit_flush_data
      0.77 ± 12%      -0.2        0.56 ±  7%  perf-profile.calltrace.cycles-pp.__account_scheduler_latency.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate
      0.71 ±  5%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.69 ±  5%      -0.1        0.55 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
      0.68 ±  5%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.schedule.md_flush_request.raid1_make_request.md_handle_request.md_make_request
      0.67 ±  5%      -0.1        0.55 ±  5%  perf-profile.calltrace.cycles-pp.__schedule.schedule.md_flush_request.raid1_make_request.md_handle_request
      0.68 ±  2%      -0.1        0.56 ±  5%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.md_submit_flush_data.process_one_work.worker_thread
      0.67 ±  2%      -0.1        0.55 ±  6%  perf-profile.calltrace.cycles-pp.autoremove_wake_function.__wake_up_common.__wake_up_common_lock.md_submit_flush_data.process_one_work
      0.82 ±  2%      -0.1        0.71 ±  6%  perf-profile.calltrace.cycles-pp.md_submit_flush_data.process_one_work.worker_thread.kthread.ret_from_fork
      0.82 ±  3%      -0.1        0.71 ±  6%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.md_submit_flush_data.process_one_work.worker_thread.kthread
      0.00            +0.5        0.54 ±  4%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_state_done_syncing.xlog_ioend_work.process_one_work.worker_thread
      0.00            +0.6        0.60 ±  9%  perf-profile.calltrace.cycles-pp.pick_next_task_fair.__schedule.schedule.__xfs_log_force_lsn.xfs_file_fsync
      0.00            +0.7        0.73 ±  3%  perf-profile.calltrace.cycles-pp.try_to_wake_up.__wake_up_common.__wake_up_common_lock.xlog_state_do_callback.xlog_ioend_work
      0.00            +0.8        0.76 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_common_lock.xlog_state_do_callback.xlog_ioend_work.process_one_work
      0.00            +0.8        0.78 ±  6%  perf-profile.calltrace.cycles-pp.xlog_state_done_syncing.xlog_ioend_work.process_one_work.worker_thread.kthread
      0.00            +0.8        0.78 ±  2%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.xlog_state_do_callback.xlog_ioend_work.process_one_work.worker_thread
      0.00            +1.0        0.97        perf-profile.calltrace.cycles-pp.xlog_state_do_callback.xlog_ioend_work.process_one_work.worker_thread.kthread
      0.00            +1.0        0.99 ±  6%  perf-profile.calltrace.cycles-pp.__schedule.schedule.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      0.00            +1.0        1.00 ±  6%  perf-profile.calltrace.cycles-pp.schedule.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      0.00            +1.7        1.75 ±  2%  perf-profile.calltrace.cycles-pp.xlog_ioend_work.process_one_work.worker_thread.kthread.ret_from_fork
      3.68 ±  8%      +6.7       10.38 ±  4%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync
      3.69 ±  8%      +6.7       10.40 ±  4%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      3.71 ±  8%      +6.7       10.44 ±  4%  perf-profile.calltrace.cycles-pp.remove_wait_queue.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
     69.70            +7.7       77.37        perf-profile.calltrace.cycles-pp.write
     69.14            +7.7       76.85        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     69.12            +7.7       76.83        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     68.92            +7.7       76.64        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     68.89            +7.7       76.61        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     68.67            +7.7       76.41        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     68.54            +7.7       76.28        perf-profile.calltrace.cycles-pp.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     65.52            +8.0       73.49        perf-profile.calltrace.cycles-pp.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write.ksys_write
      5.67 ±  3%     +13.1       18.75 ±  3%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write
      5.69 ±  3%     +13.1       18.80 ±  3%  perf-profile.calltrace.cycles-pp._raw_spin_lock.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write
      9.94           +20.8       30.78 ±  3%  perf-profile.calltrace.cycles-pp.__xfs_log_force_lsn.xfs_file_fsync.xfs_file_buffered_aio_write.new_sync_write.vfs_write
     38.37           -10.0       28.40        perf-profile.children.cycles-pp.generic_make_request
     19.07            -9.6        9.45        perf-profile.children.cycles-pp.file_write_and_wait_range
     34.80            -9.4       25.44        perf-profile.children.cycles-pp.submit_bio
     34.55            -9.3       25.20        perf-profile.children.cycles-pp.md_make_request
     34.42            -9.3       25.09        perf-profile.children.cycles-pp.md_handle_request
     34.31            -9.3       25.00        perf-profile.children.cycles-pp.raid1_make_request
     17.30            -9.2        8.05        perf-profile.children.cycles-pp.__filemap_fdatawrite_range
     17.26            -9.2        8.03        perf-profile.children.cycles-pp.do_writepages
     17.24            -9.2        8.01        perf-profile.children.cycles-pp.xfs_vm_writepages
     16.01            -9.1        6.88        perf-profile.children.cycles-pp.raid1_write_request
     16.31            -9.1        7.18        perf-profile.children.cycles-pp.xfs_submit_ioend
      8.13            -7.7        0.45 ±  8%  perf-profile.children.cycles-pp.prepare_to_wait_event
     22.35            -7.1       15.21 ±  3%  perf-profile.children.cycles-pp.do_idle
     22.34            -7.1       15.21 ±  3%  perf-profile.children.cycles-pp.secondary_startup_64
     22.34            -7.1       15.21 ±  3%  perf-profile.children.cycles-pp.cpu_startup_entry
     21.41 ±  3%      -6.4       15.06 ±  4%  perf-profile.children.cycles-pp.start_secondary
     18.46            -6.1       12.34 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter
     18.45            -6.1       12.34 ±  3%  perf-profile.children.cycles-pp.cpuidle_enter_state
     15.35            -5.2       10.15        perf-profile.children.cycles-pp.intel_idle
     33.52            -4.4       29.15        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
     17.62            -3.0       14.67        perf-profile.children.cycles-pp.xfs_log_force_lsn
      3.72 ±  3%      -0.7        3.07 ±  6%  perf-profile.children.cycles-pp.flush_bio_list
      3.41 ±  3%      -0.6        2.81 ±  6%  perf-profile.children.cycles-pp.brd_make_request
      3.66 ±  3%      -0.6        3.07 ±  6%  perf-profile.children.cycles-pp.raid1d
      1.79 ±  5%      -0.6        1.20 ±  4%  perf-profile.children.cycles-pp.autoremove_wake_function
      3.65 ±  3%      -0.6        3.07 ±  6%  perf-profile.children.cycles-pp.flush_pending_writes
      3.66 ±  3%      -0.6        3.08 ±  6%  perf-profile.children.cycles-pp.md_thread
      6.76 ±  4%      -0.6        6.20 ±  2%  perf-profile.children.cycles-pp.kthread
      6.76 ±  4%      -0.6        6.21 ±  2%  perf-profile.children.cycles-pp.ret_from_fork
      1.86 ±  5%      -0.5        1.34 ±  8%  perf-profile.children.cycles-pp.raid1_end_write_request
      1.52 ±  6%      -0.5        1.03 ± 10%  perf-profile.children.cycles-pp.raid_end_bio_io
      2.65 ±  3%      -0.4        2.23 ±  5%  perf-profile.children.cycles-pp.__wake_up_common_lock
      2.83            -0.4        2.43 ±  2%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.67 ±  5%      -0.4        0.28 ± 11%  perf-profile.children.cycles-pp.finish_wait
      4.44 ±  3%      -0.4        4.04 ±  2%  perf-profile.children.cycles-pp.__schedule
      1.68 ±  6%      -0.4        1.32 ±  3%  perf-profile.children.cycles-pp.__filemap_fdatawait_range
      1.17 ±  7%      -0.4        0.81 ±  8%  perf-profile.children.cycles-pp.menu_select
      1.54 ±  7%      -0.3        1.20 ±  4%  perf-profile.children.cycles-pp.wait_on_page_bit
      2.57 ±  2%      -0.3        2.24 ±  3%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      1.42 ±  8%      -0.3        1.11 ±  5%  perf-profile.children.cycles-pp.io_schedule
      2.91 ±  3%      -0.3        2.64 ±  2%  perf-profile.children.cycles-pp.try_to_wake_up
      1.37 ± 10%      -0.2        1.13 ±  3%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      2.87 ±  3%      -0.2        2.63 ±  2%  perf-profile.children.cycles-pp.__wake_up_common
      1.70            -0.2        1.52 ±  6%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.74 ±  3%      -0.2        0.57 ±  5%  perf-profile.children.cycles-pp.schedule_idle
      0.55 ±  7%      -0.2        0.39 ± 10%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      1.28 ±  4%      -0.1        1.13 ±  5%  perf-profile.children.cycles-pp.unwind_next_frame
      2.01 ±  4%      -0.1        1.87 ±  3%  perf-profile.children.cycles-pp.arch_stack_walk
      0.55 ± 11%      -0.1        0.41 ±  7%  perf-profile.children.cycles-pp.ktime_get
      0.27 ± 38%      -0.1        0.14 ± 18%  perf-profile.children.cycles-pp.finish_task_switch
      0.25 ±  5%      -0.1        0.12 ± 23%  perf-profile.children.cycles-pp.xlog_state_release_iclog
      0.73 ±  9%      -0.1        0.61 ±  4%  perf-profile.children.cycles-pp.dequeue_entity
      0.82 ±  2%      -0.1        0.71 ±  6%  perf-profile.children.cycles-pp.md_submit_flush_data
      0.40 ± 12%      -0.1        0.29 ±  6%  perf-profile.children.cycles-pp.tick_nohz_next_event
      1.11 ±  2%      -0.1        1.01 ±  6%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.51 ±  6%      -0.1        0.40 ±  5%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.21 ± 10%      -0.1        0.11 ± 13%  perf-profile.children.cycles-pp.queue_work_on
      0.20 ± 12%      -0.1        0.11 ± 14%  perf-profile.children.cycles-pp.__queue_work
      0.32 ± 15%      -0.1        0.23 ±  5%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.44 ±  4%      -0.1        0.35 ±  8%  perf-profile.children.cycles-pp.__orc_find
      0.69 ±  6%      -0.1        0.60 ±  9%  perf-profile.children.cycles-pp.irq_exit
      0.49 ±  8%      -0.1        0.41 ±  5%  perf-profile.children.cycles-pp.orc_find
      0.39 ±  7%      -0.1        0.31 ± 12%  perf-profile.children.cycles-pp.clockevents_program_event
      0.33 ±  8%      -0.1        0.25 ± 12%  perf-profile.children.cycles-pp.update_rq_clock
      0.30 ±  9%      -0.1        0.23 ± 14%  perf-profile.children.cycles-pp.read_tsc
      0.24 ±  9%      -0.1        0.17 ±  2%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.21 ± 10%      -0.1        0.14 ± 13%  perf-profile.children.cycles-pp._cond_resched
      0.22 ± 14%      -0.1        0.16 ± 13%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.12 ± 17%      -0.1        0.06 ± 65%  perf-profile.children.cycles-pp.irq_enter
      0.25 ± 11%      -0.1        0.19 ±  8%  perf-profile.children.cycles-pp.sched_clock
      0.12 ±  6%      -0.1        0.07 ± 17%  perf-profile.children.cycles-pp.submit_flushes
      0.28 ±  9%      -0.1        0.22 ± 11%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.22 ±  8%      -0.1        0.17 ±  8%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.27 ± 10%      -0.1        0.22 ±  3%  perf-profile.children.cycles-pp.iomap_write_end
      0.23 ±  7%      -0.0        0.19 ±  9%  perf-profile.children.cycles-pp.native_sched_clock
      0.14 ±  6%      -0.0        0.09 ± 23%  perf-profile.children.cycles-pp.xfs_ilock
      0.11 ± 12%      -0.0        0.07 ± 19%  perf-profile.children.cycles-pp.preempt_schedule_common
      0.10 ±  8%      -0.0        0.07 ± 16%  perf-profile.children.cycles-pp.wait_for_completion_io
      0.09 ± 13%      -0.0        0.06 ±  9%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.09 ± 14%      -0.0        0.06 ±  6%  perf-profile.children.cycles-pp.sb_mark_inode_writeback
      0.23 ±  7%      -0.0        0.20 ±  2%  perf-profile.children.cycles-pp.update_blocked_averages
      0.10 ± 10%      -0.0        0.08 ±  8%  perf-profile.children.cycles-pp.bio_endio
      0.15 ±  4%      -0.0        0.13 ±  6%  perf-profile.children.cycles-pp.__switch_to
      0.08 ± 10%      -0.0        0.06 ± 11%  perf-profile.children.cycles-pp.timerqueue_del
      0.14 ± 11%      +0.0        0.18 ±  8%  perf-profile.children.cycles-pp.task_tick_fair
      0.43 ±  8%      +0.0        0.48 ±  4%  perf-profile.children.cycles-pp.select_idle_sibling
      0.31            +0.0        0.35 ±  6%  perf-profile.children.cycles-pp.__kernel_text_address
      0.34            +0.1        0.39 ±  5%  perf-profile.children.cycles-pp.unwind_get_return_address
      0.04 ± 57%      +0.1        0.10 ± 17%  perf-profile.children.cycles-pp.add_wait_queue_exclusive
      1.05 ±  6%      +0.3        1.33        perf-profile.children.cycles-pp.scheduler_ipi
      1.12 ±  6%      +0.3        1.41 ±  2%  perf-profile.children.cycles-pp.reschedule_interrupt
      0.45 ±  8%      +0.3        0.78 ±  6%  perf-profile.children.cycles-pp.xlog_state_done_syncing
      0.00            +1.7        1.75 ±  2%  perf-profile.children.cycles-pp.xlog_ioend_work
     18.75            +4.2       22.95        perf-profile.children.cycles-pp.remove_wait_queue
     69.75            +7.7       77.41        perf-profile.children.cycles-pp.write
     68.92            +7.7       76.64        perf-profile.children.cycles-pp.ksys_write
     68.89            +7.7       76.61        perf-profile.children.cycles-pp.vfs_write
     68.67            +7.7       76.41        perf-profile.children.cycles-pp.new_sync_write
     68.54            +7.7       76.29        perf-profile.children.cycles-pp.xfs_file_buffered_aio_write
     69.66            +7.9       77.53        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     69.63            +7.9       77.50        perf-profile.children.cycles-pp.do_syscall_64
     65.52            +8.0       73.49        perf-profile.children.cycles-pp.xfs_file_fsync
     55.57            +9.0       64.55        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
      7.55 ±  4%     +13.0       20.57 ±  3%  perf-profile.children.cycles-pp._raw_spin_lock
     27.21           +17.9       45.08        perf-profile.children.cycles-pp.__xfs_log_force_lsn
     15.33            -5.2       10.12        perf-profile.self.cycles-pp.intel_idle
      0.80 ±  8%      -0.2        0.64 ±  6%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.50 ±  8%      -0.2        0.35 ±  9%  perf-profile.self.cycles-pp.menu_select
      0.44 ±  4%      -0.1        0.35 ±  7%  perf-profile.self.cycles-pp.__orc_find
      0.29 ±  6%      -0.1        0.21 ± 11%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.29 ±  9%      -0.1        0.21 ± 23%  perf-profile.self.cycles-pp.try_to_wake_up
      0.29 ± 11%      -0.1        0.22 ± 15%  perf-profile.self.cycles-pp.read_tsc
      0.22 ± 15%      -0.1        0.16 ± 20%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.09 ± 10%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.finish_task_switch
      0.22 ±  8%      -0.1        0.16 ±  9%  perf-profile.self.cycles-pp.update_rq_clock
      0.13 ± 14%      -0.1        0.07 ± 24%  perf-profile.self.cycles-pp.dequeue_entity
      0.23 ±  6%      -0.1        0.17 ±  8%  perf-profile.self.cycles-pp.native_sched_clock
      0.17 ± 12%      -0.1        0.12 ±  9%  perf-profile.self.cycles-pp.do_idle
      0.21 ± 10%      -0.0        0.17 ± 10%  perf-profile.self.cycles-pp.__list_del_entry_valid
      0.20 ±  8%      -0.0        0.15 ± 14%  perf-profile.self.cycles-pp._raw_spin_lock_irq
      0.17 ± 11%      -0.0        0.12 ± 12%  perf-profile.self.cycles-pp.bio_alloc_bioset
      0.15 ±  7%      -0.0        0.10 ± 25%  perf-profile.self.cycles-pp.brd_lookup_page
      0.11 ± 15%      -0.0        0.07 ± 22%  perf-profile.self.cycles-pp.md_flush_request
      0.07 ±  6%      -0.0        0.03 ±100%  perf-profile.self.cycles-pp.bio_associate_blkg_from_css
      0.09 ± 24%      -0.0        0.06 ± 26%  perf-profile.self.cycles-pp.md_make_request
      0.07 ±  6%      -0.0        0.04 ± 58%  perf-profile.self.cycles-pp.stack_access_ok
      0.10 ± 15%      -0.0        0.08 ± 14%  perf-profile.self.cycles-pp.write
      0.15 ±  8%      -0.0        0.13 ± 10%  perf-profile.self.cycles-pp.orc_find
      0.15 ±  5%      -0.0        0.12 ±  6%  perf-profile.self.cycles-pp.__switch_to
      0.08 ± 10%      -0.0        0.06 ± 11%  perf-profile.self.cycles-pp.flush_bio_list
      0.05 ± 62%      +0.0        0.09 ± 12%  perf-profile.self.cycles-pp.xlog_cil_force_lsn
      0.01 ±173%      +0.0        0.06 ±  9%  perf-profile.self.cycles-pp.check_preempt_curr
     55.36            +8.6       63.92        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                                 aim7.jobs-per-min                              
                                                                                
  3500 +-+------------------------------------------------------------------+   
       |   +..+.+   +..+.O.+.+.+..+.+.+.+..+.O.+.O..+.+.+.+.O..O.+.+.+..+.+.|   
  3000 +-O O  O O O O  O   O O O  O O O O  O   O    O O O        O O        |   
       |   :    :   :                                                       |   
  2500 +-+ :    :   :                                                       |   
       |: :      : :                                                        |   
  2000 +-+:      : :                                                        |   
       |: :      : :                                                        |   
  1500 +-+:      : :                                                        |   
       |: :      : :                                                        |   
  1000 +-+:      : :                                                        |   
       | :        :                                                         |   
   500 +-+        :                                                         |   
       | :        :                                                         |   
     0 O-+------------------------------------------------O-----------------+   
                                                                                
                                                                                                                                                                
                                aim7.time.system_time                           
                                                                                
  10000 +-+----O-O-O-O-O--O-O-O-O-O--O-O-O-O-O-O--O-O-O-O----O-O-O-O--------+   
   9000 +-O O                                                               |   
        |                                                                   |   
   8000 +-+ +..+.+   +.+..+.+.+.+.+..+.+.+.+.+.+..+.+.+.+.+..+.+.+.+.+..+.+.|   
   7000 +-+ :    :   :                                                      |   
        |   :    :   :                                                      |   
   6000 +-+:      : :                                                       |   
   5000 +-+:      : :                                                       |   
   4000 +-+:      : :                                                       |   
        |: :      : :                                                       |   
   3000 +-+:      : :                                                       |   
   2000 +-+:      : :                                                       |   
        | :        :                                                        |   
   1000 +-+        :                                                        |   
      0 O-+-----------------------------------------------O-----------------+   
                                                                                
                                                                                                                                                                
                        aim7.time.involuntary_context_switches                  
                                                                                
  4.5e+06 +-+---------------------------------------------------------------+   
          |   +.+..+   +.+.+.+.+.+..+.+.+.+.+.+.+.+..+.+.+.+.+.+.+.+..+.+.+.|   
    4e+06 +-+ :    :   :                                                    |   
  3.5e+06 +-+ :    :   :                                                    |   
          |   :    :   :                                                    |   
    3e+06 +-+:      : :                                                     |   
  2.5e+06 +-O:O O  O:O:O O O O O O  O O O O O O O O  O O O   O O O O        |   
          |: :      : :                                                     |   
    2e+06 +-+:      : :                                                     |   
  1.5e+06 +-+:      : :                                                     |   
          |: :      : :                                                     |   
    1e+06 +-+        :                                                      |   
   500000 +-+        :                                                      |   
          | :        :                                                      |   
        0 O-+----------------------------------------------O----------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.2.0-rc4-00036-g79b54d9bfcdcd" of type "text/plain" (196406 bytes)

View attachment "job-script" of type "text/plain" (8115 bytes)

View attachment "job.yaml" of type "text/plain" (5726 bytes)

View attachment "reproduce" of type "text/plain" (1016 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ