lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Date:   Mon, 13 May 2019 10:50:04 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Josef Bacik <josef@...icpanda.com>
Cc:     David Sterba <dsterba@...e.com>,
        LKML <linux-kernel@...r.kernel.org>,
        Linus Torvalds <torvalds@...ux-foundation.org>, lkp@...org
Subject: [btrfs]  c8eaeac7b7:  aim7.jobs-per-min -11.7% regression

Greeting,

FYI, we noticed a -11.7% regression of aim7.jobs-per-min due to commit:


commit: c8eaeac7b734347c3afba7008b7af62f37b9c140 ("btrfs: reserve delalloc metadata differently")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: aim7
on test machine: 40 threads Intel(R) Xeon(R) CPU E5-2690 v2 @ 3.00GHz with 384G memory
with following parameters:

	disk: 4BRD_12G
	md: RAID0
	fs: btrfs
	test: disk_rr
	load: 1500
	cpufreq_governor: performance

test-description: AIM7 is a traditional UNIX system level benchmark suite which is used to test and measure the performance of multiuser system.
test-url: https://sourceforge.net/projects/aimbench/files/aim-suite7/



Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/disk/fs/kconfig/load/md/rootfs/tbox_group/test/testcase:
  gcc-7/performance/4BRD_12G/btrfs/x86_64-rhel-7.6/1500/RAID0/debian-x86_64-2018-04-03.cgz/lkp-ivb-ep01/disk_rr/aim7

commit: 
  4297ff84dc ("btrfs: track DIO bytes in flight")
  c8eaeac7b7 ("btrfs: reserve delalloc metadata differently")

4297ff84dc24d120 c8eaeac7b734347c3afba7008b7 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           75%           3:4     dmesg.WARNING:at#for_ip_swapgs_restore_regs_and_return_to_usermode/0x
           :4           75%           3:4     dmesg.WARNING:stack_recursion
         %stddev     %change         %stddev
             \          |                \  
     43654           -11.7%      38526        aim7.jobs-per-min
    206.46           +13.3%     233.89        aim7.time.elapsed_time
    206.46           +13.3%     233.89        aim7.time.elapsed_time.max
    611427 ±  3%     +36.6%     834981 ±  2%  aim7.time.involuntary_context_switches
    103580            +1.7%     105361        aim7.time.minor_page_faults
      7636           +14.8%       8763        aim7.time.system_time
   2731877            +7.6%    2940550        aim7.time.voluntary_context_switches
      0.83 ±  2%      -0.1        0.73 ±  2%  mpstat.cpu.all.usr%
     94.44            +7.0%     101.08        iostat.md0.w/s
     16028            +7.2%      17182        iostat.md0.wkB/s
      2040           +24.8%       2547 ± 18%  meminfo.Inactive(file)
     17154           -10.2%      15396        meminfo.max_used_kB
   2.4e+08 ± 45%     -87.4%   30268527 ±145%  cpuidle.C3.time
    731742 ± 18%     -88.7%      82339 ±109%  cpuidle.C3.usage
 4.447e+08 ± 12%     +35.0%  6.005e+08 ± 19%  cpuidle.C6.time
     10907            +6.9%      11662        vmstat.io.bo
     55.33 ±  4%     +13.9%      63.00 ±  2%  vmstat.procs.r
     24049            -6.9%      22394        vmstat.system.cs
     10588 ± 30%     -44.7%       5859 ± 25%  numa-meminfo.node0.Inactive
      9856 ± 36%     -47.8%       5145 ± 34%  numa-meminfo.node0.Inactive(anon)
      4691 ±  9%    +243.5%      16115 ± 69%  numa-meminfo.node0.KernelStack
      2202 ± 29%   +1132.8%      27151 ± 92%  numa-meminfo.node0.PageTables
     11332 ± 29%     +46.9%      16649 ±  8%  numa-meminfo.node1.Inactive
     10023 ± 36%     +47.8%      14817 ± 12%  numa-meminfo.node1.Inactive(anon)
      1185           +15.5%       1369 ±  7%  slabinfo.btrfs_ordered_extent.active_objs
      1185           +15.5%       1369 ±  7%  slabinfo.btrfs_ordered_extent.num_objs
      2748 ±  3%      +6.5%       2927 ±  3%  slabinfo.kmalloc-rcl-64.active_objs
      2748 ±  3%      +6.5%       2927 ±  3%  slabinfo.kmalloc-rcl-64.num_objs
    367.33 ±  3%     +29.2%     474.75 ± 10%  slabinfo.skbuff_fclone_cache.active_objs
    367.33 ±  3%     +29.2%     474.75 ± 10%  slabinfo.skbuff_fclone_cache.num_objs
      1545 ± 11%     +14.2%       1764 ± 13%  slabinfo.xfrm_dst_cache.num_objs
      2464 ± 36%     -47.8%       1285 ± 34%  numa-vmstat.node0.nr_inactive_anon
      4690 ±  9%    +243.7%      16120 ± 69%  numa-vmstat.node0.nr_kernel_stack
    550.00 ± 29%   +1134.5%       6790 ± 92%  numa-vmstat.node0.nr_page_table_pages
    152291 ±  3%     +19.0%     181153 ±  5%  numa-vmstat.node0.nr_written
      2464 ± 36%     -47.8%       1285 ± 34%  numa-vmstat.node0.nr_zone_inactive_anon
      2505 ± 36%     +47.9%       3705 ± 12%  numa-vmstat.node1.nr_inactive_anon
    151216 ±  3%     +18.3%     178821 ±  4%  numa-vmstat.node1.nr_written
      2505 ± 36%     +47.9%       3705 ± 12%  numa-vmstat.node1.nr_zone_inactive_anon
    493640            +1.0%     498686        proc-vmstat.nr_file_pages
    510.00           +24.8%     636.50 ± 18%  proc-vmstat.nr_inactive_file
    584507 ±  2%     +20.8%     706353 ±  2%  proc-vmstat.nr_written
    510.00           +24.8%     636.50 ± 18%  proc-vmstat.nr_zone_inactive_file
     29772 ±  5%      +4.0%      30949 ±  5%  proc-vmstat.numa_hint_faults
     15567 ±  6%      +5.4%      16416 ±  6%  proc-vmstat.numa_hint_faults_local
     13816            -1.7%      13583        proc-vmstat.numa_other
    582286           +10.0%     640790        proc-vmstat.pgfault
   2337841 ±  2%     +20.9%    2825298 ±  2%  proc-vmstat.pgpgout
      2730            +5.2%       2872        turbostat.Avg_MHz
    731158 ± 18%     -88.8%      81753 ±110%  turbostat.C3
      2.80 ± 44%      -2.5        0.32 ±146%  turbostat.C3%
      4.98 ± 25%     -37.4%       3.12 ± 17%  turbostat.CPU%c1
      1.14 ± 78%     -79.4%       0.24 ±160%  turbostat.CPU%c3
    135.74           +10.2%     149.62        turbostat.CorWatt
  18209025           +13.7%   20695723        turbostat.IRQ
    162.95            +8.5%     176.76        turbostat.PkgWatt
      9586 ±  3%      +8.6%      10410 ±  2%  turbostat.SMI
     24351            -6.9%      22675        perf-stat.i.context-switches
 1.095e+11            +5.2%  1.151e+11        perf-stat.i.cpu-cycles
      2942            -8.8%       2682        perf-stat.i.cpu-migrations
      4026 ±  2%      +5.5%       4248        perf-stat.i.cycles-between-cache-misses
      0.11 ±  7%      +0.0        0.15 ± 27%  perf-stat.i.dTLB-store-miss-rate%
 1.788e+09           -10.4%  1.602e+09        perf-stat.i.dTLB-stores
   4679972 ±  2%     -36.2%    2983497 ± 47%  perf-stat.i.iTLB-loads
      0.24            -3.5%       0.23        perf-stat.i.ipc
      2646            -2.6%       2578        perf-stat.i.minor-faults
     40.94            +0.4       41.38        perf-stat.i.node-store-miss-rate%
      2646            -2.6%       2578        perf-stat.i.page-faults
      4.07            +4.1%       4.24        perf-stat.overall.cpi
      4196            +5.1%       4411        perf-stat.overall.cycles-between-cache-misses
      0.11 ±  8%      +0.0        0.15 ± 29%  perf-stat.overall.dTLB-store-miss-rate%
      0.25            -4.0%       0.24        perf-stat.overall.ipc
     24233            -6.8%      22576        perf-stat.ps.context-switches
 1.089e+11            +5.2%  1.146e+11        perf-stat.ps.cpu-cycles
      2928            -8.8%       2670        perf-stat.ps.cpu-migrations
 1.779e+09           -10.3%  1.595e+09        perf-stat.ps.dTLB-stores
   4657162 ±  2%     -36.2%    2970461 ± 47%  perf-stat.ps.iTLB-loads
      2635            -2.5%       2570        perf-stat.ps.minor-faults
      2635            -2.5%       2570        perf-stat.ps.page-faults
 5.677e+12           +14.2%  6.482e+12        perf-stat.total.instructions
     23484           +27.9%      30030 ±  2%  sched_debug.cfs_rq:/.load.avg
     51446 ±  2%    +492.7%     304901 ±  2%  sched_debug.cfs_rq:/.load.max
     11212 ±  5%    +332.6%      48506 ±  2%  sched_debug.cfs_rq:/.load.stddev
      0.21 ± 14%     -27.1%       0.15 ± 19%  sched_debug.cfs_rq:/.nr_running.stddev
     50.75 ± 10%    +138.7%     121.12 ± 91%  sched_debug.cfs_rq:/.runnable_load_avg.max
      1.33 ± 93%    +275.0%       5.00 ± 28%  sched_debug.cfs_rq:/.runnable_load_avg.min
     18594 ±  4%     +39.0%      25853 ±  3%  sched_debug.cfs_rq:/.runnable_weight.avg
     41096 ±  7%    +627.8%     299088        sched_debug.cfs_rq:/.runnable_weight.max
      9009 ±  3%    +431.5%      47888 ±  2%  sched_debug.cfs_rq:/.runnable_weight.stddev
     -4276          -670.0%      24374 ± 56%  sched_debug.cfs_rq:/.spread0.avg
     74100 ±  9%     +24.2%      92022 ± 12%  sched_debug.cfs_rq:/.spread0.max
    -57421           -37.4%     -35972        sched_debug.cfs_rq:/.spread0.min
    463.62           +12.9%     523.36 ±  6%  sched_debug.cfs_rq:/.util_est_enqueued.avg
      0.25 ± 81%   +9225.0%      23.31 ±119%  sched_debug.cfs_rq:/.util_est_enqueued.min
     18.98 ±  4%      +9.4%      20.76 ±  4%  sched_debug.cpu.cpu_load[0].avg
     47.67 ±  5%     +20.8%      57.56 ±  8%  sched_debug.cpu.cpu_load[0].max
      1.92 ± 43%    +154.3%       4.88 ± 40%  sched_debug.cpu.cpu_load[0].min
     44.75           +22.2%      54.69 ±  9%  sched_debug.cpu.cpu_load[1].max
      5.42 ± 15%     +38.5%       7.50 ±  9%  sched_debug.cpu.cpu_load[2].min
     56.17 ±  5%     +14.1%      64.06 ±  7%  sched_debug.cpu.cpu_load[3].max
      6.33 ±  9%     +28.3%       8.12 ±  9%  sched_debug.cpu.cpu_load[3].min
      7.17 ±  4%     +16.9%       8.38 ±  8%  sched_debug.cpu.cpu_load[4].min
    454.42 ± 86%    +142.9%       1103 ± 21%  sched_debug.cpu.curr->pid.min
     23570           +20.9%      28493 ±  8%  sched_debug.cpu.load.avg
     51446 ±  2%    +371.3%     242447 ± 45%  sched_debug.cpu.load.max
      4293 ± 80%    +118.5%       9381 ± 21%  sched_debug.cpu.load.min
      2039 ±  7%     +10.2%       2247 ±  2%  sched_debug.cpu.nr_load_updates.stddev
      1.06 ±  2%     +22.2%       1.29 ±  5%  sched_debug.cpu.nr_running.avg
      0.25 ± 81%    +150.0%       0.62 ± 20%  sched_debug.cpu.nr_running.min
      0.52           +13.5%       0.59 ± 10%  sched_debug.cpu.nr_running.stddev
     19645 ±  5%      +9.4%      21489 ±  4%  softirqs.CPU0.RCU
     86145           +14.7%      98787        softirqs.CPU0.TIMER
     87552 ±  7%     +12.6%      98576 ±  4%  softirqs.CPU1.TIMER
     19349 ±  9%     +13.9%      22048 ±  5%  softirqs.CPU10.RCU
     81218 ±  3%     +29.7%     105305 ± 12%  softirqs.CPU10.TIMER
     19223 ±  2%      +8.7%      20891 ±  7%  softirqs.CPU11.RCU
     83829 ±  2%     +12.5%      94302        softirqs.CPU11.TIMER
     84709           +16.0%      98222 ±  2%  softirqs.CPU12.TIMER
     85057 ±  2%     +10.5%      93955        softirqs.CPU13.TIMER
     20677 ±  7%      +6.1%      21934 ±  7%  softirqs.CPU14.RCU
     84147 ±  2%     +17.2%      98617        softirqs.CPU14.TIMER
     19576 ±  4%     +13.0%      22113 ±  7%  softirqs.CPU16.RCU
     84576 ±  2%     +33.1%     112557 ± 14%  softirqs.CPU16.TIMER
     85248 ±  4%     +14.5%      97595 ±  3%  softirqs.CPU17.TIMER
     19195 ±  6%     +15.2%      22106 ± 11%  softirqs.CPU18.RCU
     85046 ±  2%     +14.4%      97311        softirqs.CPU18.TIMER
     85248 ±  3%     +11.4%      94985 ±  2%  softirqs.CPU19.TIMER
     19415 ±  3%     +11.4%      21619 ±  3%  softirqs.CPU2.RCU
     19095 ±  4%     +10.6%      21115 ±  5%  softirqs.CPU20.RCU
     84097 ±  2%     +15.0%      96686        softirqs.CPU20.TIMER
     87892 ±  3%     +13.1%      99386 ±  5%  softirqs.CPU21.TIMER
     19125 ±  3%     +13.4%      21686 ±  5%  softirqs.CPU22.RCU
     14525 ±  5%      -9.0%      13212 ±  4%  softirqs.CPU22.SCHED
     85902 ±  5%     +15.4%      99114        softirqs.CPU22.TIMER
     19616 ±  3%     +12.4%      22042 ±  2%  softirqs.CPU23.RCU
     86349 ±  4%     +12.0%      96741        softirqs.CPU23.TIMER
     19747 ±  6%     +13.5%      22407 ±  7%  softirqs.CPU24.RCU
     84333           +17.1%      98749        softirqs.CPU24.TIMER
     19131 ±  2%     +11.2%      21279 ±  5%  softirqs.CPU25.RCU
     19625 ±  5%      +9.1%      21420 ±  5%  softirqs.CPU26.RCU
     84545 ±  3%     +15.5%      97675        softirqs.CPU26.TIMER
     20319 ±  7%      +5.3%      21389 ±  6%  softirqs.CPU27.RCU
     86276 ±  4%     +12.1%      96743        softirqs.CPU27.TIMER
     18484 ±  4%     +15.3%      21315 ±  7%  softirqs.CPU28.RCU
     83144           +16.5%      96862        softirqs.CPU28.TIMER
     19786 ±  2%     +12.0%      22157 ±  4%  softirqs.CPU3.RCU
     87487 ±  5%     +12.2%      98117        softirqs.CPU3.TIMER
     80763 ±  3%     +20.0%      96896 ±  2%  softirqs.CPU30.TIMER
     83628 ±  2%     +12.5%      94076        softirqs.CPU31.TIMER
     85112 ±  4%     +14.8%      97737 ±  3%  softirqs.CPU32.TIMER
     84626 ±  3%     +10.9%      93812        softirqs.CPU33.TIMER
     87944 ±  6%     +12.1%      98575        softirqs.CPU34.TIMER
     86859           +22.5%     106365 ± 13%  softirqs.CPU36.TIMER
     85336 ±  4%     +16.8%      99655 ±  4%  softirqs.CPU37.TIMER
     85090 ±  2%     +14.2%      97153        softirqs.CPU38.TIMER
     19528 ±  4%     +16.7%      22783 ±  8%  softirqs.CPU4.RCU
     84396           +18.0%      99614        softirqs.CPU4.TIMER
     19042 ±  4%      +8.8%      20714 ±  4%  softirqs.CPU5.RCU
     87389 ±  5%     +12.7%      98473 ±  2%  softirqs.CPU5.TIMER
     20312 ±  3%     +13.2%      22992 ±  6%  softirqs.CPU6.RCU
     84550 ±  3%     +15.8%      97877        softirqs.CPU6.TIMER
     18516 ±  7%     +24.9%      23125 ±  5%  softirqs.CPU8.RCU
     82667           +28.2%     106005 ± 11%  softirqs.CPU8.TIMER
    764529 ±  2%      +8.9%     832849 ±  5%  softirqs.RCU
   3442286 ±  2%     +13.9%    3922206        softirqs.TIMER
    275.00 ± 17%     +64.0%     451.00 ± 28%  interrupts.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    204.00 ± 56%     +41.4%     288.50 ± 52%  interrupts.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
    103.67           +13.3%     117.50        interrupts.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
     78055           +10.9%      86558        interrupts.CAL:Function_call_interrupts
    425311           +13.1%     480959        interrupts.CPU0.LOC:Local_timer_interrupts
     22301 ±  3%     +25.5%      27997        interrupts.CPU0.RES:Rescheduling_interrupts
      2033 ±  5%     +12.1%       2278 ±  3%  interrupts.CPU1.CAL:Function_call_interrupts
    425286           +13.0%     480441        interrupts.CPU1.LOC:Local_timer_interrupts
     22426 ±  3%     +26.7%      28410 ±  2%  interrupts.CPU1.RES:Rescheduling_interrupts
    424918           +13.2%     480935        interrupts.CPU10.LOC:Local_timer_interrupts
     22298 ±  2%     +25.8%      28049        interrupts.CPU10.RES:Rescheduling_interrupts
      1959 ±  5%     +12.8%       2210 ±  6%  interrupts.CPU11.CAL:Function_call_interrupts
    424741           +13.1%     480460        interrupts.CPU11.LOC:Local_timer_interrupts
     21914 ±  3%     +27.7%      27983        interrupts.CPU11.RES:Rescheduling_interrupts
    424836           +13.2%     480889        interrupts.CPU12.LOC:Local_timer_interrupts
     21863 ±  3%     +26.4%      27640        interrupts.CPU12.RES:Rescheduling_interrupts
      1870           +21.7%       2276 ±  6%  interrupts.CPU13.CAL:Function_call_interrupts
    425017           +12.9%     479899        interrupts.CPU13.LOC:Local_timer_interrupts
      1379 ±140%    +232.7%       4590 ± 62%  interrupts.CPU13.NMI:Non-maskable_interrupts
      1379 ±140%    +232.7%       4590 ± 62%  interrupts.CPU13.PMI:Performance_monitoring_interrupts
     21979 ±  2%     +30.5%      28691 ±  3%  interrupts.CPU13.RES:Rescheduling_interrupts
    424883           +13.2%     480884        interrupts.CPU14.LOC:Local_timer_interrupts
     21978           +27.1%      27940 ±  2%  interrupts.CPU14.RES:Rescheduling_interrupts
      1919 ±  8%     +18.7%       2278 ±  6%  interrupts.CPU15.CAL:Function_call_interrupts
    425309           +13.0%     480679        interrupts.CPU15.LOC:Local_timer_interrupts
     22330 ±  3%     +25.9%      28118        interrupts.CPU15.RES:Rescheduling_interrupts
      1925 ±  6%     +18.0%       2271 ±  4%  interrupts.CPU16.CAL:Function_call_interrupts
    424967           +13.1%     480548        interrupts.CPU16.LOC:Local_timer_interrupts
     21993 ±  3%     +28.3%      28215        interrupts.CPU16.RES:Rescheduling_interrupts
      2102            +9.7%       2305 ±  5%  interrupts.CPU17.CAL:Function_call_interrupts
    424760           +13.0%     480175        interrupts.CPU17.LOC:Local_timer_interrupts
     22036 ±  2%     +28.5%      28307 ±  2%  interrupts.CPU17.RES:Rescheduling_interrupts
    424881           +13.1%     480700        interrupts.CPU18.LOC:Local_timer_interrupts
     22029 ±  3%     +26.9%      27950        interrupts.CPU18.RES:Rescheduling_interrupts
    424963           +13.1%     480489        interrupts.CPU19.LOC:Local_timer_interrupts
     23004 ±  3%     +27.6%      29354 ±  2%  interrupts.CPU19.RES:Rescheduling_interrupts
    424990           +13.1%     480714        interrupts.CPU2.LOC:Local_timer_interrupts
     22102 ±  3%     +26.5%      27967        interrupts.CPU2.RES:Rescheduling_interrupts
      1757 ±  6%     +17.1%       2057 ±  6%  interrupts.CPU20.CAL:Function_call_interrupts
    425145           +13.1%     480765        interrupts.CPU20.LOC:Local_timer_interrupts
     22303 ±  2%     +25.8%      28066        interrupts.CPU20.RES:Rescheduling_interrupts
    425267           +12.9%     480312        interrupts.CPU21.LOC:Local_timer_interrupts
     21813 ±  2%     +27.1%      27723        interrupts.CPU21.RES:Rescheduling_interrupts
    424618           +13.2%     480680        interrupts.CPU22.LOC:Local_timer_interrupts
     22015 ±  3%     +26.0%      27734        interrupts.CPU22.RES:Rescheduling_interrupts
      1912 ±  8%     +18.9%       2273 ±  5%  interrupts.CPU23.CAL:Function_call_interrupts
    424735           +13.1%     480469        interrupts.CPU23.LOC:Local_timer_interrupts
     22040           +27.5%      28098 ±  3%  interrupts.CPU23.RES:Rescheduling_interrupts
      1924 ±  5%      +6.4%       2048 ±  4%  interrupts.CPU24.CAL:Function_call_interrupts
    424618           +13.2%     480598        interrupts.CPU24.LOC:Local_timer_interrupts
     21873 ±  2%     +28.1%      28009        interrupts.CPU24.RES:Rescheduling_interrupts
    424698           +13.1%     480416        interrupts.CPU25.LOC:Local_timer_interrupts
     22041           +28.3%      28287 ±  2%  interrupts.CPU25.RES:Rescheduling_interrupts
    275.00 ± 17%     +64.0%     451.00 ± 28%  interrupts.CPU26.37:IR-PCI-MSI.524289-edge.eth0-TxRx-0
    424902           +13.1%     480512        interrupts.CPU26.LOC:Local_timer_interrupts
     22002 ±  3%     +27.0%      27949        interrupts.CPU26.RES:Rescheduling_interrupts
    424698           +13.2%     480806        interrupts.CPU27.LOC:Local_timer_interrupts
      5859 ± 30%     -81.3%       1093 ±173%  interrupts.CPU27.NMI:Non-maskable_interrupts
      5859 ± 30%     -81.3%       1093 ±173%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
     22290           +25.4%      27948 ±  3%  interrupts.CPU27.RES:Rescheduling_interrupts
      1904 ±  9%     +12.6%       2144 ±  8%  interrupts.CPU28.CAL:Function_call_interrupts
    424963           +13.1%     480696        interrupts.CPU28.LOC:Local_timer_interrupts
     21860           +28.7%      28145        interrupts.CPU28.RES:Rescheduling_interrupts
      1933 ±  7%     +18.4%       2288 ±  4%  interrupts.CPU29.CAL:Function_call_interrupts
    424503           +13.2%     480645        interrupts.CPU29.LOC:Local_timer_interrupts
     22101 ±  4%     +27.4%      28162 ±  3%  interrupts.CPU29.RES:Rescheduling_interrupts
      1994 ±  2%     +11.0%       2214 ±  4%  interrupts.CPU3.CAL:Function_call_interrupts
    424920           +13.1%     480485        interrupts.CPU3.LOC:Local_timer_interrupts
     21988 ±  2%     +27.7%      28074 ±  2%  interrupts.CPU3.RES:Rescheduling_interrupts
    425052           +13.1%     480827        interrupts.CPU30.LOC:Local_timer_interrupts
     22012 ±  2%     +26.6%      27873        interrupts.CPU30.RES:Rescheduling_interrupts
      1986 ±  6%     +12.8%       2240 ±  5%  interrupts.CPU31.CAL:Function_call_interrupts
    424997           +13.1%     480581        interrupts.CPU31.LOC:Local_timer_interrupts
     21947 ±  3%     +27.3%      27935 ±  2%  interrupts.CPU31.RES:Rescheduling_interrupts
    204.00 ± 56%     +41.4%     288.50 ± 52%  interrupts.CPU32.40:IR-PCI-MSI.524292-edge.eth0-TxRx-3
    424705           +13.1%     480486        interrupts.CPU32.LOC:Local_timer_interrupts
     21887           +28.5%      28124        interrupts.CPU32.RES:Rescheduling_interrupts
    425192           +13.0%     480284        interrupts.CPU33.LOC:Local_timer_interrupts
     22206 ±  3%     +25.4%      27853 ±  2%  interrupts.CPU33.RES:Rescheduling_interrupts
      1972 ±  6%     +12.8%       2224 ±  6%  interrupts.CPU34.CAL:Function_call_interrupts
    424557           +13.2%     480684        interrupts.CPU34.LOC:Local_timer_interrupts
     22117 ±  3%     +25.6%      27777        interrupts.CPU34.RES:Rescheduling_interrupts
      1894           +13.4%       2147 ±  4%  interrupts.CPU35.CAL:Function_call_interrupts
    424767           +13.1%     480496        interrupts.CPU35.LOC:Local_timer_interrupts
     22043 ±  3%     +28.0%      28225        interrupts.CPU35.RES:Rescheduling_interrupts
    424967           +13.1%     480760        interrupts.CPU36.LOC:Local_timer_interrupts
     21838 ±  3%     +28.6%      28082        interrupts.CPU36.RES:Rescheduling_interrupts
      1855 ±  3%     +19.7%       2220 ±  7%  interrupts.CPU37.CAL:Function_call_interrupts
    424903           +13.0%     480330        interrupts.CPU37.LOC:Local_timer_interrupts
     22221 ±  2%     +27.8%      28392 ±  2%  interrupts.CPU37.RES:Rescheduling_interrupts
      1880           +15.3%       2167 ±  6%  interrupts.CPU38.CAL:Function_call_interrupts
    425042           +13.1%     480659        interrupts.CPU38.LOC:Local_timer_interrupts
     21922 ±  3%     +28.6%      28182        interrupts.CPU38.RES:Rescheduling_interrupts
      1845 ±  2%     +19.8%       2212 ±  6%  interrupts.CPU39.CAL:Function_call_interrupts
    425307           +13.0%     480459        interrupts.CPU39.LOC:Local_timer_interrupts
     21737 ±  2%     +30.3%      28324 ±  2%  interrupts.CPU39.RES:Rescheduling_interrupts
      1901 ±  8%     +18.2%       2247 ±  4%  interrupts.CPU4.CAL:Function_call_interrupts
    424921           +13.2%     480871        interrupts.CPU4.LOC:Local_timer_interrupts
     22205 ±  2%     +26.1%      28002        interrupts.CPU4.RES:Rescheduling_interrupts
    425096           +13.1%     480655        interrupts.CPU5.LOC:Local_timer_interrupts
     22004           +28.7%      28319 ±  2%  interrupts.CPU5.RES:Rescheduling_interrupts
    103.67           +13.3%     117.50        interrupts.CPU6.94:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
    425006           +13.1%     480843        interrupts.CPU6.LOC:Local_timer_interrupts
     22177           +26.5%      28053        interrupts.CPU6.RES:Rescheduling_interrupts
      1962 ±  5%     +17.3%       2301 ±  7%  interrupts.CPU7.CAL:Function_call_interrupts
    424930           +13.1%     480702        interrupts.CPU7.LOC:Local_timer_interrupts
      1402 ±138%    +387.3%       6831 ± 49%  interrupts.CPU7.NMI:Non-maskable_interrupts
      1402 ±138%    +387.3%       6831 ± 49%  interrupts.CPU7.PMI:Performance_monitoring_interrupts
     21891 ±  2%     +28.1%      28048 ±  2%  interrupts.CPU7.RES:Rescheduling_interrupts
      1765 ±  6%     +15.8%       2044 ±  5%  interrupts.CPU8.CAL:Function_call_interrupts
    423965           +13.4%     480881        interrupts.CPU8.LOC:Local_timer_interrupts
     22606 ±  4%     +23.5%      27917        interrupts.CPU8.RES:Rescheduling_interrupts
    424961           +13.1%     480710        interrupts.CPU9.LOC:Local_timer_interrupts
     22153           +27.1%      28146 ±  2%  interrupts.CPU9.RES:Rescheduling_interrupts
  16996312           +13.1%   19224399        interrupts.LOC:Local_timer_interrupts
    883558 ±  2%     +27.2%    1124085        interrupts.RES:Rescheduling_interrupts
     43.50           -10.6       32.91        perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
     42.95           -10.6       32.38        perf-profile.calltrace.cycles-pp.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
     41.58           -10.4       31.23        perf-profile.calltrace.cycles-pp._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write.btrfs_file_write_iter
     41.28           -10.3       30.96        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_buffered_write
     42.18           -10.3       31.90        perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter
     42.48           -10.3       32.21        perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
     42.65           -10.3       32.39        perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
     42.08           -10.2       31.85        perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_buffered_write
      0.87 ±  4%      -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.87 ±  4%      -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64
      0.87 ±  4%      -0.6        0.27 ±100%  perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64
      0.88 ±  2%      -0.5        0.40 ± 58%  perf-profile.calltrace.cycles-pp.secondary_startup_64
      2.01            -0.4        1.60 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
      1.55 ±  2%      -0.3        1.22 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
      1.78            -0.3        1.46        perf-profile.calltrace.cycles-pp.read
      1.53 ±  2%      -0.3        1.21 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages.btrfs_buffered_write
      1.10 ±  2%      -0.3        0.81 ±  5%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent.btrfs_dirty_pages
      1.05 ±  2%      -0.3        0.77 ±  6%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.btrfs_get_extent
      1.31            -0.2        1.06        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.32            -0.2        1.09        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
      1.22            -0.2        1.01        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.16            -0.2        0.95        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      1.43 ±  4%      -0.2        1.23 ±  3%  perf-profile.calltrace.cycles-pp.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
      0.76 ±  2%      -0.2        0.58 ±  3%  perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.close
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.75 ±  2%      -0.2        0.57 ±  3%  perf-profile.calltrace.cycles-pp.dput.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64
      0.76 ±  2%      -0.2        0.59 ±  3%  perf-profile.calltrace.cycles-pp.close
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.calltrace.cycles-pp.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.calltrace.cycles-pp.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe.close
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.calltrace.cycles-pp.__fput.task_work_run.exit_to_usermode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.74 ±  2%      -0.2        0.57 ±  3%  perf-profile.calltrace.cycles-pp.btrfs_evict_inode.evict.__dentry_kill.dentry_kill.dput
      0.75 ±  2%      -0.2        0.57 ±  3%  perf-profile.calltrace.cycles-pp.dentry_kill.dput.__fput.task_work_run.exit_to_usermode_loop
      0.75 ±  2%      -0.2        0.57 ±  3%  perf-profile.calltrace.cycles-pp.__dentry_kill.dentry_kill.dput.__fput.task_work_run
      0.74 ±  2%      -0.2        0.57 ±  3%  perf-profile.calltrace.cycles-pp.evict.__dentry_kill.dentry_kill.dput.__fput
      0.86            -0.1        0.73        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.72 ±  2%      -0.1        0.58 ±  2%  perf-profile.calltrace.cycles-pp.lseek64
      0.83            -0.1        0.70        perf-profile.calltrace.cycles-pp.generic_file_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
      0.74 ±  4%      -0.1        0.61 ±  5%  perf-profile.calltrace.cycles-pp.pagecache_get_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
      0.64 ±  5%      -0.1        0.58 ±  2%  perf-profile.calltrace.cycles-pp.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
      0.59 ±  4%      -0.0        0.54 ±  2%  perf-profile.calltrace.cycles-pp.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write.btrfs_file_write_iter
      0.58 ±  5%      -0.0        0.53 ±  2%  perf-profile.calltrace.cycles-pp.__extent_read_full_page.extent_read_full_page.prepare_uptodate_page.prepare_pages.btrfs_buffered_write
     94.94            +1.2       96.10        perf-profile.calltrace.cycles-pp.write
     94.38            +1.2       95.60        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     94.47            +1.2       95.69        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     94.49            +1.2       95.72        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     94.29            +1.2       95.53        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     93.96            +1.3       95.23        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     93.90            +1.3       95.19        perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     93.59            +1.3       94.92        perf-profile.calltrace.cycles-pp.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
     41.79           +12.8       54.57        perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release
      3.76           +22.7       26.46        perf-profile.calltrace.cycles-pp.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write.vfs_write
      0.00           +23.2       23.16        perf-profile.calltrace.cycles-pp._raw_spin_lock.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent
      0.00           +23.2       23.20        perf-profile.calltrace.cycles-pp.space_info_add_old_bytes.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit
      0.83 ±  3%     +23.3       24.09        perf-profile.calltrace.cycles-pp.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter.new_sync_write
      0.81 ±  3%     +23.3       24.09        perf-profile.calltrace.cycles-pp.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write.btrfs_file_write_iter
      0.58 ±  2%     +23.3       23.88        perf-profile.calltrace.cycles-pp.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages
      0.60 ±  2%     +23.3       23.90        perf-profile.calltrace.cycles-pp.clear_state_bit.__clear_extent_bit.clear_extent_bit.btrfs_dirty_pages.btrfs_buffered_write
      0.00           +23.4       23.36        perf-profile.calltrace.cycles-pp.block_rsv_release_bytes.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit
      0.00           +23.5       23.48        perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_clear_delalloc_extent.clear_state_bit.__clear_extent_bit.clear_extent_bit
     43.13           -10.6       32.50        perf-profile.children.cycles-pp.reserve_metadata_bytes
     43.51           -10.6       32.92        perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
      2.02            -0.4        1.60 ±  3%  perf-profile.children.cycles-pp.btrfs_get_extent
      0.88 ±  2%      -0.4        0.51 ± 10%  perf-profile.children.cycles-pp.secondary_startup_64
      0.87 ±  4%      -0.4        0.50 ±  9%  perf-profile.children.cycles-pp.start_secondary
      0.88 ±  2%      -0.4        0.51 ± 10%  perf-profile.children.cycles-pp.cpu_startup_entry
      0.88 ±  2%      -0.4        0.51 ± 10%  perf-profile.children.cycles-pp.do_idle
      0.83 ±  3%      -0.4        0.48 ±  9%  perf-profile.children.cycles-pp.cpuidle_enter_state
      0.82 ±  4%      -0.3        0.47 ±  9%  perf-profile.children.cycles-pp.intel_idle
      1.64 ±  2%      -0.3        1.30 ±  3%  perf-profile.children.cycles-pp.btrfs_search_slot
      1.55 ±  2%      -0.3        1.22 ±  3%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
      1.81            -0.3        1.50        perf-profile.children.cycles-pp.read
      1.15 ±  3%      -0.3        0.86 ±  4%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
      1.10 ±  4%      -0.3        0.81 ±  6%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
      1.01 ±  4%      -0.3        0.74 ±  7%  perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.23            -0.2        1.01        perf-profile.children.cycles-pp.ksys_read
      1.43 ±  4%      -0.2        1.24 ±  3%  perf-profile.children.cycles-pp.prepare_pages
      1.16            -0.2        0.96        perf-profile.children.cycles-pp.vfs_read
      1.07 ±  3%      -0.2        0.87 ±  3%  perf-profile.children.cycles-pp.__set_extent_bit
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.children.cycles-pp.task_work_run
      0.65 ±  5%      -0.2        0.48 ±  9%  perf-profile.children.cycles-pp.prepare_to_wait_event
      0.76 ±  2%      -0.2        0.59 ±  3%  perf-profile.children.cycles-pp.close
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.children.cycles-pp.__fput
      0.75 ±  2%      -0.2        0.58 ±  3%  perf-profile.children.cycles-pp.dput
      0.74 ±  2%      -0.2        0.57 ±  3%  perf-profile.children.cycles-pp.btrfs_evict_inode
      0.75 ±  2%      -0.2        0.57 ±  3%  perf-profile.children.cycles-pp.dentry_kill
      0.75 ±  2%      -0.2        0.57 ±  3%  perf-profile.children.cycles-pp.__dentry_kill
      0.75 ±  2%      -0.2        0.58 ±  4%  perf-profile.children.cycles-pp.exit_to_usermode_loop
      0.74 ±  2%      -0.2        0.57 ±  3%  perf-profile.children.cycles-pp.evict
      0.92 ±  2%      -0.2        0.76 ±  3%  perf-profile.children.cycles-pp.pagecache_get_page
      0.75 ±  2%      -0.1        0.61        perf-profile.children.cycles-pp.lseek64
      0.87            -0.1        0.73        perf-profile.children.cycles-pp.new_sync_read
      0.84            -0.1        0.70        perf-profile.children.cycles-pp.generic_file_read_iter
      0.68 ±  2%      -0.1        0.55        perf-profile.children.cycles-pp.entry_SYSCALL_64
      0.46            -0.1        0.34 ±  5%  perf-profile.children.cycles-pp.lock_and_cleanup_extent_if_need
      0.32 ±  6%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.creat
      0.32 ±  5%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.do_sys_open
      0.32 ±  5%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.do_filp_open
      0.32 ±  5%      -0.1        0.21 ±  6%  perf-profile.children.cycles-pp.path_openat
      1.00 ±  2%      -0.1        0.90 ±  4%  perf-profile.children.cycles-pp.can_overcommit
      0.62            -0.1        0.52        perf-profile.children.cycles-pp.syscall_return_via_sysret
      0.43 ±  5%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.lock_extent_bits
      0.57 ±  4%      -0.1        0.48 ±  3%  perf-profile.children.cycles-pp.set_extent_bit
      0.31 ±  5%      -0.1        0.22 ±  9%  perf-profile.children.cycles-pp.finish_wait
      0.43 ±  2%      -0.1        0.35        perf-profile.children.cycles-pp.alloc_extent_state
      0.16 ± 17%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.call_rwsem_down_write_failed
      0.16 ± 17%      -0.1        0.09 ±  8%  perf-profile.children.cycles-pp.rwsem_down_write_failed
      0.20 ± 12%      -0.1        0.13 ±  7%  perf-profile.children.cycles-pp.down_write
      0.23 ±  2%      -0.1        0.16 ±  6%  perf-profile.children.cycles-pp.do_unlinkat
      0.23            -0.1        0.17 ±  3%  perf-profile.children.cycles-pp.unlink
      0.31 ±  3%      -0.1        0.25 ±  2%  perf-profile.children.cycles-pp.truncate_inode_pages_range
      0.64 ±  5%      -0.1        0.58 ±  2%  perf-profile.children.cycles-pp.prepare_uptodate_page
      0.46            -0.1        0.40 ±  3%  perf-profile.children.cycles-pp.copy_page_to_iter
      0.40 ±  5%      -0.1        0.34 ±  3%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.48 ±  2%      -0.1        0.42 ±  2%  perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.46            -0.1        0.40 ±  2%  perf-profile.children.cycles-pp.__do_readpage
      0.38 ±  6%      -0.1        0.32 ±  2%  perf-profile.children.cycles-pp.btrfs_set_extent_delalloc
      0.31            -0.1        0.25 ±  5%  perf-profile.children.cycles-pp.security_file_permission
      0.29 ±  4%      -0.1        0.24        perf-profile.children.cycles-pp.find_get_entry
      0.17 ±  4%      -0.1        0.12 ±  7%  perf-profile.children.cycles-pp.btrfs_create
      0.18 ±  9%      -0.1        0.13 ±  5%  perf-profile.children.cycles-pp.btrfs_trans_release_metadata
      0.59 ±  4%      -0.0        0.54 ±  2%  perf-profile.children.cycles-pp.extent_read_full_page
      0.17 ±  4%      -0.0        0.12 ±  4%  perf-profile.children.cycles-pp.btrfs_unlink
      0.18 ±  9%      -0.0        0.14 ±  8%  perf-profile.children.cycles-pp.__btrfs_end_transaction
      0.17 ±  4%      -0.0        0.13 ±  3%  perf-profile.children.cycles-pp.vfs_unlink
      0.31 ±  5%      -0.0        0.27 ±  3%  perf-profile.children.cycles-pp.btrfs_alloc_data_chunk_ondemand
      0.14 ± 11%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
      0.33 ±  5%      -0.0        0.29 ±  5%  perf-profile.children.cycles-pp.btrfs_check_data_free_space
      0.28 ±  7%      -0.0        0.24 ±  5%  perf-profile.children.cycles-pp.add_to_page_cache_lru
      0.21 ±  8%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.memset_erms
      0.11 ± 15%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.__btrfs_btree_balance_dirty
      0.06 ±  7%      -0.0        0.03 ±100%  perf-profile.children.cycles-pp.__btrfs_unlink_inode
      0.19 ±  4%      -0.0        0.15 ±  2%  perf-profile.children.cycles-pp.___might_sleep
      0.16 ±  5%      -0.0        0.12 ±  3%  perf-profile.children.cycles-pp.xas_load
      0.11 ± 19%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.btrfs_delayed_inode_release_metadata
      0.24 ±  5%      -0.0        0.20 ±  3%  perf-profile.children.cycles-pp.btrfs_copy_from_user
      0.07 ± 17%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.rwsem_spin_on_owner
      0.21 ±  3%      -0.0        0.17 ±  2%  perf-profile.children.cycles-pp.selinux_file_permission
      0.21 ±  6%      -0.0        0.17 ±  4%  perf-profile.children.cycles-pp.set_state_bits
      0.36 ±  2%      -0.0        0.33 ±  3%  perf-profile.children.cycles-pp.copyout
      0.10 ±  8%      -0.0        0.07 ±  6%  perf-profile.children.cycles-pp.start_transaction
      0.09 ± 18%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.btrfs_commit_inode_delayed_inode
      0.18 ±  4%      -0.0        0.15 ± 11%  perf-profile.children.cycles-pp.__alloc_pages_nodemask
      0.17 ±  5%      -0.0        0.14 ±  5%  perf-profile.children.cycles-pp.iov_iter_copy_from_user_atomic
      0.12 ± 10%      -0.0        0.09 ± 12%  perf-profile.children.cycles-pp.free_extent_state
      0.07 ±  6%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.btrfs_delalloc_release_metadata
      0.09 ± 10%      -0.0        0.07 ±  7%  perf-profile.children.cycles-pp.btrfs_block_rsv_add
      0.10 ±  4%      -0.0        0.08 ±  6%  perf-profile.children.cycles-pp.truncate_cleanup_page
      0.18 ±  7%      -0.0        0.15 ±  7%  perf-profile.children.cycles-pp.btrfs_calculate_inode_block_rsv_size
      0.06 ±  7%      -0.0        0.04 ± 57%  perf-profile.children.cycles-pp.btrfs_unlink_inode
      0.14 ±  3%      -0.0        0.11 ±  7%  perf-profile.children.cycles-pp.copyin
      0.18 ±  2%      -0.0        0.15 ±  5%  perf-profile.children.cycles-pp.__set_page_dirty_nobuffers
      0.08 ± 10%      -0.0        0.06 ± 14%  perf-profile.children.cycles-pp.kfree
      0.12 ±  3%      -0.0        0.10 ±  8%  perf-profile.children.cycles-pp.release_pages
      0.09 ±  9%      -0.0        0.07 ± 10%  perf-profile.children.cycles-pp.xas_store
      0.08 ±  6%      -0.0        0.06 ±  7%  perf-profile.children.cycles-pp.split_state
      0.07            -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.current_time
      0.12 ±  4%      -0.0        0.10 ±  7%  perf-profile.children.cycles-pp.__pagevec_release
      0.07 ±  7%      -0.0        0.05 ±  8%  perf-profile.children.cycles-pp.try_merge_map
      0.10            -0.0        0.09 ±  4%  perf-profile.children.cycles-pp.btrfs_add_extent_mapping
      0.09 ± 24%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.uart_console_write
      0.09 ± 24%      +0.0        0.12 ± 13%  perf-profile.children.cycles-pp.serial8250_console_putchar
      0.16 ±  7%      +0.1        0.28 ±  3%  perf-profile.children.cycles-pp.btrfs_free_reserved_data_space_noquota
     97.42            +0.6       98.04        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     97.35            +0.6       97.97        perf-profile.children.cycles-pp.do_syscall_64
     94.98            +1.2       96.13        perf-profile.children.cycles-pp.write
     94.38            +1.2       95.61        perf-profile.children.cycles-pp.ksys_write
     94.30            +1.2       95.54        perf-profile.children.cycles-pp.vfs_write
     93.96            +1.3       95.24        perf-profile.children.cycles-pp.new_sync_write
     93.91            +1.3       95.19        perf-profile.children.cycles-pp.btrfs_file_write_iter
     93.60            +1.3       94.93        perf-profile.children.cycles-pp.btrfs_buffered_write
     85.03            +2.0       87.03        perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
     86.13            +2.3       88.43        perf-profile.children.cycles-pp._raw_spin_lock
     42.75           +12.7       55.42        perf-profile.children.cycles-pp.space_info_add_old_bytes
     43.08           +12.8       55.90        perf-profile.children.cycles-pp.block_rsv_release_bytes
     42.99           +12.9       55.92        perf-profile.children.cycles-pp.btrfs_inode_rsv_release
      3.76           +22.7       26.47        perf-profile.children.cycles-pp.btrfs_dirty_pages
      1.36 ±  3%     +23.2       24.57        perf-profile.children.cycles-pp.__clear_extent_bit
      0.89 ±  2%     +23.2       24.14        perf-profile.children.cycles-pp.clear_extent_bit
      0.94 ±  2%     +23.3       24.21        perf-profile.children.cycles-pp.clear_state_bit
      0.67           +23.3       23.95        perf-profile.children.cycles-pp.btrfs_clear_delalloc_extent
      0.82 ±  4%      -0.3        0.47 ±  9%  perf-profile.self.cycles-pp.intel_idle
      0.60 ±  2%      -0.1        0.49 ±  3%  perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.62            -0.1        0.52        perf-profile.self.cycles-pp.syscall_return_via_sysret
      0.30 ± 13%      -0.1        0.21 ±  7%  perf-profile.self.cycles-pp.reserve_metadata_bytes
      0.48            -0.1        0.42 ±  3%  perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.51 ±  5%      -0.1        0.45 ±  3%  perf-profile.self.cycles-pp.can_overcommit
      0.23 ±  7%      -0.0        0.18 ±  8%  perf-profile.self.cycles-pp.__set_extent_bit
      0.23 ±  8%      -0.0        0.19 ±  6%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.08 ± 12%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.btrfs_get_extent
      0.20 ±  6%      -0.0        0.17 ±  6%  perf-profile.self.cycles-pp.memset_erms
      0.18 ± 10%      -0.0        0.14 ±  5%  perf-profile.self.cycles-pp.do_syscall_64
      0.18 ±  6%      -0.0        0.15 ±  2%  perf-profile.self.cycles-pp.___might_sleep
      0.07 ± 17%      -0.0        0.04 ± 57%  perf-profile.self.cycles-pp.rwsem_spin_on_owner
      0.14 ±  6%      -0.0        0.11        perf-profile.self.cycles-pp.btrfs_calculate_inode_block_rsv_size
      0.11 ±  8%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.free_extent_state
      0.16 ±  6%      -0.0        0.13 ±  5%  perf-profile.self.cycles-pp.find_get_entry
      0.10 ± 12%      -0.0        0.08 ±  8%  perf-profile.self.cycles-pp.xas_load
      0.14 ±  3%      -0.0        0.11 ±  4%  perf-profile.self.cycles-pp.selinux_file_permission
      0.14            -0.0        0.12 ±  8%  perf-profile.self.cycles-pp.btrfs_file_write_iter
      0.15 ±  5%      -0.0        0.13 ±  3%  perf-profile.self.cycles-pp.btrfs_buffered_write
      0.11 ±  4%      -0.0        0.09 ±  7%  perf-profile.self.cycles-pp.generic_file_read_iter
      0.10 ±  4%      -0.0        0.09 ±  4%  perf-profile.self.cycles-pp.btrfs_dirty_pages
      0.06            -0.0        0.05        perf-profile.self.cycles-pp.rb_next
      0.16 ±  2%      +0.0        0.18 ±  2%  perf-profile.self.cycles-pp.btrfs_drop_pages
      0.05 ±  8%      +0.0        0.09 ± 20%  perf-profile.self.cycles-pp.btrfs_delalloc_reserve_metadata
      0.00            +0.1        0.12 ±  8%  perf-profile.self.cycles-pp.btrfs_free_reserved_data_space_noquota
     84.44            +2.1       86.53        perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath


                                                                                
                                  aim7.jobs-per-min                             
                                                                                
  50000 +-+-----------------------------------------------------------------+   
  45000 +-+  .+..+..+..+..+..+..+..+..+..+.+..        .+..     .+           |   
        |  +.                                 +..+..+.    +..+. :     +..+..|   
  40000 O-+   O     O  O  O  O  O  O  O    O  O  O  O  O     O  O  O  O  O  |   
  35000 +-+O     O                                               :    :     |   
        |                                                        :   :      |   
  30000 +-+                                                      :   :      |   
  25000 +-+                                                      :   :      |   
  20000 +-+                                                       :  :      |   
        |                                                         : :       |   
  15000 +-+                                                       : :       |   
  10000 +-+                                                       : :       |   
        |                                                          ::       |   
   5000 +-+                                                        :        |   
      0 +-+------------------------------O----------------O-----------------+   
                                                                                
                                                                                                                                                                
                                aim7.time.system_time                           
                                                                                
  10000 +-+-----------------------------------------------------------------+   
   9000 O-+O     O           O     O       O                                |   
        |     O     O  O  O     O     O       O  O  O  O     O  O  O  O  O  |   
   8000 +-++..                                  .+..+..  .+..+..      +..+..|   
   7000 +-+   +..+..+..+..+..+..+..+..+..+.+..+.       +.       +     :     |   
        |                                                       :    :      |   
   6000 +-+                                                      :   :      |   
   5000 +-+                                                      :   :      |   
   4000 +-+                                                      :   :      |   
        |                                                         : :       |   
   3000 +-+                                                       : :       |   
   2000 +-+                                                       : :       |   
        |                                                         : :       |   
   1000 +-+                                                        :        |   
      0 +-+------------------------------O----------------O-----------------+   
                                                                                
                                                                                                                                                                
                              aim7.time.elapsed_time                            
                                                                                
  250 +-+------O------------------------------------------------------------+   
      O  O  O     O  O  O  O  O  O  O      O  O  O  O  O     O  O  O  O  O  |   
      |                                         .+..+..               +..+..|   
  200 +-++..+..+..+..+..+..+..+..+..+..+...+..+.       +..+..+..+     :     |   
      |                                                         :     :     |   
      |                                                          :   :      |   
  150 +-+                                                        :   :      |   
      |                                                          :   :      |   
  100 +-+                                                        :   :      |   
      |                                                           : :       |   
      |                                                           : :       |   
   50 +-+                                                         : :       |   
      |                                                           : :       |   
      |                                                            :        |   
    0 +-+------------------------------O------------------O-----------------+   
                                                                                
                                                                                                                                                                
                            aim7.time.elapsed_time.max                          
                                                                                
  250 +-+------O------------------------------------------------------------+   
      O  O  O     O  O  O  O  O  O  O      O  O  O  O  O     O  O  O  O  O  |   
      |                                         .+..+..               +..+..|   
  200 +-++..+..+..+..+..+..+..+..+..+..+...+..+.       +..+..+..+     :     |   
      |                                                         :     :     |   
      |                                                          :   :      |   
  150 +-+                                                        :   :      |   
      |                                                          :   :      |   
  100 +-+                                                        :   :      |   
      |                                                           : :       |   
      |                                                           : :       |   
   50 +-+                                                         : :       |   
      |                                                           : :       |   
      |                                                            :        |   
    0 +-+------------------------------O------------------O-----------------+   
                                                                                
                                                                                                                                                                
                       aim7.time.involuntary_context_switches                   
                                                                                
   1e+06 +-+----------------------------------------------------------------+   
  900000 O-+O     O                                                         |   
         |     O     O  O O  O  O  O  O     O  O     O          O     O  O  |   
  800000 +-+                                      O     O    O     O        |   
  700000 +-+                                                                |   
         |                                       .+..                 +..   |   
  600000 +-++..+..+..+..+.+..+..+..+..+..+..+..+.    +..+..+.+..+     :  +..|   
  500000 +-+                                                    :    :      |   
  400000 +-+                                                     :   :      |   
         |                                                       :   :      |   
  300000 +-+                                                      : :       |   
  200000 +-+                                                      : :       |   
         |                                                        : :       |   
  100000 +-+                                                       :        |   
       0 +-+-----------------------------O-----------------O----------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.1.0-rc7-00219-gc8eaeac7" of type "text/plain" (188896 bytes)

View attachment "job-script" of type "text/plain" (8009 bytes)

View attachment "job.yaml" of type "text/plain" (5695 bytes)

View attachment "reproduce" of type "text/plain" (1035 bytes)

Powered by blists - more mailing lists