lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Date:   Tue, 31 Mar 2020 17:01:45 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Omar Sandoval <osandov@...com>
Cc:     David Sterba <dsterba@...e.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [btrfs] 1eb52c8bd8: fio.write_bw_MBps -46.1% regression

Greeting,

FYI, we noticed a -46.1% regression of fio.write_bw_MBps due to commit:


commit: 1eb52c8bd8d6b056caa06737242830f03777da32 ("btrfs: get rid of one layer of bios in direct I/O")
https://git.kernel.org/cgit/linux/kernel/git/next/linux-next.git master

in testcase: fio-basic
on test machine: 160 threads Intel(R) Xeon(R) CPU E7-8890 v4 @ 2.20GHz with 256G memory
with following parameters:

	runtime: 300s
	disk: 1SSD
	fs: btrfs
	nr_task: 100%
	test_size: 128G
	rw: write
	bs: 4k
	ioengine: sync
	direct: direct
	cpufreq_governor: performance
	ucode: 0xb000038

test-description: Fio is a tool that will spawn a number of threads or processes doing a particular type of I/O action as specified by the user.
test-url: https://github.com/axboe/fio



If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
bs/compiler/cpufreq_governor/direct/disk/fs/ioengine/kconfig/nr_task/rootfs/runtime/rw/tbox_group/test_size/testcase/ucode:
  4k/gcc-7/performance/direct/1SSD/btrfs/sync/x86_64-rhel-7.6/100%/debian-x86_64-20191114.cgz/300s/write/lkp-bdw-ex2/128G/fio-basic/0xb000038

commit: 
  ba9d3fc7cb ("btrfs: put direct I/O checksums in btrfs_dio_private instead of bio")
  1eb52c8bd8 ("btrfs: get rid of one layer of bios in direct I/O")

ba9d3fc7cb61296b 1eb52c8bd8d6b056caa06737242 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
           :4           25%           1:4     dmesg.WARNING:at#for_ip_interrupt_entry/0x
         %stddev     %change         %stddev
             \          |                \  
      0.57 ± 25%      -0.2        0.35 ±  3%  fio.latency_1000us%
      0.13 ±  9%      -0.1        0.06 ±  5%  fio.latency_100us%
      0.18 ±  3%     +24.3       24.45 ± 17%  fio.latency_10ms%
      0.34 ±  5%      -0.1        0.23 ±  6%  fio.latency_250us%
     93.88           -39.1       54.78 ±  9%  fio.latency_2ms%
      3.95 ±  5%     +15.1       19.03 ± 12%  fio.latency_4ms%
      0.52 ±  4%      +0.1        0.58 ±  4%  fio.latency_500us%
      0.08 ± 20%      -0.1        0.02 ± 11%  fio.latency_50us%
      0.35 ±  3%      +0.1        0.48 ±  2%  fio.latency_750us%
 2.681e+08           -45.3%  1.466e+08 ±  5%  fio.time.file_system_outputs
     23020           -47.8%      12023 ±  6%  fio.time.involuntary_context_switches
    159091            -8.4%     145794        fio.time.minor_page_faults
      2351           -39.8%       1415 ±  5%  fio.time.percent_of_cpu_this_job_got
      6836           -39.0%       4170 ±  5%  fio.time.system_time
    141.07 ±  2%     -37.2%      88.62 ±  6%  fio.time.user_time
  74688391           -36.0%   47773188 ±  3%  fio.time.voluntary_context_switches
  33510952           -45.3%   18323895 ±  5%  fio.workload
    442.52           -46.1%     238.59 ±  5%  fio.write_bw_MBps
   1757184          +196.5%    5210112        fio.write_clat_90%_us
   1945600          +187.2%    5586944        fio.write_clat_95%_us
   2441216          +157.7%    6291456        fio.write_clat_99%_us
   1409946           +86.2%    2625138 ±  5%  fio.write_clat_mean_us
    371509 ±  3%    +331.6%    1603575 ±  4%  fio.write_clat_stddev
    113285           -46.1%      61078 ±  5%  fio.write_iops
      0.76 ±  6%      -0.5        0.29 ±  6%  mpstat.cpu.all.iowait%
     28.05            -5.7       22.39 ±  2%  mpstat.cpu.all.sys%
      0.26            -0.1        0.17 ±  5%  mpstat.cpu.all.usr%
     56771 ±  4%      -9.3%      51506        softirqs.CPU100.SCHED
     55528 ±  3%      -6.8%      51743 ±  4%  softirqs.CPU148.SCHED
    113291 ±  2%     +24.7%     141291 ± 15%  softirqs.CPU49.TIMER
 1.562e+10 ± 20%     +58.3%  2.472e+10 ± 18%  cpuidle.C6.time
  35854532 ±  5%     +21.6%   43598798 ±  5%  cpuidle.C6.usage
  44587503 ±  5%     -29.7%   31323017 ±  3%  cpuidle.POLL.time
   9963711 ±  5%     -29.0%    7069945 ±  3%  cpuidle.POLL.usage
     31160           -49.6%      15709 ± 70%  numa-numastat.node0.other_node
    573019 ± 20%     -46.9%     304525 ± 16%  numa-numastat.node2.local_node
    587387 ± 20%     -43.6%     331125 ± 16%  numa-numastat.node2.numa_hit
    563047 ± 14%     -47.7%     294353 ±  6%  numa-numastat.node3.local_node
    588001 ± 13%     -45.4%     321048 ±  4%  numa-numastat.node3.numa_hit
     70.25            +9.6%      77.00        vmstat.cpu.id
    443711           -45.4%     242237 ±  5%  vmstat.io.bo
      3.75 ± 11%     -40.0%       2.25 ± 19%  vmstat.procs.b
     54.00           -21.3%      42.50 ±  2%  vmstat.procs.r
    501610           -11.1%     445923        vmstat.system.in
     71.10            +8.7%      77.25        iostat.cpu.idle
     27.88           -20.1%      22.29 ±  2%  iostat.cpu.system
      1.15 ±  5%     -68.8%       0.36 ±  7%  iostat.nvme0n1.avgqu-sz
      1.08 ± 80%     -52.1%       0.52 ±  5%  iostat.nvme0n1.await.max
    110528           -45.5%      60285 ±  5%  iostat.nvme0n1.w/s
      1.08 ± 80%     -52.1%       0.52 ±  5%  iostat.nvme0n1.w_await.max
    442960           -45.3%     242228 ±  5%  iostat.nvme0n1.wkB/s
     96186 ±  2%     +74.4%     167794 ± 15%  meminfo.Active(file)
  10471936 ±  4%     +12.4%   11775488 ±  7%  meminfo.DirectMap2M
     13601           +40.4%      19096 ±  4%  meminfo.Dirty
   5884172           -12.5%    5148653 ±  2%  meminfo.Memused
   2826160           -27.3%    2053832 ±  3%  meminfo.SUnreclaim
   2936311           -26.3%    2164032 ±  3%  meminfo.Slab
     27663           -25.2%      20686 ±  3%  meminfo.max_used_kB
      7889           +24.8%       9844 ±  2%  slabinfo.blkdev_ioc.active_objs
      7903           +24.6%       9845 ±  2%  slabinfo.blkdev_ioc.num_objs
  16616823           -33.2%   11099554 ±  4%  slabinfo.btrfs_extent_map.active_objs
    296728           -33.2%     198206 ±  4%  slabinfo.btrfs_extent_map.active_slabs
  16616833           -33.2%   11099554 ±  4%  slabinfo.btrfs_extent_map.num_objs
    296728           -33.2%     198206 ±  4%  slabinfo.btrfs_extent_map.num_slabs
     13880          +172.9%      37881 ±  3%  slabinfo.kmalloc-128.active_objs
    217.25          +172.3%     591.50 ±  3%  slabinfo.kmalloc-128.active_slabs
     13947          +171.6%      37881 ±  3%  slabinfo.kmalloc-128.num_objs
    217.25          +172.3%     591.50 ±  3%  slabinfo.kmalloc-128.num_slabs
     12405           +24.7%      15471 ±  7%  slabinfo.numa_policy.active_objs
     12405           +24.7%      15471 ±  7%  slabinfo.numa_policy.num_objs
    101066            -8.6%      92337 ±  5%  proc-vmstat.nr_active_anon
     24037 ±  2%     +74.7%      41999 ± 14%  proc-vmstat.nr_active_file
     67593           +45.1%      98064 ± 12%  proc-vmstat.nr_dirtied
      3423           +39.7%       4783 ±  4%  proc-vmstat.nr_dirty
     29713            +1.5%      30172        proc-vmstat.nr_kernel_stack
    134152            -6.0%     126155 ±  4%  proc-vmstat.nr_shmem
    706371           -27.3%     513804 ±  3%  proc-vmstat.nr_slab_unreclaimable
     63441 ±  4%     +35.9%      86242 ± 13%  proc-vmstat.nr_written
    101066            -8.6%      92337 ±  5%  proc-vmstat.nr_zone_active_anon
     24037 ±  2%     +74.7%      41999 ± 14%  proc-vmstat.nr_zone_active_file
      3421           +39.9%       4785 ±  4%  proc-vmstat.nr_zone_write_pending
     39946 ± 19%     -36.8%      25244 ±  9%  proc-vmstat.numa_hint_faults
     24644 ± 17%     -43.6%      13893 ±  5%  proc-vmstat.numa_hint_faults_local
   2258358           -30.4%    1571696 ±  2%  proc-vmstat.numa_hit
   2164392           -31.7%    1477659 ±  2%  proc-vmstat.numa_local
   3576883 ±  2%     -42.7%    2047837 ±  3%  proc-vmstat.pgalloc_normal
   2141630 ±  4%     -44.4%    1190106 ±  2%  proc-vmstat.pgfree
 1.342e+08           -45.1%   73640384 ±  5%  proc-vmstat.pgpgout
     24028 ±  5%     +76.3%      42370 ± 25%  numa-meminfo.node0.Active(file)
      3436           +37.4%       4722 ±  9%  numa-meminfo.node0.Dirty
    315977 ±  4%     +41.5%     447077 ± 42%  numa-meminfo.node0.FilePages
    714746 ±  2%     -27.3%     519800 ±  3%  numa-meminfo.node0.SUnreclaim
    737117 ±  3%     -26.0%     545687 ±  4%  numa-meminfo.node0.Slab
     24334 ±  4%     +72.9%      42072 ± 13%  numa-meminfo.node1.Active(file)
      3360 ±  3%     +40.7%       4727 ±  8%  numa-meminfo.node1.Dirty
     30338 ± 52%     -92.1%       2398 ± 22%  numa-meminfo.node1.Inactive(file)
    707990 ±  3%     -27.5%     513144 ±  3%  numa-meminfo.node1.SUnreclaim
    730781 ±  2%     -25.4%     545123        numa-meminfo.node1.Slab
    261518            +9.7%     286771 ±  2%  numa-meminfo.node1.Unevictable
     23970 ±  4%     +75.7%      42114 ± 11%  numa-meminfo.node2.Active(file)
      3414 ±  3%     +45.9%       4980 ± 10%  numa-meminfo.node2.Dirty
      2089 ±  4%    +464.7%      11797 ±135%  numa-meminfo.node2.Inactive(file)
    684851           -24.9%     514538 ±  6%  numa-meminfo.node2.SUnreclaim
    718787           -24.6%     541680 ±  6%  numa-meminfo.node2.Slab
     23770 ±  3%     +72.9%      41104 ± 14%  numa-meminfo.node3.Active(file)
      3419 ±  2%     +37.0%       4686 ±  5%  numa-meminfo.node3.Dirty
    621874 ± 33%     -38.5%     382252 ±  8%  numa-meminfo.node3.FilePages
    211446 ± 95%     -97.2%       5918 ± 70%  numa-meminfo.node3.Inactive(anon)
      8189 ± 12%     -28.9%       5826 ±  2%  numa-meminfo.node3.KernelStack
    210498 ± 95%     -96.2%       8099 ± 24%  numa-meminfo.node3.Mapped
   1634119 ± 14%     -28.5%    1167711 ±  8%  numa-meminfo.node3.MemUsed
      8986 ± 65%     -93.8%     558.00 ± 42%  numa-meminfo.node3.PageTables
    716951 ±  3%     -29.5%     505575 ±  3%  numa-meminfo.node3.SUnreclaim
    307384 ± 66%     -78.1%      67417 ± 27%  numa-meminfo.node3.Shmem
    748007 ±  4%     -29.0%     530773 ±  3%  numa-meminfo.node3.Slab
    236951 ±  4%     +74.8%     414267 ± 20%  sched_debug.cfs_rq:/.load.avg
   1051986 ± 14%   +2775.4%   30248961 ± 28%  sched_debug.cfs_rq:/.load.max
    382024 ±  5%    +599.5%    2672216 ± 29%  sched_debug.cfs_rq:/.load.stddev
    266.10 ± 18%     +55.7%     414.41 ±  7%  sched_debug.cfs_rq:/.load_avg.avg
      0.24 ±  3%     -14.3%       0.21 ±  5%  sched_debug.cfs_rq:/.nr_running.avg
      0.02 ±173%  +13188.9%       2.46 ± 52%  sched_debug.cfs_rq:/.removed.load_avg.avg
      3.54 ±173%  +10209.6%     365.13 ± 65%  sched_debug.cfs_rq:/.removed.load_avg.max
      0.26 ±173%  +11413.7%      29.43 ± 58%  sched_debug.cfs_rq:/.removed.load_avg.stddev
      0.86 ±173%   +9170.6%      79.74 ± 47%  sched_debug.cfs_rq:/.removed.runnable_sum.avg
    164.29 ±173%   +6197.3%      10345 ± 28%  sched_debug.cfs_rq:/.removed.runnable_sum.max
     11.86 ±173%   +7400.3%     889.28 ± 36%  sched_debug.cfs_rq:/.removed.runnable_sum.stddev
      0.02 ±173%   +5513.9%       0.94 ± 67%  sched_debug.cfs_rq:/.removed.util_avg.avg
      3.21 ±173%   +3881.6%     127.74 ± 56%  sched_debug.cfs_rq:/.removed.util_avg.max
      0.23 ±173%   +4540.6%      10.74 ± 59%  sched_debug.cfs_rq:/.removed.util_avg.stddev
    870.28 ±  9%    +135.2%       2047 ± 25%  sched_debug.cfs_rq:/.runnable_load_avg.max
    237569 ±  4%     +74.0%     413454 ± 20%  sched_debug.cfs_rq:/.runnable_weight.avg
   1050573 ± 14%   +2779.2%   30248310 ± 28%  sched_debug.cfs_rq:/.runnable_weight.max
    383684 ±  5%    +596.4%    2671965 ± 29%  sched_debug.cfs_rq:/.runnable_weight.stddev
    150.81 ±  7%     -15.6%     127.27 ±  5%  sched_debug.cfs_rq:/.util_est_enqueued.avg
   1448440 ± 18%     -21.5%    1137742 ± 11%  sched_debug.cpu.avg_idle.max
    771.64 ±  6%     -14.8%     657.35 ±  6%  sched_debug.cpu.curr->pid.avg
      0.00 ±  9%     -15.6%       0.00 ±  6%  sched_debug.cpu.next_balance.stddev
      0.24 ±  6%     -14.2%       0.20 ±  4%  sched_debug.cpu.nr_running.avg
      0.55 ±  9%     +16.3%       0.64        sched_debug.cpu.nr_uninterruptible.avg
    -65.21           +67.9%    -109.47        sched_debug.cpu.nr_uninterruptible.min
     28.57 ±  7%     +30.6%      37.32 ±  8%  sched_debug.cpu.nr_uninterruptible.stddev
     81977 ± 23%     +95.3%     160125 ± 10%  sched_debug.cpu.ttwu_count.min
     15904 ± 19%     -72.6%       4359 ± 13%  sched_debug.cpu.ttwu_local.avg
    108359 ± 19%     -68.6%      34043 ± 18%  sched_debug.cpu.ttwu_local.max
     35141 ± 19%     -75.0%       8783 ± 13%  sched_debug.cpu.ttwu_local.stddev
      6014 ±  5%     +76.3%      10603 ± 25%  numa-vmstat.node0.nr_active_file
      7304 ± 13%    +128.6%      16699 ± 41%  numa-vmstat.node0.nr_dirtied
    856.00           +38.1%       1182 ±  8%  numa-vmstat.node0.nr_dirty
     79002 ±  4%     +41.5%     111781 ± 42%  numa-vmstat.node0.nr_file_pages
    178849 ±  2%     -27.3%     130010 ±  3%  numa-vmstat.node0.nr_slab_unreclaimable
      6442 ± 15%    +130.8%      14871 ± 44%  numa-vmstat.node0.nr_written
      6014 ±  5%     +76.3%      10603 ± 25%  numa-vmstat.node0.nr_zone_active_file
    856.25           +38.1%       1182 ±  9%  numa-vmstat.node0.nr_zone_write_pending
      6091 ±  4%     +72.8%      10527 ± 13%  numa-vmstat.node1.nr_active_file
    837.50 ±  2%     +41.3%       1183 ±  9%  numa-vmstat.node1.nr_dirty
      7582 ± 52%     -92.1%     600.00 ± 22%  numa-vmstat.node1.nr_inactive_file
     27196 ±162%     +97.9%      53815 ± 94%  numa-vmstat.node1.nr_mapped
    177163 ±  3%     -27.6%     128347 ±  3%  numa-vmstat.node1.nr_slab_unreclaimable
     65379            +9.7%      71692 ±  2%  numa-vmstat.node1.nr_unevictable
      6091 ±  4%     +72.8%      10527 ± 13%  numa-vmstat.node1.nr_zone_active_file
      7582 ± 52%     -92.1%     600.00 ± 22%  numa-vmstat.node1.nr_zone_inactive_file
     65379            +9.7%      71692 ±  2%  numa-vmstat.node1.nr_zone_unevictable
    837.25 ±  2%     +41.4%       1184 ±  9%  numa-vmstat.node1.nr_zone_write_pending
      5999 ±  4%     +75.6%      10538 ± 11%  numa-vmstat.node2.nr_active_file
      6769 ±  3%    +105.8%      13934 ± 28%  numa-vmstat.node2.nr_dirtied
    850.25 ±  3%     +46.5%       1245 ± 10%  numa-vmstat.node2.nr_dirty
    522.25 ±  4%    +464.6%       2948 ±134%  numa-vmstat.node2.nr_inactive_file
    171381           -24.9%     128699 ±  6%  numa-vmstat.node2.nr_slab_unreclaimable
      5917 ±  3%    +104.3%      12088 ± 32%  numa-vmstat.node2.nr_written
      5999 ±  4%     +75.6%      10538 ± 11%  numa-vmstat.node2.nr_zone_active_file
    522.25 ±  4%    +464.6%       2948 ±134%  numa-vmstat.node2.nr_zone_inactive_file
    849.75 ±  3%     +46.7%       1246 ± 10%  numa-vmstat.node2.nr_zone_write_pending
      5949 ±  3%     +72.9%      10285 ± 14%  numa-vmstat.node3.nr_active_file
    851.00 ±  2%     +37.7%       1172 ±  5%  numa-vmstat.node3.nr_dirty
    155526 ± 33%     -38.5%      95575 ±  8%  numa-vmstat.node3.nr_file_pages
     52896 ± 95%     -97.2%       1493 ± 70%  numa-vmstat.node3.nr_inactive_anon
      8188 ± 12%     -28.9%       5825 ±  2%  numa-vmstat.node3.nr_kernel_stack
     52666 ± 94%     -96.0%       2086 ± 24%  numa-vmstat.node3.nr_mapped
      2244 ± 65%     -93.8%     139.00 ± 42%  numa-vmstat.node3.nr_page_table_pages
     76896 ± 66%     -78.1%      16857 ± 27%  numa-vmstat.node3.nr_shmem
    179393 ±  3%     -29.5%     126463 ±  3%  numa-vmstat.node3.nr_slab_unreclaimable
      5949 ±  3%     +72.9%      10285 ± 14%  numa-vmstat.node3.nr_zone_active_file
     52896 ± 95%     -97.2%       1493 ± 70%  numa-vmstat.node3.nr_zone_inactive_anon
    851.50 ±  2%     +37.7%       1172 ±  6%  numa-vmstat.node3.nr_zone_write_pending
    722738 ± 17%     -32.7%     486464 ±  9%  numa-vmstat.node3.numa_hit
    611153 ± 21%     -39.0%     372997 ± 13%  numa-vmstat.node3.numa_local
     16.63           -15.3%      14.08        perf-stat.i.MPKI
 1.078e+10           -18.9%  8.744e+09 ±  2%  perf-stat.i.branch-instructions
      1.00 ±  6%      +0.1        1.12 ±  2%  perf-stat.i.branch-miss-rate%
      7.06 ±  3%      +1.9        8.92        perf-stat.i.cache-miss-rate%
  53207426 ±  2%     -11.8%   46918906 ±  2%  perf-stat.i.cache-misses
 7.476e+08 ±  3%     -29.0%  5.305e+08 ±  3%  perf-stat.i.cache-references
 1.524e+11           -19.1%  1.233e+11 ±  2%  perf-stat.i.cpu-cycles
      3821 ±  2%     +16.8%       4464 ±  8%  perf-stat.i.cpu-migrations
      3170 ± 16%     -15.2%       2690 ±  3%  perf-stat.i.cycles-between-cache-misses
      0.21 ±  8%      +0.0        0.25 ±  6%  perf-stat.i.dTLB-load-miss-rate%
 1.167e+10           -19.1%  9.447e+09 ±  2%  perf-stat.i.dTLB-loads
 2.345e+09           -10.6%  2.096e+09 ±  2%  perf-stat.i.dTLB-stores
     28.83 ±  7%      +3.2       31.99 ±  3%  perf-stat.i.iTLB-load-miss-rate%
   8884989 ±  7%     -21.4%    6986487 ±  6%  perf-stat.i.iTLB-load-misses
  22097174 ±  2%     -33.3%   14729666 ±  2%  perf-stat.i.iTLB-loads
 4.576e+10           -18.6%  3.722e+10 ±  2%  perf-stat.i.instructions
      0.30            +0.8%       0.30        perf-stat.i.ipc
      3987            -2.5%       3887        perf-stat.i.minor-faults
     95.84            +1.3       97.15        perf-stat.i.node-load-miss-rate%
  24251167           -12.8%   21155958 ±  2%  perf-stat.i.node-load-misses
   1041611 ±  2%     -37.6%     649903 ±  4%  perf-stat.i.node-loads
     76.92            -1.5       75.42        perf-stat.i.node-store-miss-rate%
  11698329           -17.4%    9659586 ±  2%  perf-stat.i.node-store-misses
   3420696            -8.6%    3125060 ±  2%  perf-stat.i.node-stores
      3987            -2.5%       3887        perf-stat.i.page-faults
     16.34 ±  2%     -12.7%      14.26        perf-stat.overall.MPKI
      0.93 ±  3%      +0.2        1.10 ±  2%  perf-stat.overall.branch-miss-rate%
      7.12 ±  3%      +1.7        8.85        perf-stat.overall.cache-miss-rate%
      2865            -8.3%       2626 ±  2%  perf-stat.overall.cycles-between-cache-misses
      0.21 ±  9%      +0.1        0.26 ±  6%  perf-stat.overall.dTLB-load-miss-rate%
     28.66 ±  7%      +3.5       32.15 ±  4%  perf-stat.overall.iTLB-load-miss-rate%
     95.88            +1.1       97.02        perf-stat.overall.node-load-miss-rate%
     77.37            -1.8       75.55        perf-stat.overall.node-store-miss-rate%
    408903           +49.6%     611741 ±  3%  perf-stat.overall.path-length
 1.074e+10           -18.8%  8.719e+09 ±  2%  perf-stat.ps.branch-instructions
  53036173 ±  2%     -11.7%   46815332 ±  2%  perf-stat.ps.cache-misses
 7.451e+08 ±  3%     -29.0%  5.294e+08 ±  3%  perf-stat.ps.cache-references
 1.519e+11           -19.1%  1.229e+11 ±  2%  perf-stat.ps.cpu-cycles
      3809 ±  2%     +17.2%       4463 ±  8%  perf-stat.ps.cpu-migrations
 1.163e+10           -19.0%  9.421e+09 ±  2%  perf-stat.ps.dTLB-loads
 2.337e+09           -10.5%  2.091e+09 ±  2%  perf-stat.ps.dTLB-stores
   8853815 ±  7%     -21.3%    6971170 ±  6%  perf-stat.ps.iTLB-load-misses
  22023628 ±  2%     -33.3%   14694997 ±  2%  perf-stat.ps.iTLB-loads
 4.561e+10           -18.6%  3.712e+10 ±  2%  perf-stat.ps.instructions
      3973            -2.4%       3877        perf-stat.ps.minor-faults
  24172195           -12.7%   21107094 ±  2%  perf-stat.ps.node-load-misses
   1038351 ±  2%     -37.5%     648941 ±  4%  perf-stat.ps.node-loads
  11658597           -17.4%    9635694 ±  2%  perf-stat.ps.node-store-misses
   3409902            -8.6%    3117668 ±  2%  perf-stat.ps.node-stores
      3973            -2.4%       3877        perf-stat.ps.page-faults
  1.37e+13           -18.3%  1.119e+13 ±  2%  perf-stat.total.instructions
      3869 ± 17%     -43.8%       2174 ± 31%  interrupts.CPU103.NMI:Non-maskable_interrupts
      3869 ± 17%     -43.8%       2174 ± 31%  interrupts.CPU103.PMI:Performance_monitoring_interrupts
     20628 ± 15%     -26.7%      15129 ± 19%  interrupts.CPU106.RES:Rescheduling_interrupts
      3940 ± 26%     -50.7%       1943 ± 54%  interrupts.CPU109.NMI:Non-maskable_interrupts
      3940 ± 26%     -50.7%       1943 ± 54%  interrupts.CPU109.PMI:Performance_monitoring_interrupts
     20852 ± 19%     -28.7%      14858 ± 26%  interrupts.CPU113.RES:Rescheduling_interrupts
     94.00 ± 49%    +105.1%     192.75 ± 41%  interrupts.CPU115.TLB:TLB_shootdowns
      3113 ± 12%     -39.6%       1881 ± 25%  interrupts.CPU117.NMI:Non-maskable_interrupts
      3113 ± 12%     -39.6%       1881 ± 25%  interrupts.CPU117.PMI:Performance_monitoring_interrupts
     15318 ± 33%     +37.0%      20988 ± 22%  interrupts.CPU117.RES:Rescheduling_interrupts
     22815 ± 19%     -34.4%      14976 ± 37%  interrupts.CPU122.RES:Rescheduling_interrupts
      3803 ± 18%     -41.6%       2221 ± 43%  interrupts.CPU124.NMI:Non-maskable_interrupts
      3803 ± 18%     -41.6%       2221 ± 43%  interrupts.CPU124.PMI:Performance_monitoring_interrupts
    663.25 ± 50%     +36.0%     902.00 ±  4%  interrupts.CPU125.CAL:Function_call_interrupts
      2736 ± 35%     -50.9%       1343 ± 48%  interrupts.CPU125.NMI:Non-maskable_interrupts
      2736 ± 35%     -50.9%       1343 ± 48%  interrupts.CPU125.PMI:Performance_monitoring_interrupts
    207.50 ± 24%     +30.8%     271.50 ± 12%  interrupts.CPU126.TLB:TLB_shootdowns
     15286 ± 29%     +37.8%      21061 ± 20%  interrupts.CPU129.RES:Rescheduling_interrupts
    192.00 ± 27%     +50.7%     289.25 ± 14%  interrupts.CPU129.TLB:TLB_shootdowns
      3027 ± 18%     -45.7%       1645 ± 51%  interrupts.CPU131.NMI:Non-maskable_interrupts
      3027 ± 18%     -45.7%       1645 ± 51%  interrupts.CPU131.PMI:Performance_monitoring_interrupts
     14710 ± 28%     +40.6%      20677 ± 20%  interrupts.CPU133.RES:Rescheduling_interrupts
    181.00 ± 13%     +39.6%     252.75 ± 14%  interrupts.CPU136.TLB:TLB_shootdowns
     15260 ± 24%     +34.3%      20497 ± 16%  interrupts.CPU137.RES:Rescheduling_interrupts
    174.25 ± 18%     +42.9%     249.00 ±  9%  interrupts.CPU138.TLB:TLB_shootdowns
      3465 ± 19%     -37.6%       2161 ± 25%  interrupts.CPU139.NMI:Non-maskable_interrupts
      3465 ± 19%     -37.6%       2161 ± 25%  interrupts.CPU139.PMI:Performance_monitoring_interrupts
    169.00 ± 15%     +44.2%     243.75 ± 10%  interrupts.CPU139.TLB:TLB_shootdowns
      3139 ± 11%     -35.0%       2041 ± 29%  interrupts.CPU140.NMI:Non-maskable_interrupts
      3139 ± 11%     -35.0%       2041 ± 29%  interrupts.CPU140.PMI:Performance_monitoring_interrupts
     12983 ±  8%      -9.7%      11722 ±  3%  interrupts.CPU143.RES:Rescheduling_interrupts
      3586 ± 14%     -29.7%       2522 ± 14%  interrupts.CPU144.NMI:Non-maskable_interrupts
      3586 ± 14%     -29.7%       2522 ± 14%  interrupts.CPU144.PMI:Performance_monitoring_interrupts
      3389 ± 13%     -27.1%       2469 ± 17%  interrupts.CPU148.NMI:Non-maskable_interrupts
      3389 ± 13%     -27.1%       2469 ± 17%  interrupts.CPU148.PMI:Performance_monitoring_interrupts
      3892 ± 15%     -22.3%       3024 ± 19%  interrupts.CPU160.NMI:Non-maskable_interrupts
      3892 ± 15%     -22.3%       3024 ± 19%  interrupts.CPU160.PMI:Performance_monitoring_interrupts
     15714 ± 20%     +25.5%      19727 ± 17%  interrupts.CPU163.RES:Rescheduling_interrupts
      2880 ± 14%     -37.3%       1807 ± 32%  interrupts.CPU164.NMI:Non-maskable_interrupts
      2880 ± 14%     -37.3%       1807 ± 32%  interrupts.CPU164.PMI:Performance_monitoring_interrupts
      3083 ±  3%     -29.9%       2161 ± 24%  interrupts.CPU165.NMI:Non-maskable_interrupts
      3083 ±  3%     -29.9%       2161 ± 24%  interrupts.CPU165.PMI:Performance_monitoring_interrupts
     12511 ±  8%     -15.5%      10570 ± 10%  interrupts.CPU167.RES:Rescheduling_interrupts
      3183 ± 14%     -30.8%       2202 ±  9%  interrupts.CPU172.NMI:Non-maskable_interrupts
      3183 ± 14%     -30.8%       2202 ±  9%  interrupts.CPU172.PMI:Performance_monitoring_interrupts
     15555 ± 26%     +37.3%      21360 ± 20%  interrupts.CPU172.RES:Rescheduling_interrupts
      3044 ±  4%     -29.8%       2137 ± 12%  interrupts.CPU174.NMI:Non-maskable_interrupts
      3044 ±  4%     -29.8%       2137 ± 12%  interrupts.CPU174.PMI:Performance_monitoring_interrupts
     14849 ± 26%     +41.1%      20958 ± 18%  interrupts.CPU175.RES:Rescheduling_interrupts
      3430 ± 17%     -50.6%       1693 ± 32%  interrupts.CPU177.NMI:Non-maskable_interrupts
      3430 ± 17%     -50.6%       1693 ± 32%  interrupts.CPU177.PMI:Performance_monitoring_interrupts
    213.50 ± 43%     -45.4%     116.50 ±  4%  interrupts.CPU179.TLB:TLB_shootdowns
      4220 ± 26%     -51.1%       2065 ± 37%  interrupts.CPU185.NMI:Non-maskable_interrupts
      4220 ± 26%     -51.1%       2065 ± 37%  interrupts.CPU185.PMI:Performance_monitoring_interrupts
     12366 ±  4%      -9.3%      11218 ±  5%  interrupts.CPU186.RES:Rescheduling_interrupts
      4485 ± 35%     -42.8%       2565 ± 26%  interrupts.CPU187.NMI:Non-maskable_interrupts
      4485 ± 35%     -42.8%       2565 ± 26%  interrupts.CPU187.PMI:Performance_monitoring_interrupts
      3775 ± 28%     -32.6%       2545 ± 19%  interrupts.CPU20.NMI:Non-maskable_interrupts
      3775 ± 28%     -32.6%       2545 ± 19%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      3184 ± 13%     -46.9%       1691 ± 23%  interrupts.CPU29.NMI:Non-maskable_interrupts
      3184 ± 13%     -46.9%       1691 ± 23%  interrupts.CPU29.PMI:Performance_monitoring_interrupts
      3601 ± 18%     -40.1%       2155 ± 27%  interrupts.CPU32.NMI:Non-maskable_interrupts
      3601 ± 18%     -40.1%       2155 ± 27%  interrupts.CPU32.PMI:Performance_monitoring_interrupts
    658.25 ± 51%     +32.6%     873.00 ±  4%  interrupts.CPU34.CAL:Function_call_interrupts
    138.25 ± 45%     +62.4%     224.50 ± 16%  interrupts.CPU37.TLB:TLB_shootdowns
    657.00 ± 50%     +33.9%     879.75 ±  5%  interrupts.CPU38.CAL:Function_call_interrupts
      3674 ± 23%     -40.7%       2179 ± 45%  interrupts.CPU38.NMI:Non-maskable_interrupts
      3674 ± 23%     -40.7%       2179 ± 45%  interrupts.CPU38.PMI:Performance_monitoring_interrupts
      3842 ± 14%     -46.6%       2051 ± 38%  interrupts.CPU50.NMI:Non-maskable_interrupts
      3842 ± 14%     -46.6%       2051 ± 38%  interrupts.CPU50.PMI:Performance_monitoring_interrupts
      2875 ± 20%     -42.9%       1641 ± 25%  interrupts.CPU54.NMI:Non-maskable_interrupts
      2875 ± 20%     -42.9%       1641 ± 25%  interrupts.CPU54.PMI:Performance_monitoring_interrupts
     14360 ±  4%     -14.6%      12262 ±  6%  interrupts.CPU55.RES:Rescheduling_interrupts
      3257 ± 18%     -38.9%       1988 ± 20%  interrupts.CPU57.NMI:Non-maskable_interrupts
      3257 ± 18%     -38.9%       1988 ± 20%  interrupts.CPU57.PMI:Performance_monitoring_interrupts
    332.75 ± 85%     -68.2%     105.75 ±141%  interrupts.CPU59.TLB:TLB_shootdowns
      3046 ± 27%     -43.1%       1734 ± 35%  interrupts.CPU69.NMI:Non-maskable_interrupts
      3046 ± 27%     -43.1%       1734 ± 35%  interrupts.CPU69.PMI:Performance_monitoring_interrupts
      2768 ± 22%     -52.3%       1320 ± 54%  interrupts.CPU70.NMI:Non-maskable_interrupts
      2768 ± 22%     -52.3%       1320 ± 54%  interrupts.CPU70.PMI:Performance_monitoring_interrupts
      2896 ± 19%     -33.6%       1922 ± 18%  interrupts.CPU71.NMI:Non-maskable_interrupts
      2896 ± 19%     -33.6%       1922 ± 18%  interrupts.CPU71.PMI:Performance_monitoring_interrupts
      4418 ± 16%     -35.8%       2838 ±  9%  interrupts.CPU73.NMI:Non-maskable_interrupts
      4418 ± 16%     -35.8%       2838 ±  9%  interrupts.CPU73.PMI:Performance_monitoring_interrupts
      3848 ±  7%     -26.6%       2825 ± 12%  interrupts.CPU75.NMI:Non-maskable_interrupts
      3848 ±  7%     -26.6%       2825 ± 12%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
     15568 ±  5%     -13.5%      13469 ±  2%  interrupts.CPU79.RES:Rescheduling_interrupts
      4364 ± 12%     -38.2%       2699 ± 21%  interrupts.CPU90.NMI:Non-maskable_interrupts
      4364 ± 12%     -38.2%       2699 ± 21%  interrupts.CPU90.PMI:Performance_monitoring_interrupts
      4179 ±  6%     -49.8%       2098 ± 20%  interrupts.CPU94.NMI:Non-maskable_interrupts
      4179 ±  6%     -49.8%       2098 ± 20%  interrupts.CPU94.PMI:Performance_monitoring_interrupts
     15345 ±  9%     -17.3%      12688 ±  6%  interrupts.CPU99.RES:Rescheduling_interrupts
    350.75           -56.6%     152.25 ± 60%  interrupts.IWI:IRQ_work_interrupts
    612737 ±  3%     -24.2%     464206 ± 19%  interrupts.NMI:Non-maskable_interrupts
    612737 ±  3%     -24.2%     464206 ± 19%  interrupts.PMI:Performance_monitoring_interrupts
     21.44 ±  2%      -7.0       14.48 ± 33%  perf-profile.calltrace.cycles-pp.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     21.45 ±  2%      -7.0       14.50 ± 33%  perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     21.37 ±  2%      -6.9       14.43 ± 33%  perf-profile.calltrace.cycles-pp.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter.new_sync_write.vfs_write
     21.38 ±  2%      -6.9       14.44 ± 33%  perf-profile.calltrace.cycles-pp.generic_file_direct_write.btrfs_file_write_iter.new_sync_write.vfs_write.ksys_write
     10.50 ±  3%      -6.1        4.36 ± 73%  perf-profile.calltrace.cycles-pp.btrfs_release_path.btrfs_free_path.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper
     10.50 ±  3%      -6.1        4.36 ± 73%  perf-profile.calltrace.cycles-pp.btrfs_free_path.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
     10.39 ±  3%      -6.1        4.27 ± 74%  perf-profile.calltrace.cycles-pp.__wake_up_common_lock.btrfs_release_path.btrfs_free_path.btrfs_mark_extent_written.btrfs_finish_ordered_io
      9.67 ±  3%      -5.8        3.86 ± 78%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.__wake_up_common_lock.btrfs_release_path.btrfs_free_path
      9.68 ±  3%      -5.8        3.87 ± 78%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.btrfs_release_path.btrfs_free_path.btrfs_mark_extent_written
     16.52            -4.4       12.14 ± 25%  perf-profile.calltrace.cycles-pp.do_blockdev_direct_IO.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter.new_sync_write
     15.70 ±  2%      -4.2       11.54 ± 13%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node
     15.74 ±  2%      -4.1       11.60 ± 13%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot
     15.44            -4.1       11.35 ± 24%  perf-profile.calltrace.cycles-pp.btrfs_get_blocks_direct.do_blockdev_direct_IO.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter
     14.74            -4.0       10.73 ± 24%  perf-profile.calltrace.cycles-pp.can_nocow_extent.btrfs_get_blocks_direct.do_blockdev_direct_IO.btrfs_direct_IO.generic_file_direct_write
     13.61            -4.0        9.64 ± 25%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.can_nocow_extent
     13.65            -4.0        9.69 ± 25%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent.can_nocow_extent.btrfs_get_blocks_direct
     13.91            -3.8       10.06 ± 23%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_file_extent.can_nocow_extent.btrfs_get_blocks_direct.do_blockdev_direct_IO
     13.92            -3.8       10.07 ± 23%  perf-profile.calltrace.cycles-pp.btrfs_lookup_file_extent.can_nocow_extent.btrfs_get_blocks_direct.do_blockdev_direct_IO.btrfs_direct_IO
      7.93            -2.7        5.23 ± 22%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      3.68 ±  5%      -2.1        1.53 ±100%  perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_space.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter.new_sync_write
      3.66 ±  5%      -2.1        1.52 ±100%  perf-profile.calltrace.cycles-pp.btrfs_delalloc_reserve_metadata.btrfs_delalloc_reserve_space.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter
      3.59 ±  6%      -2.1        1.48 ±100%  perf-profile.calltrace.cycles-pp.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_delalloc_reserve_space.btrfs_direct_IO.generic_file_direct_write
      3.36 ±  6%      -2.0        1.35 ±100%  perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_delalloc_reserve_space.btrfs_direct_IO
      3.29 ±  6%      -2.0        1.32 ±100%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_reserve_metadata_bytes.btrfs_delalloc_reserve_metadata.btrfs_delalloc_reserve_space
      5.31 ±  2%      -1.7        3.63 ± 27%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_file_extent
      7.47 ±  3%      -0.9        6.57 ±  8%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
      1.15 ±  6%      -0.8        0.39 ±100%  perf-profile.calltrace.cycles-pp.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread.ret_from_fork
      1.15 ±  5%      -0.8        0.39 ±100%  perf-profile.calltrace.cycles-pp.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread.kthread
      1.15 ±  5%      -0.8        0.39 ±100%  perf-profile.calltrace.cycles-pp.__btrfs_run_delayed_items.flush_space.btrfs_async_reclaim_metadata_space.process_one_work.worker_thread
      1.14 ±  6%      -0.8        0.39 ±100%  perf-profile.calltrace.cycles-pp.__btrfs_update_delayed_inode.__btrfs_run_delayed_items.flush_space.btrfs_async_reclaim_metadata_space.process_one_work
      1.05 ±  6%      -0.7        0.36 ±100%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode.__btrfs_run_delayed_items.flush_space
      1.05 ±  6%      -0.7        0.36 ±100%  perf-profile.calltrace.cycles-pp.btrfs_lookup_inode.__btrfs_update_delayed_inode.__btrfs_run_delayed_items.flush_space.btrfs_async_reclaim_metadata_space
      1.04 ±  6%      -0.7        0.35 ±100%  perf-profile.calltrace.cycles-pp.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode.__btrfs_run_delayed_items
      1.04 ±  6%      -0.7        0.35 ±100%  perf-profile.calltrace.cycles-pp.btrfs_tree_read_lock.btrfs_read_lock_root_node.btrfs_search_slot.btrfs_lookup_inode.__btrfs_update_delayed_inode
      1.23 ±  7%      -0.7        0.56 ±100%  perf-profile.calltrace.cycles-pp.btrfs_remove_ordered_extent.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread
      1.17 ±  7%      -0.6        0.52 ±100%  perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_remove_ordered_extent.btrfs_finish_ordered_io.btrfs_work_helper
      1.17 ±  7%      -0.6        0.52 ±100%  perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_remove_ordered_extent.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
      0.95 ±  8%      -0.5        0.41 ±100%  perf-profile.calltrace.cycles-pp._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_remove_ordered_extent.btrfs_finish_ordered_io
      0.93 ±  9%      -0.5        0.40 ±100%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_remove_ordered_extent
      0.70 ± 18%      -0.4        0.28 ±100%  perf-profile.calltrace.cycles-pp.btrfs_block_rsv_release.btrfs_inode_rsv_release.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter
      0.70 ± 17%      -0.4        0.29 ±100%  perf-profile.calltrace.cycles-pp.btrfs_inode_rsv_release.btrfs_direct_IO.generic_file_direct_write.btrfs_file_write_iter.new_sync_write
      0.62 ±  8%      -0.3        0.28 ±100%  perf-profile.calltrace.cycles-pp.ret_from_intr.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.62 ±  8%      -0.3        0.28 ±100%  perf-profile.calltrace.cycles-pp.do_IRQ.ret_from_intr.cpuidle_enter_state.cpuidle_enter.do_idle
      0.53 ±  3%      +0.3        0.81 ± 25%  perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending
      0.56 ±  3%      +0.3        0.84 ± 24%  perf-profile.calltrace.cycles-pp.enqueue_task_fair.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle
      0.56 ±  3%      +0.3        0.84 ± 25%  perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary
      0.56 ±  3%      +0.3        0.84 ± 25%  perf-profile.calltrace.cycles-pp.activate_task.ttwu_do_activate.sched_ttwu_pending.do_idle.cpu_startup_entry
      0.64 ±  2%      +0.3        0.99 ± 26%  perf-profile.calltrace.cycles-pp.sched_ttwu_pending.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.75 ±  6%      +0.4        1.18 ± 27%  perf-profile.calltrace.cycles-pp.smp_apic_timer_interrupt.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
      0.26 ±100%      +0.5        0.72 ± 23%  perf-profile.calltrace.cycles-pp.menu_select.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64
      0.82 ±  6%      +0.5        1.31 ± 27%  perf-profile.calltrace.cycles-pp.apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
      0.00            +1.3        1.31 ± 37%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      0.00            +1.6        1.58 ± 33%  perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
      2.91 ±  3%      +4.2        7.08 ± 41%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node
      2.92 ±  3%      +4.2        7.11 ± 41%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
      2.93 ±  3%      +4.2        7.15 ± 41%  perf-profile.calltrace.cycles-pp.finish_wait.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
      4.20 ±  2%      +6.7       10.93 ± 50%  perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node
     35.04 ±  3%      +6.8       41.85 ±  7%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.22 ±  2%      +6.8       11.03 ± 50%  perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot
     35.15 ±  3%      +6.9       42.01 ±  7%  perf-profile.calltrace.cycles-pp.ret_from_fork
     35.15 ±  3%      +6.9       42.01 ±  7%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
     35.14 ±  3%      +6.9       42.00 ±  7%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      4.29 ±  2%      +7.0       11.26 ± 50%  perf-profile.calltrace.cycles-pp.prepare_to_wait_event.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written
     33.85 ±  3%      +7.2       41.02 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread.kthread
     33.86 ±  3%      +7.5       41.36 ±  8%  perf-profile.calltrace.cycles-pp.btrfs_work_helper.process_one_work.worker_thread.kthread.ret_from_fork
     31.32 ±  3%      +7.9       39.25 ± 11%  perf-profile.calltrace.cycles-pp.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work.worker_thread
     20.65 ±  3%     +12.8       33.42 ± 25%  perf-profile.calltrace.cycles-pp.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper.process_one_work
      7.60 ±  3%     +12.9       20.49 ± 43%  perf-profile.calltrace.cycles-pp.btrfs_tree_lock.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io
      7.61 ±  3%     +12.9       20.50 ± 43%  perf-profile.calltrace.cycles-pp.btrfs_lock_root_node.btrfs_search_slot.btrfs_mark_extent_written.btrfs_finish_ordered_io.btrfs_work_helper
     21.55 ±  2%      -7.0       14.59 ± 33%  perf-profile.children.cycles-pp.ksys_write
     21.54 ±  2%      -7.0       14.57 ± 33%  perf-profile.children.cycles-pp.vfs_write
     21.48 ±  2%      -7.0       14.52 ± 33%  perf-profile.children.cycles-pp.new_sync_write
     21.73 ±  2%      -7.0       14.77 ± 32%  perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     21.45 ±  2%      -7.0       14.49 ± 33%  perf-profile.children.cycles-pp.btrfs_file_write_iter
     21.73 ±  2%      -7.0       14.77 ± 33%  perf-profile.children.cycles-pp.do_syscall_64
     21.39 ±  2%      -7.0       14.44 ± 33%  perf-profile.children.cycles-pp.generic_file_direct_write
     21.37 ±  2%      -6.9       14.43 ± 33%  perf-profile.children.cycles-pp.btrfs_direct_IO
     10.57 ±  3%      -6.2        4.41 ± 73%  perf-profile.children.cycles-pp.btrfs_free_path
     10.65 ±  3%      -6.1        4.54 ± 72%  perf-profile.children.cycles-pp.btrfs_release_path
     27.59 ±  2%      -5.5       22.07 ± 17%  perf-profile.children.cycles-pp.btrfs_read_lock_root_node
     27.41 ±  2%      -5.5       21.94 ± 17%  perf-profile.children.cycles-pp.btrfs_tree_read_lock
     10.82 ±  3%      -4.6        6.18 ± 55%  perf-profile.children.cycles-pp.__wake_up_common_lock
     16.52            -4.4       12.14 ± 25%  perf-profile.children.cycles-pp.do_blockdev_direct_IO
     15.44            -4.1       11.35 ± 24%  perf-profile.children.cycles-pp.btrfs_get_blocks_direct
     14.73            -4.0       10.73 ± 24%  perf-profile.children.cycles-pp.can_nocow_extent
     13.92            -3.8       10.07 ± 23%  perf-profile.children.cycles-pp.btrfs_lookup_file_extent
      5.67 ±  6%      -3.0        2.70 ± 81%  perf-profile.children.cycles-pp._raw_spin_lock
      3.68 ±  5%      -2.0        1.67 ± 83%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_space
      3.65 ±  6%      -2.0        1.65 ± 84%  perf-profile.children.cycles-pp.btrfs_delalloc_reserve_metadata
      3.59 ±  6%      -2.0        1.60 ± 85%  perf-profile.children.cycles-pp.btrfs_reserve_metadata_bytes
      2.31 ±  6%      -1.2        1.13 ± 75%  perf-profile.children.cycles-pp.btrfs_block_rsv_release
      2.24 ±  6%      -1.1        1.11 ± 74%  perf-profile.children.cycles-pp.btrfs_inode_rsv_release
      1.15 ±  6%      -0.8        0.39 ±100%  perf-profile.children.cycles-pp.btrfs_async_reclaim_metadata_space
      1.15 ±  5%      -0.8        0.39 ±100%  perf-profile.children.cycles-pp.flush_space
      1.15 ±  5%      -0.8        0.39 ±100%  perf-profile.children.cycles-pp.__btrfs_run_delayed_items
      1.14 ±  6%      -0.8        0.39 ±100%  perf-profile.children.cycles-pp.__btrfs_update_delayed_inode
      1.05 ±  6%      -0.7        0.36 ±100%  perf-profile.children.cycles-pp.btrfs_lookup_inode
      1.23 ±  7%      -0.6        0.65 ± 75%  perf-profile.children.cycles-pp.btrfs_remove_ordered_extent
      0.48 ±  3%      -0.1        0.34 ± 38%  perf-profile.children.cycles-pp.btrfs_try_granting_tickets
      0.29 ±  2%      -0.1        0.15 ± 53%  perf-profile.children.cycles-pp.btrfs_update_inode_fallback
      0.29 ±  2%      -0.1        0.15 ± 53%  perf-profile.children.cycles-pp.btrfs_update_inode
      0.26            -0.1        0.13 ± 54%  perf-profile.children.cycles-pp.btrfs_delayed_update_inode
      0.27            -0.1        0.15 ± 35%  perf-profile.children.cycles-pp.btrfs_map_bio
      0.14 ±  3%      -0.1        0.04 ±100%  perf-profile.children.cycles-pp.__mutex_lock
      0.18 ±  2%      -0.1        0.08 ± 31%  perf-profile.children.cycles-pp.submit_bio
      0.13 ±  5%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.mutex_spin_on_owner
      0.17 ±  2%      -0.1        0.08 ± 31%  perf-profile.children.cycles-pp.generic_make_request
      0.30 ±  2%      -0.1        0.24 ± 21%  perf-profile.children.cycles-pp.find_extent_buffer
      0.12            -0.1        0.06 ± 63%  perf-profile.children.cycles-pp.blk_mq_make_request
      0.16 ±  2%      -0.1        0.11 ± 36%  perf-profile.children.cycles-pp.check_committed_ref
      0.05 ±  8%      +0.0        0.07 ± 10%  perf-profile.children.cycles-pp.ttwu_do_wakeup
      0.05 ±  8%      +0.0        0.07 ± 11%  perf-profile.children.cycles-pp.__switch_to
      0.11 ±  7%      +0.0        0.13 ±  5%  perf-profile.children.cycles-pp.scheduler_tick
      0.07 ±  6%      +0.0        0.09 ±  4%  perf-profile.children.cycles-pp.__unwind_start
      0.05            +0.0        0.08 ± 10%  perf-profile.children.cycles-pp.__switch_to_asm
      0.08 ±  8%      +0.0        0.11 ± 19%  perf-profile.children.cycles-pp.read_tsc
      0.05 ±  8%      +0.0        0.08 ± 15%  perf-profile.children.cycles-pp.native_sched_clock
      0.17 ±  4%      +0.0        0.20 ±  3%  perf-profile.children.cycles-pp.select_task_rq_fair
      0.05 ±  9%      +0.0        0.09 ± 16%  perf-profile.children.cycles-pp.sched_clock
      0.06 ±  6%      +0.0        0.10 ± 18%  perf-profile.children.cycles-pp.sched_clock_cpu
      0.17 ±  6%      +0.0        0.21 ± 13%  perf-profile.children.cycles-pp.dequeue_entity
      0.09 ±  4%      +0.0        0.14 ± 12%  perf-profile.children.cycles-pp.update_rq_clock
      0.01 ±173%      +0.0        0.06 ± 11%  perf-profile.children.cycles-pp.tick_nohz_idle_exit
      0.11 ±  7%      +0.0        0.16 ± 23%  perf-profile.children.cycles-pp.__next_timer_interrupt
      0.29            +0.1        0.34 ±  4%  perf-profile.children.cycles-pp.schedule_idle
      0.20 ±  5%      +0.1        0.25 ± 14%  perf-profile.children.cycles-pp.dequeue_task_fair
      0.19 ±  5%      +0.1        0.25 ± 13%  perf-profile.children.cycles-pp.update_process_times
      0.08 ±  5%      +0.1        0.15 ± 35%  perf-profile.children.cycles-pp.__tree_search
      0.00            +0.1        0.07 ± 25%  perf-profile.children.cycles-pp.hrtimer_next_event_without
      0.20 ±  4%      +0.1        0.27 ± 13%  perf-profile.children.cycles-pp.tick_sched_handle
      0.16 ±  5%      +0.1        0.23 ± 25%  perf-profile.children.cycles-pp.get_next_timer_interrupt
      0.18 ±  4%      +0.1        0.25 ± 22%  perf-profile.children.cycles-pp.__orc_find
      0.22 ±  6%      +0.1        0.30 ± 13%  perf-profile.children.cycles-pp.tick_sched_timer
      0.62 ±  2%      +0.1        0.71 ±  8%  perf-profile.children.cycles-pp.schedule
      0.06 ±  6%      +0.1        0.15 ± 31%  perf-profile.children.cycles-pp._raw_write_lock
      0.00            +0.1        0.09 ± 35%  perf-profile.children.cycles-pp.btrfs_set_token_32
      0.20 ±  4%      +0.1        0.29 ± 24%  perf-profile.children.cycles-pp.tick_nohz_next_event
      0.34 ±  7%      +0.1        0.44 ±  7%  perf-profile.children.cycles-pp.__softirqentry_text_start
      0.00            +0.1        0.11 ± 37%  perf-profile.children.cycles-pp.btrfs_submit_bio_start_direct_io
      0.00            +0.1        0.11 ± 34%  perf-profile.children.cycles-pp.run_one_async_start
      0.25 ±  5%      +0.1        0.38 ± 24%  perf-profile.children.cycles-pp.tick_nohz_get_sleep_length
      0.89 ±  2%      +0.1        1.02 ±  7%  perf-profile.children.cycles-pp.__sched_text_start
      0.41 ±  7%      +0.1        0.54 ± 10%  perf-profile.children.cycles-pp.irq_exit
      0.08 ±  6%      +0.1        0.21 ± 40%  perf-profile.children.cycles-pp.btrfs_get_token_32
      0.00            +0.2        0.16 ± 33%  perf-profile.children.cycles-pp.run_one_async_done
      0.36 ±  5%      +0.2        0.52 ± 20%  perf-profile.children.cycles-pp.__hrtimer_run_queues
      0.45 ±  5%      +0.2        0.61 ± 18%  perf-profile.children.cycles-pp.unwind_next_frame
      0.50 ±  5%      +0.2        0.72 ± 23%  perf-profile.children.cycles-pp.menu_select
      0.67 ±  5%      +0.2        0.90 ± 18%  perf-profile.children.cycles-pp.arch_stack_walk
      0.55 ±  6%      +0.2        0.79 ± 20%  perf-profile.children.cycles-pp.hrtimer_interrupt
      0.00            +0.3        0.26 ± 32%  perf-profile.children.cycles-pp.btrfs_wq_submit_bio
      0.71 ±  5%      +0.3        0.97 ± 18%  perf-profile.children.cycles-pp.stack_trace_save_tsk
      0.00            +0.3        0.28 ± 45%  perf-profile.children.cycles-pp.setup_leaf_for_split
      0.87 ±  5%      +0.3        1.17 ± 16%  perf-profile.children.cycles-pp.__account_scheduler_latency
      1.10 ±  4%      +0.3        1.41 ± 12%  perf-profile.children.cycles-pp.enqueue_entity
      1.15 ±  4%      +0.3        1.47 ± 12%  perf-profile.children.cycles-pp.ttwu_do_activate
      1.14 ±  4%      +0.3        1.46 ± 12%  perf-profile.children.cycles-pp.enqueue_task_fair
      1.14 ±  4%      +0.3        1.47 ± 12%  perf-profile.children.cycles-pp.activate_task
      0.67 ±  2%      +0.4        1.02 ± 26%  perf-profile.children.cycles-pp.sched_ttwu_pending
      1.03 ±  6%      +0.4        1.46 ± 18%  perf-profile.children.cycles-pp.smp_apic_timer_interrupt
      0.00            +0.4        0.44 ± 40%  perf-profile.children.cycles-pp.btrfs_dio_private_put
      0.00            +0.5        0.46 ± 39%  perf-profile.children.cycles-pp.btrfs_end_dio_bio
      1.15 ±  5%      +0.5        1.65 ± 18%  perf-profile.children.cycles-pp.apic_timer_interrupt
      0.03 ±100%      +0.9        0.96 ± 80%  perf-profile.children.cycles-pp.btrfs_unlock_up_safe
      0.05            +1.0        1.04 ± 78%  perf-profile.children.cycles-pp.setup_items_for_insert
      0.29 ±  5%      +1.0        1.28 ± 65%  perf-profile.children.cycles-pp.queued_read_lock_slowpath
      0.05            +1.2        1.28 ± 70%  perf-profile.children.cycles-pp.btrfs_duplicate_item
      0.24 ±  6%      +1.4        1.65 ± 32%  perf-profile.children.cycles-pp.queued_write_lock_slowpath
     35.05 ±  3%      +6.8       41.85 ±  7%  perf-profile.children.cycles-pp.process_one_work
     35.15 ±  3%      +6.9       42.01 ±  7%  perf-profile.children.cycles-pp.ret_from_fork
     35.15 ±  3%      +6.9       42.01 ±  7%  perf-profile.children.cycles-pp.kthread
     35.14 ±  3%      +6.9       42.00 ±  7%  perf-profile.children.cycles-pp.worker_thread
     33.85 ±  3%      +7.2       41.02 ±  8%  perf-profile.children.cycles-pp.btrfs_finish_ordered_io
     33.86 ±  3%      +7.5       41.36 ±  8%  perf-profile.children.cycles-pp.btrfs_work_helper
     31.32 ±  3%      +7.9       39.25 ± 11%  perf-profile.children.cycles-pp.btrfs_mark_extent_written
     36.36 ±  2%      +8.2       44.60 ± 12%  perf-profile.children.cycles-pp.btrfs_search_slot
      7.61 ±  3%     +13.1       20.67 ± 43%  perf-profile.children.cycles-pp.btrfs_lock_root_node
      7.61 ±  3%     +13.1       20.67 ± 43%  perf-profile.children.cycles-pp.btrfs_tree_lock
      0.13 ±  5%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.mutex_spin_on_owner
      0.08 ±  6%      +0.0        0.10 ±  5%  perf-profile.self.cycles-pp.select_task_rq_fair
      0.14            +0.0        0.16 ±  5%  perf-profile.self.cycles-pp.__sched_text_start
      0.07 ±  5%      +0.0        0.10 ± 18%  perf-profile.self.cycles-pp._find_next_bit
      0.05            +0.0        0.08 ± 10%  perf-profile.self.cycles-pp.__switch_to_asm
      0.04 ± 58%      +0.0        0.07 ± 12%  perf-profile.self.cycles-pp.__switch_to
      0.08 ±  5%      +0.0        0.11 ± 19%  perf-profile.self.cycles-pp.read_tsc
      0.05            +0.0        0.08 ± 15%  perf-profile.self.cycles-pp.native_sched_clock
      0.07            +0.0        0.10 ± 10%  perf-profile.self.cycles-pp.update_rq_clock
      0.07 ± 10%      +0.0        0.12 ± 26%  perf-profile.self.cycles-pp.do_idle
      0.00            +0.1        0.05 ±  8%  perf-profile.self.cycles-pp.enqueue_task_fair
      0.00            +0.1        0.06 ± 16%  perf-profile.self.cycles-pp.__next_timer_interrupt
      0.00            +0.1        0.06 ± 20%  perf-profile.self.cycles-pp.stack_trace_save_tsk
      0.06 ±  6%      +0.1        0.12 ± 36%  perf-profile.self.cycles-pp.__tree_search
      0.17 ±  7%      +0.1        0.23 ± 14%  perf-profile.self.cycles-pp.unwind_next_frame
      0.12 ±  6%      +0.1        0.19 ± 25%  perf-profile.self.cycles-pp.cpuidle_enter_state
      0.10 ±  8%      +0.1        0.17 ± 18%  perf-profile.self.cycles-pp.prepare_to_wait_event
      0.18 ±  4%      +0.1        0.25 ± 22%  perf-profile.self.cycles-pp.__orc_find
      0.07 ± 10%      +0.1        0.15 ± 48%  perf-profile.self.cycles-pp.queued_read_lock_slowpath
      0.00            +0.1        0.08 ± 39%  perf-profile.self.cycles-pp.btrfs_set_token_32
      0.06 ±  6%      +0.1        0.15 ± 34%  perf-profile.self.cycles-pp._raw_write_lock
      0.08 ±  6%      +0.1        0.20 ± 38%  perf-profile.self.cycles-pp.btrfs_get_token_32
      0.53 ±  3%      +0.2        0.71 ±  8%  perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      0.13 ±  3%      +0.2        0.33 ± 19%  perf-profile.self.cycles-pp.queued_write_lock_slowpath


                                                                                
                                 fio.write_bw_MBps                              
                                                                                
  500 +---------------------------------------------------------------------+   
      |     .+  +                                              +      .+    |   
  450 |+++++  ++ ++++++.+++++++++++.+++++++++++.++++++++++.++++ ++++++  ++++|   
  400 |-+                                                                   |   
      |                                                                     |   
  350 |-+    OO OO                O  O                                      |   
      |           OOO   O O O O        O                                    |   
  300 |-+  O                                                                |   
      |                                                                     |   
  250 |-+                                 OO                                |   
  200 |-+            O     O O OO     O OO                                  |   
      |        O      O  O       O  O                                       |   
  150 |OOOO                                                                 |   
      |                                                                     |   
  100 +---------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.6.0-rc7-00220-g1eb52c8bd8d6b" of type "text/plain" (203570 bytes)

View attachment "job-script" of type "text/plain" (8118 bytes)

View attachment "job.yaml" of type "text/plain" (5455 bytes)

View attachment "reproduce" of type "text/plain" (722 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ