lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [day] [month] [year] [list]
Message-ID: <20210214140921.GD6321@xsang-OptiPlex-9020>
Date:   Sun, 14 Feb 2021 22:09:21 +0800
From:   kernel test robot <oliver.sang@...el.com>
To:     Harshad Shirwadkar <harshadshirwadkar@...il.com>
Cc:     0day robot <lkp@...el.com>, LKML <linux-kernel@...r.kernel.org>,
        lkp@...ts.01.org, ying.huang@...el.com, feng.tang@...el.com,
        zhengjun.xing@...el.com, linux-ext4@...r.kernel.org, tytso@....edu,
        bzzz@...mcloud.com, artem.blagodarenko@...il.com, sihara@....com,
        adilger@...ger.ca, Harshad Shirwadkar <harshadshirwadkar@...il.com>
Subject: [ext4]  ef4eebad9c:
 fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec -9.8% regression


Greeting,

FYI, we noticed a -9.8% regression of fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec due to commit:


commit: ef4eebad9c018a972a470b7b41e68bc981b31d00 ("ext4: improve cr 0 / cr 1 group scanning")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git Harshad-Shirwadkar/ext4-drop-s_mb_bal_lock-and-convert-protected-fields-to-atomic/20210210-054647


in testcase: fxmark
on test machine: 288 threads Intel(R) Xeon Phi(TM) CPU 7295 @ 1.50GHz with 80G memory
with following parameters:

	disk: 1HDD
	media: hdd
	test: MWCL
	fstype: ext4_no_jnl
	directio: bufferedio
	cpufreq_governor: performance
	ucode: 0x11




If you fix the issue, kindly add following tag
Reported-by: kernel test robot <oliver.sang@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install                job.yaml  # job file is attached in this email
        bin/lkp split-job --compatible job.yaml
        bin/lkp run                    compatible-job.yaml

=========================================================================================
compiler/cpufreq_governor/directio/disk/fstype/kconfig/media/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/bufferedio/1HDD/ext4_no_jnl/x86_64-rhel-8.3/hdd/debian-10.4-x86_64-20200603.cgz/lkp-knm01/MWCL/fxmark/0x11

commit: 
  a932b2b788 ("ext4: add MB_NUM_ORDERS macro")
  ef4eebad9c ("ext4: improve cr 0 / cr 1 group scanning")

a932b2b7885865bd ef4eebad9c018a972a470b7b41e 
---------------- --------------------------- 
       fail:runs  %reproduction    fail:runs
           |             |             |    
          0:4            5%           0:4     perf-profile.children.cycles-pp.error_return
          1:4           10%           1:4     perf-profile.children.cycles-pp.error_entry
           :4            4%           0:4     perf-profile.self.cycles-pp.error_return
          1:4            9%           1:4     perf-profile.self.cycles-pp.error_entry
         %stddev     %change         %stddev
             \          |                \  
    315979            -9.8%     284922        fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works
     10532            -9.8%       9497        fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works/sec
      0.01 ±100%  +38150.0%       1.91 ± 11%  fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_sec
      0.01 ±100%  +38189.0%       3.16 ± 11%  fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_util
      5.33 ± 17%     +22.5%       6.52 ±  4%  fxmark.hdd_ext4_no_jnl_MWCL_36_bufferedio.idle_sec
      0.49 ± 16%     +22.2%       0.60 ±  4%  fxmark.hdd_ext4_no_jnl_MWCL_36_bufferedio.idle_util
      6.50 ±  9%     -21.6%       5.09 ±  8%  fxmark.hdd_ext4_no_jnl_MWCL_45_bufferedio.idle_sec
      0.48 ±  9%     -22.6%       0.37 ± 10%  fxmark.hdd_ext4_no_jnl_MWCL_45_bufferedio.idle_util
      0.00 ±173%  +75800.0%       1.90 ± 22%  fxmark.hdd_ext4_no_jnl_MWCL_4_bufferedio.iowait_sec
      0.00 ±173%  +75915.1%       1.57 ± 22%  fxmark.hdd_ext4_no_jnl_MWCL_4_bufferedio.iowait_util
      0.52 ±  6%     -11.1%       0.46 ±  4%  fxmark.hdd_ext4_no_jnl_MWCL_54_bufferedio.softirq_util
      1090            +3.2%       1124        fxmark.time.elapsed_time
      1090            +3.2%       1124        fxmark.time.elapsed_time.max
     65107            -5.9%      61260        fxmark.time.involuntary_context_switches
     69.50            -5.8%      65.50        fxmark.time.percent_of_cpu_this_job_got
     28.28            -4.1%      27.11 ±  2%  fxmark.time.user_time
      5.50 ±  3%      +2.8        8.26 ±  4%  mpstat.cpu.all.iowait%
     58.50            -2.6%      57.00        vmstat.cpu.id
     38021            -6.2%      35647        vmstat.io.bo
     85553            -4.1%      82045        vmstat.system.in
     58.98            -2.7%      57.37        iostat.cpu.idle
      5.57 ±  4%     +49.8%       8.34 ±  4%  iostat.cpu.iowait
     30.35            -3.1%      29.41        iostat.cpu.system
      2.81            -5.3%       2.66        iostat.cpu.user
    711278           +15.3%     820380        meminfo.Dirty
   7003710            -9.0%    6376219        meminfo.KReclaimable
      1840 ± 12%     +21.4%       2233        meminfo.Mlocked
   7003710            -9.0%    6376219        meminfo.SReclaimable
    710759           +15.4%     820265        numa-meminfo.node0.Dirty
   6994361            -9.0%    6365487        numa-meminfo.node0.KReclaimable
      1053 ± 12%     +21.6%       1281        numa-meminfo.node0.Mlocked
   6994361            -9.0%    6365487        numa-meminfo.node0.SReclaimable
    177664           +15.5%     205237        numa-vmstat.node0.nr_dirty
    262.75 ± 12%     +21.9%     320.25        numa-vmstat.node0.nr_mlock
   1751239            -9.0%    1594254        numa-vmstat.node0.nr_slab_reclaimable
    178395           +15.4%     205952        numa-vmstat.node0.nr_zone_write_pending
      2244 ± 68%     -82.7%     387.72 ± 15%  sched_debug.cfs_rq:/.load_avg.max
    309.86 ± 59%     -72.6%      84.98 ± 14%  sched_debug.cfs_rq:/.load_avg.stddev
    385204 ±  8%     -35.5%     248625 ±  6%  sched_debug.cfs_rq:/.min_vruntime.stddev
   -681107           -51.9%    -327811        sched_debug.cfs_rq:/.spread0.min
    385220 ±  8%     -35.5%     248625 ±  6%  sched_debug.cfs_rq:/.spread0.stddev
     10.05 ± 51%    +506.0%      60.92 ± 32%  sched_debug.cfs_rq:/.util_est_enqueued.min
    125.29 ± 14%     -18.5%     102.09 ±  7%  sched_debug.cfs_rq:/.util_est_enqueued.stddev
     24.34 ±  8%     -21.6%      19.08 ±  2%  sched_debug.cpu.clock.stddev
     61783 ±  8%     +33.0%      82157 ±  7%  sched_debug.cpu.nr_switches.avg
     35702 ±  8%     +55.3%      55461 ± 11%  sched_debug.cpu.nr_switches.min
      7989 ± 25%     +87.6%      14991 ± 24%  softirqs.CPU1.BLOCK
    123512 ±  3%      -7.6%     114086 ±  2%  softirqs.CPU21.RCU
    122473 ±  3%      -6.6%     114426 ±  2%  softirqs.CPU25.RCU
     66489 ±  5%     -11.7%      58718 ±  5%  softirqs.CPU29.SCHED
     99247 ±  3%      -8.6%      90723 ±  5%  softirqs.CPU33.RCU
     56394 ±  3%     -13.5%      48805 ±  5%  softirqs.CPU36.SCHED
     43799 ±  4%     -12.9%      38133 ±  4%  softirqs.CPU45.SCHED
     44447 ±  4%     -12.0%      39128 ±  5%  softirqs.CPU51.SCHED
    169512 ±  3%     -11.3%     150299 ±  3%  softirqs.CPU6.RCU
     33198 ±  5%     -14.9%      28240 ± 11%  softirqs.CPU60.SCHED
    147310 ±  6%      -9.0%     134107 ±  2%  softirqs.CPU9.RCU
      0.04 ±  6%      -0.0        0.03 ± 14%  perf-stat.i.branch-miss-rate%
    326874 ±  8%     -15.0%     277893 ± 12%  perf-stat.i.branch-misses
     41754            -4.6%      39817        perf-stat.i.cpu-clock
     85.39            -2.9%      82.87        perf-stat.i.cpu-migrations
      0.38 ± 10%     -16.2%       0.32 ± 11%  perf-stat.i.instructions-per-iTLB-miss
      0.00 ± 11%     -17.2%       0.00 ± 11%  perf-stat.i.ipc
      1.06 ±  3%      -7.8%       0.98        perf-stat.i.major-faults
      0.35            +4.1%       0.37        perf-stat.i.metric.K/sec
     41754            -4.6%      39817        perf-stat.i.task-clock
    348107 ±  7%     -14.8%     296451 ± 12%  perf-stat.ps.branch-misses
     41967            -4.6%      40020        perf-stat.ps.cpu-clock
     85.62            -2.9%      83.09        perf-stat.ps.cpu-migrations
      1.05 ±  3%      -7.7%       0.97        perf-stat.ps.major-faults
     41967            -4.6%      40020        perf-stat.ps.task-clock
      0.11 ±  8%     -33.2%       0.07 ± 28%  perf-sched.sch_delay.avg.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      0.02 ±  9%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.kthreadd.ret_from_fork
      0.28 ± 83%     -86.7%       0.04 ± 33%  perf-sched.sch_delay.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.submit_bh_wbc
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_preempt_disabled.kthread.ret_from_fork
      0.06 ± 19%     -28.4%       0.04 ±  8%  perf-sched.sch_delay.avg.ms.schedule_timeout.rcu_gp_kthread.kthread.ret_from_fork
      0.01 ±100%    +141.3%       0.03 ±  8%  perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion.__flush_work.lru_add_drain_all
      0.06 ± 10%    -100.0%       0.00        perf-sched.sch_delay.avg.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
      0.35 ±113%     -79.7%       0.07 ± 40%  perf-sched.sch_delay.max.ms.do_task_dead.do_exit.do_group_exit.__x64_sys_exit_group.do_syscall_64
      2.77 ± 40%     -46.4%       1.49 ± 53%  perf-sched.sch_delay.max.ms.io_schedule.rq_qos_wait.wbt_wait.__rq_qos_throttle
      0.03 ± 21%    -100.0%       0.00        perf-sched.sch_delay.max.ms.kthreadd.ret_from_fork
      0.01 ± 11%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_preempt_disabled.kthread.ret_from_fork
      0.06 ± 13%    -100.0%       0.00        perf-sched.sch_delay.max.ms.schedule_timeout.wait_for_completion_killable.__kthread_create_on_node.kthread_create_on_node
    139.75 ±  7%     -13.4%     121.00 ±  3%  perf-sched.wait_and_delay.count.preempt_schedule_common._cond_resched.shrink_dentry_list.prune_dcache_sb.super_cache_scan
      8210 ± 10%     -26.3%       6048 ± 12%  perf-sched.wait_and_delay.max.ms.worker_thread.kthread.ret_from_fork
     88.37 ± 15%     -18.2%      72.31 ± 11%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mempool_alloc.bio_alloc_bioset.submit_bh_wbc
     79.45 ±109%    +329.8%     341.45 ± 42%  perf-sched.wait_time.avg.ms.preempt_schedule_common._cond_resched.mutex_lock.drm_gem_shmem_vunmap.mgag200_handle_damage
    129.91 ±  2%     +52.5%     198.10 ± 48%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.submit_bio_checks.submit_bio_noacct.submit_bio
    130.18 ±  3%     +72.5%     224.52 ± 51%  perf-sched.wait_time.max.ms.preempt_schedule_common._cond_resched.write_cache_pages.generic_writepages.do_writepages
      8210 ± 10%     -26.3%       6048 ± 12%  perf-sched.wait_time.max.ms.worker_thread.kthread.ret_from_fork
    639.00            -4.1%     613.00        proc-vmstat.nr_active_anon
    109230            -4.7%     104085        proc-vmstat.nr_active_file
   9734223            -3.3%    9414937        proc-vmstat.nr_dirtied
    178266           +15.5%     205864        proc-vmstat.nr_dirty
    460.75 ± 12%     +21.4%     559.50        proc-vmstat.nr_mlock
   1758100            -8.9%    1601542        proc-vmstat.nr_slab_reclaimable
     68945            -3.0%      66853        proc-vmstat.nr_slab_unreclaimable
   9734223            -3.3%    9414937        proc-vmstat.nr_written
    639.00            -4.1%     613.00        proc-vmstat.nr_zone_active_anon
    109230            -4.7%     104085        proc-vmstat.nr_zone_active_file
    179007           +15.4%     206596        proc-vmstat.nr_zone_write_pending
  24225927            -2.2%   23703313        proc-vmstat.numa_hit
  24225924            -2.2%   23703311        proc-vmstat.numa_local
  47793203            -3.0%   46353511        proc-vmstat.pgalloc_normal
   4923908           +11.4%    5485129        proc-vmstat.pgdeactivate
   3348086            +2.3%    3425886        proc-vmstat.pgfault
  47786479            -3.0%   46346216        proc-vmstat.pgfree
  41377300            -3.3%   40023642        proc-vmstat.pgpgout
    264776            +2.5%     271513        proc-vmstat.pgreuse
   4916073           +11.4%    5477332        proc-vmstat.pgrotated
 1.779e+08            -2.8%  1.729e+08        proc-vmstat.slabs_scanned
   9334464            +2.8%    9594624        proc-vmstat.unevictable_pgs_scanned
    662.25 ±  8%     -15.7%     558.50 ±  6%  slabinfo.Acpi-Parse.active_objs
   3972051            -9.0%    3616212        slabinfo.dentry.active_objs
    189593            -8.9%     172660        slabinfo.dentry.active_slabs
   3981471            -8.9%    3625865        slabinfo.dentry.num_objs
    189593            -8.9%     172660        slabinfo.dentry.num_slabs
      3665          +602.8%      25759        slabinfo.ext4_extent_status.active_objs
     39.75          +558.5%     261.75        slabinfo.ext4_extent_status.active_slabs
      4090          +554.0%      26752        slabinfo.ext4_extent_status.num_objs
     39.75          +558.5%     261.75        slabinfo.ext4_extent_status.num_slabs
      4203 ±  3%    -100.0%       0.00        slabinfo.ext4_groupinfo_4k.active_objs
      4254 ±  2%    -100.0%       0.00        slabinfo.ext4_groupinfo_4k.num_objs
   5178202            -9.1%    4707049        slabinfo.ext4_inode_cache.active_objs
    191816            -9.1%     174364        slabinfo.ext4_inode_cache.active_slabs
   5179060            -9.1%    4707847        slabinfo.ext4_inode_cache.num_objs
    191816            -9.1%     174364        slabinfo.ext4_inode_cache.num_slabs
      1133 ±  5%     -14.9%     965.00 ± 11%  slabinfo.kmalloc-rcl-96.num_objs
     20676           +24.1%      25662        slabinfo.radix_tree_node.active_objs
      1642           +14.8%       1885        slabinfo.radix_tree_node.active_slabs
     23002           +14.8%      26403        slabinfo.radix_tree_node.num_objs
      1642           +14.8%       1885        slabinfo.radix_tree_node.num_slabs
      1069 ±  7%     +16.6%       1246 ±  6%  slabinfo.skbuff_fclone_cache.active_objs
      1250 ±  5%     +15.8%       1448 ±  6%  slabinfo.skbuff_fclone_cache.num_objs
      3019          +122.2%       6710        interrupts.CPU0.180:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
     14733 ± 10%    +135.6%      34711 ± 39%  interrupts.CPU1.180:IR-PCI-MSI.512000-edge.ahci[0000:00:1f.2]
     74.25 ± 41%    +328.6%     318.25 ± 54%  interrupts.CPU1.37:IR-PCI-MSI.4194305-edge.eth0-TxRx-0
      4354 ± 25%     +29.9%       5655 ± 13%  interrupts.CPU11.CAL:Function_call_interrupts
      1283           +25.2%       1607 ± 29%  interrupts.CPU127.CAL:Function_call_interrupts
     10568 ± 28%     +34.3%      14193 ± 15%  interrupts.CPU2.CAL:Function_call_interrupts
    985.00 ± 22%    +130.9%       2274 ± 42%  interrupts.CPU2.RES:Rescheduling_interrupts
    263.25 ±  4%     +24.5%     327.75 ± 20%  interrupts.CPU2.TLB:TLB_shootdowns
    312.00 ± 72%     -50.9%     153.25 ± 22%  interrupts.CPU20.NMI:Non-maskable_interrupts
    312.00 ± 72%     -50.9%     153.25 ± 22%  interrupts.CPU20.PMI:Performance_monitoring_interrupts
      4243 ± 10%     +45.5%       6172 ±  5%  interrupts.CPU22.CAL:Function_call_interrupts
      3434 ± 20%     +58.2%       5433 ± 35%  interrupts.CPU25.CAL:Function_call_interrupts
    491.25 ± 29%     -55.7%     217.75 ± 35%  interrupts.CPU27.NMI:Non-maskable_interrupts
    491.25 ± 29%     -55.7%     217.75 ± 35%  interrupts.CPU27.PMI:Performance_monitoring_interrupts
    390.50 ± 40%     -46.4%     209.50 ±  9%  interrupts.CPU29.RES:Rescheduling_interrupts
    189.50 ± 11%     +23.9%     234.75 ±  5%  interrupts.CPU3.TLB:TLB_shootdowns
    234.75 ± 32%     -39.8%     141.25 ± 29%  interrupts.CPU30.NMI:Non-maskable_interrupts
    234.75 ± 32%     -39.8%     141.25 ± 29%  interrupts.CPU30.PMI:Performance_monitoring_interrupts
    639.50 ± 65%     -53.0%     300.75 ± 26%  interrupts.CPU30.RES:Rescheduling_interrupts
    371.50 ± 24%     -32.5%     250.75 ±  8%  interrupts.CPU34.RES:Rescheduling_interrupts
    246.00 ± 23%     -32.5%     166.00 ±  7%  interrupts.CPU37.RES:Rescheduling_interrupts
    550.25 ± 11%     +91.9%       1055 ± 28%  interrupts.CPU4.RES:Rescheduling_interrupts
    165.75 ± 20%    +108.1%     345.00 ± 47%  interrupts.CPU47.NMI:Non-maskable_interrupts
    165.75 ± 20%    +108.1%     345.00 ± 47%  interrupts.CPU47.PMI:Performance_monitoring_interrupts
      2914 ± 10%     +50.3%       4380 ± 23%  interrupts.CPU48.CAL:Function_call_interrupts
      6123 ±  9%     +43.8%       8808 ± 18%  interrupts.CPU5.CAL:Function_call_interrupts
    146.25 ± 10%    +185.0%     416.75 ± 30%  interrupts.CPU5.NMI:Non-maskable_interrupts
    146.25 ± 10%    +185.0%     416.75 ± 30%  interrupts.CPU5.PMI:Performance_monitoring_interrupts
    477.50 ± 62%     -70.2%     142.50 ± 22%  interrupts.CPU6.NMI:Non-maskable_interrupts
    477.50 ± 62%     -70.2%     142.50 ± 22%  interrupts.CPU6.PMI:Performance_monitoring_interrupts
    580.00 ± 27%    +127.7%       1320 ± 42%  interrupts.CPU6.RES:Rescheduling_interrupts
    479.50 ± 35%     -56.8%     207.25 ± 62%  interrupts.CPU62.NMI:Non-maskable_interrupts
    479.50 ± 35%     -56.8%     207.25 ± 62%  interrupts.CPU62.PMI:Performance_monitoring_interrupts
      1816 ± 14%     +35.6%       2463 ± 29%  interrupts.CPU65.CAL:Function_call_interrupts
    142.25 ±100%     -66.3%      48.00 ± 10%  interrupts.CPU66.RES:Rescheduling_interrupts
    459.50 ± 10%     +42.2%     653.50 ± 16%  interrupts.CPU7.RES:Rescheduling_interrupts
      1282           +32.5%       1699 ± 27%  interrupts.CPU97.CAL:Function_call_interrupts
      1301 ±  2%     +26.9%       1650 ± 28%  interrupts.CPU98.CAL:Function_call_interrupts
     12.78 ±  2%      -1.9       10.92 ±  5%  perf-profile.calltrace.cycles-pp.ret_from_fork
     12.78 ±  2%      -1.9       10.92 ±  5%  perf-profile.calltrace.cycles-pp.kthread.ret_from_fork
      4.48 ±  6%      -1.6        2.90 ±  9%  perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork
      4.55 ±  6%      -1.6        2.98 ±  9%  perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.write_cache_pages.generic_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.wb_workfn.process_one_work.worker_thread.kthread.ret_from_fork
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.wb_writeback.wb_workfn.process_one_work.worker_thread.kthread
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work.worker_thread
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn.process_one_work
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback.wb_workfn
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb.wb_writeback
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.calltrace.cycles-pp.generic_writepages.do_writepages.__writeback_single_inode.writeback_sb_inodes.__writeback_inodes_wb
      3.07 ±  8%      -1.3        1.80 ± 10%  perf-profile.calltrace.cycles-pp.__writepage.write_cache_pages.generic_writepages.do_writepages.__writeback_single_inode
      2.98 ±  8%      -1.2        1.75 ± 10%  perf-profile.calltrace.cycles-pp.__block_write_full_page.__writepage.write_cache_pages.generic_writepages.do_writepages
      2.06 ±  8%      -0.9        1.20 ± 11%  perf-profile.calltrace.cycles-pp.submit_bh_wbc.__block_write_full_page.__writepage.write_cache_pages.generic_writepages
     14.52 ±  2%      -0.6       13.89        perf-profile.calltrace.cycles-pp.shrink_dentry_list.prune_dcache_sb.super_cache_scan.do_shrink_slab.shrink_slab
     16.80            -0.6       16.21        perf-profile.calltrace.cycles-pp.prune_dcache_sb.super_cache_scan.do_shrink_slab.shrink_slab.drop_slab_node
      1.34 ±  9%      -0.6        0.78 ±  8%  perf-profile.calltrace.cycles-pp.submit_bio.submit_bh_wbc.__block_write_full_page.__writepage.write_cache_pages
      1.29 ± 10%      -0.5        0.77 ±  9%  perf-profile.calltrace.cycles-pp.submit_bio_noacct.submit_bio.submit_bh_wbc.__block_write_full_page.__writepage
      0.94 ±  7%      -0.5        0.48 ± 59%  perf-profile.calltrace.cycles-pp.end_bio_bh_io_sync.blk_update_request.scsi_end_request.scsi_io_completion.blk_done_softirq
      1.23 ±  7%      -0.4        0.81 ± 14%  perf-profile.calltrace.cycles-pp.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn.kthread
      1.21 ±  7%      -0.4        0.81 ± 14%  perf-profile.calltrace.cycles-pp.scsi_io_completion.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd.smpboot_thread_fn
      1.21 ±  7%      -0.4        0.81 ± 14%  perf-profile.calltrace.cycles-pp.scsi_end_request.scsi_io_completion.blk_done_softirq.__softirqentry_text_start.run_ksoftirqd
      1.17 ±  7%      -0.4        0.77 ± 14%  perf-profile.calltrace.cycles-pp.blk_update_request.scsi_end_request.scsi_io_completion.blk_done_softirq.__softirqentry_text_start
      2.99 ±  2%      -0.2        2.79 ±  3%  perf-profile.calltrace.cycles-pp.__d_drop.__dentry_kill.shrink_dentry_list.prune_dcache_sb.super_cache_scan
      2.93 ±  2%      -0.2        2.73 ±  3%  perf-profile.calltrace.cycles-pp.___d_drop.__d_drop.__dentry_kill.shrink_dentry_list.prune_dcache_sb
      2.30            -0.1        2.18 ±  3%  perf-profile.calltrace.cycles-pp.shrink_lock_dentry.shrink_dentry_list.prune_dcache_sb.super_cache_scan.do_shrink_slab
      1.18 ±  4%      -0.1        1.09 ±  3%  perf-profile.calltrace.cycles-pp.rcu_cblist_dequeue.rcu_do_batch.rcu_core.__softirqentry_text_start.run_ksoftirqd
      0.56 ±  6%      +0.1        0.70 ±  8%  perf-profile.calltrace.cycles-pp.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.58 ±  4%      +0.1        0.73 ±  7%  perf-profile.calltrace.cycles-pp.ext4_discard_preallocations.ext4_clear_inode.ext4_evict_inode.evict.dispose_list
      0.79 ±  4%      +0.2        0.95 ± 10%  perf-profile.calltrace.cycles-pp.rcu_sched_clock_irq.update_process_times.tick_sched_handle.tick_sched_timer.__hrtimer_run_queues
      5.57            +0.3        5.88 ±  4%  perf-profile.calltrace.cycles-pp.evict.dispose_list.prune_icache_sb.super_cache_scan.do_shrink_slab
      7.17            +0.4        7.55 ±  2%  perf-profile.calltrace.cycles-pp.dispose_list.prune_icache_sb.super_cache_scan.do_shrink_slab.shrink_slab
      8.87            +0.5        9.33 ±  2%  perf-profile.calltrace.cycles-pp.prune_icache_sb.super_cache_scan.do_shrink_slab.shrink_slab.drop_slab_node
      5.66 ±  2%      +0.5        6.16 ±  4%  perf-profile.calltrace.cycles-pp.tick_sched_timer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack
      0.00            +0.6        0.57 ±  9%  perf-profile.calltrace.cycles-pp.timerqueue_del.__remove_hrtimer.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt
      8.54 ±  2%      +0.9        9.47 ±  3%  perf-profile.calltrace.cycles-pp.__hrtimer_run_queues.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt
     26.26            +1.1       27.37 ±  3%  perf-profile.calltrace.cycles-pp.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry
     24.24 ±  2%      +1.1       25.38 ±  3%  perf-profile.calltrace.cycles-pp.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter.do_idle
     14.93 ±  2%      +1.3       16.23 ±  2%  perf-profile.calltrace.cycles-pp.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state
     14.28 ±  2%      +1.4       15.63 ±  3%  perf-profile.calltrace.cycles-pp.hrtimer_interrupt.__sysvec_apic_timer_interrupt.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt
     15.05 ±  2%      +1.4       16.42 ±  2%  perf-profile.calltrace.cycles-pp.asm_call_sysvec_on_stack.sysvec_apic_timer_interrupt.asm_sysvec_apic_timer_interrupt.cpuidle_enter_state.cpuidle_enter
     53.89 ±  2%      +2.1       56.02        perf-profile.calltrace.cycles-pp.secondary_startup_64_no_verify
     53.09 ±  2%      +2.2       55.29        perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     42.65 ±  2%      +2.2       44.86        perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary
     43.61 ±  2%      +2.2       45.83        perf-profile.calltrace.cycles-pp.cpuidle_enter.do_idle.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     53.16 ±  2%      +2.2       55.40        perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.secondary_startup_64_no_verify
     53.16 ±  2%      +2.2       55.40        perf-profile.calltrace.cycles-pp.start_secondary.secondary_startup_64_no_verify
     12.81 ±  2%      -1.9       10.94 ±  5%  perf-profile.children.cycles-pp.ret_from_fork
     12.78 ±  2%      -1.9       10.92 ±  5%  perf-profile.children.cycles-pp.kthread
      4.48 ±  6%      -1.6        2.90 ±  9%  perf-profile.children.cycles-pp.process_one_work
      4.55 ±  6%      -1.6        2.98 ±  9%  perf-profile.children.cycles-pp.worker_thread
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.wb_workfn
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.wb_writeback
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.__writeback_inodes_wb
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.writeback_sb_inodes
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.__writeback_single_inode
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.do_writepages
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.generic_writepages
      3.70 ±  7%      -1.5        2.19 ± 10%  perf-profile.children.cycles-pp.write_cache_pages
      3.07 ±  8%      -1.3        1.80 ± 10%  perf-profile.children.cycles-pp.__writepage
      2.98 ±  8%      -1.2        1.75 ± 10%  perf-profile.children.cycles-pp.__block_write_full_page
      2.06 ±  8%      -0.9        1.20 ± 10%  perf-profile.children.cycles-pp.submit_bh_wbc
      1.78 ±  6%      -0.6        1.13 ± 17%  perf-profile.children.cycles-pp.blk_done_softirq
      1.76 ±  6%      -0.6        1.11 ± 17%  perf-profile.children.cycles-pp.scsi_io_completion
      1.76 ±  6%      -0.6        1.11 ± 17%  perf-profile.children.cycles-pp.scsi_end_request
     14.55 ±  2%      -0.6       13.92        perf-profile.children.cycles-pp.shrink_dentry_list
      1.68 ±  7%      -0.6        1.07 ± 17%  perf-profile.children.cycles-pp.blk_update_request
     16.80            -0.6       16.21        perf-profile.children.cycles-pp.prune_dcache_sb
      1.34 ± 10%      -0.6        0.78 ±  9%  perf-profile.children.cycles-pp.submit_bio
      1.29 ± 10%      -0.5        0.77 ±  8%  perf-profile.children.cycles-pp.submit_bio_noacct
      1.35 ±  7%      -0.5        0.84 ± 18%  perf-profile.children.cycles-pp.end_bio_bh_io_sync
      0.97 ±  8%      -0.3        0.62 ± 17%  perf-profile.children.cycles-pp.end_page_writeback
      0.79 ±  6%      -0.3        0.49 ±  9%  perf-profile.children.cycles-pp.blk_mq_submit_bio
      0.67 ± 12%      -0.3        0.40 ± 12%  perf-profile.children.cycles-pp.__test_set_page_writeback
      0.57 ±  8%      -0.2        0.35 ± 30%  perf-profile.children.cycles-pp.sysvec_call_function_single
      0.57 ±  8%      -0.2        0.35 ± 31%  perf-profile.children.cycles-pp.asm_sysvec_call_function_single
      3.01 ±  2%      -0.2        2.80 ±  3%  perf-profile.children.cycles-pp.__d_drop
      0.55 ± 10%      -0.2        0.34 ± 21%  perf-profile.children.cycles-pp.test_clear_page_writeback
      2.94 ±  2%      -0.2        2.75 ±  3%  perf-profile.children.cycles-pp.___d_drop
      0.42 ±  8%      -0.2        0.23 ± 17%  perf-profile.children.cycles-pp.bio_alloc_bioset
      0.40 ± 16%      -0.2        0.23 ± 14%  perf-profile.children.cycles-pp.submit_bio_checks
      0.51 ± 12%      -0.2        0.36 ±  8%  perf-profile.children.cycles-pp.kmem_cache_alloc
      0.32 ± 12%      -0.1        0.17 ± 19%  perf-profile.children.cycles-pp.mempool_alloc
      0.32 ±  6%      -0.1        0.18 ± 10%  perf-profile.children.cycles-pp.clear_page_dirty_for_io
      0.38 ±  8%      -0.1        0.25 ± 17%  perf-profile.children.cycles-pp.rotate_reclaimable_page
      2.31            -0.1        2.19 ±  3%  perf-profile.children.cycles-pp.shrink_lock_dentry
      0.45 ± 11%      -0.1        0.33 ±  5%  perf-profile.children.cycles-pp.try_to_wake_up
      0.28 ± 18%      -0.1        0.16 ± 27%  perf-profile.children.cycles-pp.end_buffer_async_write
      0.28 ±  7%      -0.1        0.18 ± 21%  perf-profile.children.cycles-pp.blk_attempt_plug_merge
      0.19 ± 15%      -0.1        0.09 ±  7%  perf-profile.children.cycles-pp.percpu_counter_add_batch
      0.16 ± 16%      -0.1        0.08 ± 68%  perf-profile.children.cycles-pp.__slab_alloc
      0.29 ± 11%      -0.1        0.21 ± 15%  perf-profile.children.cycles-pp.pagevec_lru_move_fn
      0.21 ± 21%      -0.1        0.13 ± 11%  perf-profile.children.cycles-pp.open64
      0.28 ± 14%      -0.1        0.20 ±  4%  perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
      0.15 ± 19%      -0.1        0.07 ± 71%  perf-profile.children.cycles-pp.fscrypt_drop_inode
      0.17 ± 13%      -0.1        0.10 ± 11%  perf-profile.children.cycles-pp.bio_attempt_back_merge
      0.15 ± 15%      -0.1        0.07 ± 67%  perf-profile.children.cycles-pp.___slab_alloc
      0.24 ± 14%      -0.1        0.16 ± 11%  perf-profile.children.cycles-pp.pagevec_move_tail_fn
      0.14 ± 21%      -0.1        0.07 ± 19%  perf-profile.children.cycles-pp.blk_throtl_bio
      0.21 ± 14%      -0.1        0.15 ±  9%  perf-profile.children.cycles-pp.blk_mq_dispatch_rq_list
      0.10 ± 14%      -0.1        0.04 ±101%  perf-profile.children.cycles-pp.allocate_slab
      0.12 ± 25%      -0.1        0.06 ± 26%  perf-profile.children.cycles-pp.__mod_lruvec_state
      0.20 ± 13%      -0.1        0.15 ± 11%  perf-profile.children.cycles-pp.scsi_queue_rq
      0.10 ± 25%      -0.1        0.05 ± 62%  perf-profile.children.cycles-pp.__close_nocancel
      0.08 ± 15%      -0.1        0.03 ±100%  perf-profile.children.cycles-pp.__split_vma
      0.17 ±  8%      -0.1        0.12 ±  9%  perf-profile.children.cycles-pp.can_stop_idle_tick
      0.15 ± 19%      -0.0        0.11 ± 28%  perf-profile.children.cycles-pp.get_page_from_freelist
      0.09 ± 20%      -0.0        0.05 ± 62%  perf-profile.children.cycles-pp.__vm_munmap
      0.15 ± 10%      -0.0        0.11 ± 11%  perf-profile.children.cycles-pp.schedule_timeout
      0.14 ± 13%      -0.0        0.10 ± 25%  perf-profile.children.cycles-pp.call_timer_fn
      0.09 ± 13%      -0.0        0.05 ± 58%  perf-profile.children.cycles-pp.enqueue_entity
      0.23 ±  7%      -0.0        0.20 ±  4%  perf-profile.children.cycles-pp.rcu_segcblist_enqueue
      0.23 ±  6%      -0.0        0.20 ±  5%  perf-profile.children.cycles-pp.rcu_gp_kthread
      0.17 ±  9%      -0.0        0.14 ±  3%  perf-profile.children.cycles-pp.tick_nohz_idle_got_tick
      0.10 ±  8%      -0.0        0.08 ± 19%  perf-profile.children.cycles-pp.enqueue_task_fair
      0.04 ± 60%      +0.0        0.08 ±  5%  perf-profile.children.cycles-pp.rcu_irq_enter
      0.06 ± 11%      +0.0        0.10 ± 12%  perf-profile.children.cycles-pp.arch_cpu_idle_exit
      0.14 ±  7%      +0.0        0.19 ± 16%  perf-profile.children.cycles-pp.update_dl_rq_load_avg
      0.07 ± 58%      +0.1        0.12 ± 12%  perf-profile.children.cycles-pp.delay_tsc
      0.44 ±  5%      +0.1        0.49 ±  4%  perf-profile.children.cycles-pp.truncate_inode_pages_final
      0.18 ± 26%      +0.1        0.23 ±  5%  perf-profile.children.cycles-pp.update_ts_time_stats
      0.00            +0.1        0.07 ± 17%  perf-profile.children.cycles-pp.perf_iterate_sb
      0.11 ± 17%      +0.1        0.19 ± 21%  perf-profile.children.cycles-pp.tick_program_event
      0.17 ± 17%      +0.1        0.24 ±  4%  perf-profile.children.cycles-pp.cpuidle_not_available
      0.46 ±  6%      +0.1        0.54 ±  6%  perf-profile.children.cycles-pp.__x86_retpoline_rax
      0.02 ±173%      +0.1        0.11 ± 25%  perf-profile.children.cycles-pp.cpuidle_get_cpu_driver
      0.80 ±  4%      +0.1        0.90 ±  2%  perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.58 ± 10%      +0.1        0.71 ±  7%  perf-profile.children.cycles-pp.enqueue_hrtimer
      1.74            +0.1        1.87 ±  4%  perf-profile.children.cycles-pp.__list_del_entry_valid
      0.45 ± 12%      +0.1        0.59 ±  6%  perf-profile.children.cycles-pp.timerqueue_add
      0.59 ±  4%      +0.1        0.73 ±  7%  perf-profile.children.cycles-pp.ext4_discard_preallocations
      0.87 ±  6%      +0.2        1.02 ± 10%  perf-profile.children.cycles-pp.rcu_sched_clock_irq
      0.53 ±  6%      +0.2        0.71 ±  9%  perf-profile.children.cycles-pp.timerqueue_del
      0.66 ±  9%      +0.2        0.84 ±  8%  perf-profile.children.cycles-pp.__remove_hrtimer
      0.26 ± 35%      +0.2        0.45 ± 18%  perf-profile.children.cycles-pp.timekeeping_max_deferment
      7.18            +0.4        7.55 ±  2%  perf-profile.children.cycles-pp.dispose_list
      5.14            +0.4        5.53 ±  3%  perf-profile.children.cycles-pp.kmem_cache_free
      8.87            +0.5        9.33 ±  2%  perf-profile.children.cycles-pp.prune_icache_sb
     10.50 ±  2%      +1.7       12.19 ± 11%  perf-profile.children.cycles-pp.__hrtimer_run_queues
     44.20 ±  2%      +2.1       46.30        perf-profile.children.cycles-pp.cpuidle_enter
     16.58 ±  2%      +2.1       18.70 ±  8%  perf-profile.children.cycles-pp.hrtimer_interrupt
     53.89 ±  2%      +2.1       56.02        perf-profile.children.cycles-pp.secondary_startup_64_no_verify
     53.89 ±  2%      +2.1       56.02        perf-profile.children.cycles-pp.cpu_startup_entry
     53.89 ±  2%      +2.1       56.02        perf-profile.children.cycles-pp.do_idle
     44.06 ±  2%      +2.2       46.23        perf-profile.children.cycles-pp.cpuidle_enter_state
     28.19 ±  2%      +2.2       30.37 ±  3%  perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
     17.26            +2.2       19.47 ±  7%  perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
     53.16 ±  2%      +2.2       55.40        perf-profile.children.cycles-pp.start_secondary
     29.75 ±  2%      +2.3       32.02 ±  3%  perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
      2.56 ±  2%      -0.3        2.25 ±  7%  perf-profile.self.cycles-pp.___d_drop
      0.77 ±  6%      -0.2        0.59 ± 10%  perf-profile.self.cycles-pp.tick_nohz_next_event
      0.12 ± 23%      -0.1        0.04 ±101%  perf-profile.self.cycles-pp.fscrypt_drop_inode
      0.16 ± 10%      -0.1        0.08 ± 10%  perf-profile.self.cycles-pp.percpu_counter_add_batch
      0.15 ± 22%      -0.1        0.09 ± 20%  perf-profile.self.cycles-pp.__test_set_page_writeback
      0.09 ± 14%      -0.1        0.03 ±100%  perf-profile.self.cycles-pp.clear_page_dirty_for_io
      0.17 ±  8%      -0.1        0.11 ± 21%  perf-profile.self.cycles-pp.__block_write_full_page
      0.19 ± 21%      -0.1        0.13 ±  3%  perf-profile.self.cycles-pp.kmem_cache_alloc
      0.19 ±  5%      -0.1        0.14 ± 10%  perf-profile.self.cycles-pp.cpuidle_governor_latency_req
      0.12 ±  7%      -0.1        0.07 ± 62%  perf-profile.self.cycles-pp.cpuidle_enter
      0.10 ± 14%      -0.1        0.05 ± 60%  perf-profile.self.cycles-pp.end_bio_bh_io_sync
      0.17 ±  8%      -0.1        0.12 ±  9%  perf-profile.self.cycles-pp.can_stop_idle_tick
      0.23 ±  7%      -0.0        0.19 ±  3%  perf-profile.self.cycles-pp.rcu_segcblist_enqueue
      0.08 ± 23%      -0.0        0.04 ± 63%  perf-profile.self.cycles-pp.find_get_pages_range_tag
      0.08 ±  6%      -0.0        0.04 ± 59%  perf-profile.self.cycles-pp.__d_drop
      0.09 ± 13%      +0.0        0.12 ± 15%  perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.10 ± 10%      +0.0        0.14 ±  5%  perf-profile.self.cycles-pp.tick_sched_handle
      0.36 ±  5%      +0.0        0.40 ±  2%  perf-profile.self.cycles-pp.__x86_retpoline_rax
      0.09 ± 27%      +0.0        0.13 ± 17%  perf-profile.self.cycles-pp.tick_nohz_tick_stopped
      0.16 ±  7%      +0.1        0.21 ± 16%  perf-profile.self.cycles-pp.timerqueue_del
      0.07 ± 58%      +0.1        0.12 ± 12%  perf-profile.self.cycles-pp.delay_tsc
      0.01 ±173%      +0.1        0.07 ±  7%  perf-profile.self.cycles-pp.arch_cpu_idle_exit
      0.18 ± 10%      +0.1        0.23 ± 17%  perf-profile.self.cycles-pp.update_blocked_averages
      0.13 ±  8%      +0.1        0.19 ± 16%  perf-profile.self.cycles-pp.update_dl_rq_load_avg
      0.11 ± 15%      +0.1        0.18 ± 22%  perf-profile.self.cycles-pp.tick_program_event
      0.00            +0.1        0.07 ± 17%  perf-profile.self.cycles-pp.rcu_irq_enter
      0.19 ±  5%      +0.1        0.26 ± 11%  perf-profile.self.cycles-pp.__hrtimer_get_next_event
      0.10 ± 27%      +0.1        0.17 ±  8%  perf-profile.self.cycles-pp.update_ts_time_stats
      0.27 ±  6%      +0.1        0.34 ±  8%  perf-profile.self.cycles-pp.__sysvec_apic_timer_interrupt
      0.16 ± 19%      +0.1        0.24 ±  3%  perf-profile.self.cycles-pp.cpuidle_not_available
      0.40 ±  6%      +0.1        0.48 ± 10%  perf-profile.self.cycles-pp.ext4_discard_preallocations
      0.02 ±173%      +0.1        0.10 ± 22%  perf-profile.self.cycles-pp.cpuidle_get_cpu_driver
      0.33 ± 11%      +0.1        0.42 ± 14%  perf-profile.self.cycles-pp.rb_erase
      0.20 ± 16%      +0.1        0.29 ±  7%  perf-profile.self.cycles-pp.timerqueue_add
      0.18 ±  8%      +0.1        0.28 ± 18%  perf-profile.self.cycles-pp.irq_exit_rcu
      0.71 ±  5%      +0.1        0.84 ±  2%  perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.67 ±  8%      +0.2        0.84 ± 12%  perf-profile.self.cycles-pp.rcu_sched_clock_irq
      0.25 ± 36%      +0.2        0.45 ± 18%  perf-profile.self.cycles-pp.timekeeping_max_deferment


                                                                                
                 fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.works_sec             
                                                                                
  23000 +-------------------------------------------------------------------+   
        |.+..+      +..+.+.+..+.+..+.+..+.+.+..+.+..+.+..+.+.+..+.+..+.+..+.|   
  22500 |-+                                                                 |   
  22000 |-+                                                                 |   
        |                                                                   |   
  21500 |-+                                                                 |   
        |           O  O           O        O  O      O  O        O         |   
  21000 |-+              O    O O    O  O O      O  O      O O  O           |   
        |                  O                                                |   
  20500 |-+                                                                 |   
  20000 |-+                                                                 |   
        |                                                                   |   
  19500 |-+  O    O                                                         |   
        | O    O                                                            |   
  19000 +-------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_sec             
                                                                                
  2.5 +---------------------------------------------------------------------+   
      |                         O                                           |   
      |       O            O O            O          O            O         |   
    2 |-+                               O         O         O               |   
      |  O                         O O                         O    O       |   
      |         O  O  O O                    O O       O  O                 |   
  1.5 |-+  O                                                                |   
      |                                                                     |   
    1 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
  0.5 |-+                                                                   |   
      |                                                                     |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
               fxmark.hdd_ext4_no_jnl_MWCL_2_bufferedio.iowait_util             
                                                                                
    4 +---------------------------------------------------------------------+   
      |                    O O  O                                 O         |   
  3.5 |-+     O                         O O       O  O      O               |   
    3 |-+                                                                   |   
      |  O            O            O O                         O    O       |   
  2.5 |-+  O    O  O    O                    O O       O  O                 |   
      |                                                                     |   
    2 |-+                                                                   |   
      |                                                                     |   
  1.5 |-+                                                                   |   
    1 |-+                                                                   |   
      |                                                                     |   
  0.5 |-+                                                                   |   
      |                                                                     |   
    0 +---------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                   fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works               
                                                                                
  320000 +------------------------------------------------------------------+   
         |. .. +  .+.+.+..    .+.+.+..+.+..+.+.+..+.+.+..+.  .+.+.+..+.+..+.|   
  310000 |-+    +.        +.+.                             +.               |   
         |                                                                  |   
  300000 |-+                                                                |   
         |                                                                  |   
  290000 |-+              O        O           O         O                  |   
         |           O O    O  O O      O  O O    O O O    O  O   O         |   
  280000 |-+                          O                         O           |   
         |                                                                  |   
  270000 |-+                                                                |   
         |    O                                                             |   
  260000 |-O    O  O                                                        |   
         |                                                                  |   
  250000 +------------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                 fxmark.hdd_ext4_no_jnl_MWCL_1_bufferedio.works_sec             
                                                                                
  10800 +-------------------------------------------------------------------+   
  10600 |-+  +        .+                           .+.  .+   +..            |   
        |. .. +  .+.+.  + .+..+.+..+.+..+.+.+..+.+.   +.  + +   +.+..+.+..+.|   
  10400 |-+    +.        +                                 +                |   
  10200 |-+                                                                 |   
  10000 |-+                                                                 |   
   9800 |-+                                                                 |   
        |                O         O                                        |   
   9600 |-+                   O           O O  O    O O  O O O    O         |   
   9400 |-+         O  O   O    O    O  O        O              O           |   
   9200 |-+                                                                 |   
   9000 |-+                                                                 |   
        |                                                                   |   
   8800 |-O  O O                                                            |   
   8600 +-------------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample



Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Oliver Sang


View attachment "config-5.11.0-rc6-00009-gef4eebad9c01" of type "text/plain" (174007 bytes)

View attachment "job-script" of type "text/plain" (8363 bytes)

View attachment "job.yaml" of type "text/plain" (5673 bytes)

View attachment "reproduce" of type "text/plain" (254 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ