lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200706012029.GZ3874@shao2-debian>
Date:   Mon, 6 Jul 2020 09:20:29 +0800
From:   kernel test robot <rong.a.chen@...el.com>
To:     Mel Gorman <mgorman@...hsingularity.net>
Cc:     Linus Torvalds <torvalds@...ux-foundation.org>,
        Maxim Levitsky <mlevitsk@...hat.com>,
        LKML <linux-kernel@...r.kernel.org>, lkp@...ts.01.org
Subject: [fs] b6509f6a8c: will-it-scale.per_thread_ops -12.6% regression

Greeting,

FYI, we noticed a -12.6% regression of will-it-scale.per_thread_ops due to commit:


commit: b6509f6a8c4313c068c69785c001451415969e44 ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

in testcase: will-it-scale
on test machine: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
with following parameters:

	nr_task: 100%
	mode: thread
	test: eventfd1
	cpufreq_governor: performance
	ucode: 0x5002f01

test-description: Will It Scale takes a testcase and runs it from 1 through to n parallel copies to see if the testcase will scale. It builds both a process and threads based test in order to see any differences between the two.
test-url: https://github.com/antonblanchard/will-it-scale

In addition to that, the commit also has significant impact on the following tests:

+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_process_ops -6.4% regression             |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=process                                                              |
|                  | nr_task=100%                                                              |
|                  | test=unix1                                                                |
|                  | ucode=0x5002f01                                                           |
+------------------+---------------------------------------------------------------------------+
| testcase: change | will-it-scale: will-it-scale.per_thread_ops -2.3% regression              |
| test machine     | 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory |
| test parameters  | cpufreq_governor=performance                                              |
|                  | mode=thread                                                               |
|                  | nr_task=100%                                                              |
|                  | test=pipe1                                                                |
|                  | ucode=0x5002f01                                                           |
+------------------+---------------------------------------------------------------------------+


If you fix the issue, kindly add following tag
Reported-by: kernel test robot <rong.a.chen@...el.com>


Details are as below:
-------------------------------------------------------------------------------------------------->


To reproduce:

        git clone https://github.com/intel/lkp-tests.git
        cd lkp-tests
        bin/lkp install job.yaml  # job file is attached in this email
        bin/lkp run     job.yaml

=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/eventfd1/will-it-scale/0x5002f01

commit: 
  v5.8-rc3
  b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"")

        v5.8-rc3 b6509f6a8c4313c068c69785c00 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   1652352           -12.6%    1444002 ±  2%  will-it-scale.per_thread_ops
 3.173e+08           -12.6%  2.772e+08 ±  2%  will-it-scale.workload
     23.95            -3.1       20.90 ±  2%  mpstat.cpu.all.usr%
     23.00           -12.0%      20.25 ±  4%  vmstat.cpu.us
    578849 ±  4%      +8.6%     628428 ±  4%  sched_debug.cpu.max_idle_balance_cost.max
      0.00 ±  9%     -14.4%       0.00 ± 13%  sched_debug.cpu.next_balance.stddev
      2207 ± 24%     +34.7%       2973 ± 20%  numa-vmstat.node0.nr_mapped
      6949 ± 22%     +43.7%       9984 ± 10%  numa-vmstat.node0.nr_slab_reclaimable
      9739 ± 13%     -29.0%       6913 ± 12%  numa-vmstat.node1.nr_slab_reclaimable
      7247 ± 13%     -28.4%       5190 ± 19%  slabinfo.eventpoll_pwq.active_objs
      7247 ± 13%     -28.4%       5190 ± 19%  slabinfo.eventpoll_pwq.num_objs
    960.00 ± 11%     -40.0%     576.00 ± 26%  slabinfo.kmalloc-rcl-128.active_objs
    960.00 ± 11%     -40.0%     576.00 ± 26%  slabinfo.kmalloc-rcl-128.num_objs
     27801 ± 22%     +43.7%      39940 ± 10%  numa-meminfo.node0.KReclaimable
      8718 ± 25%     +35.0%      11773 ± 20%  numa-meminfo.node0.Mapped
     27801 ± 22%     +43.7%      39940 ± 10%  numa-meminfo.node0.SReclaimable
     38959 ± 13%     -29.0%      27656 ± 12%  numa-meminfo.node1.KReclaimable
     38959 ± 13%     -29.0%      27656 ± 12%  numa-meminfo.node1.SReclaimable
    107406 ±  2%     -13.6%      92844 ±  6%  numa-meminfo.node1.Slab
      0.00            +1.1        1.09 ± 16%  perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +1.3        1.26 ± 18%  perf-profile.calltrace.cycles-pp.fsnotify_parent.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.00            +1.8        1.76 ± 21%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.7        3.70 ± 20%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +3.0        3.02 ± 17%  perf-profile.children.cycles-pp.fsnotify_parent
      0.00            +5.5        5.55 ± 20%  perf-profile.children.cycles-pp.fsnotify
      0.00            +2.7        2.66 ± 17%  perf-profile.self.cycles-pp.fsnotify_parent
      0.00            +5.4        5.35 ± 20%  perf-profile.self.cycles-pp.fsnotify
     10890 ± 21%     +53.7%      16738 ± 12%  interrupts.CPU168.CAL:Function_call_interrupts
      4602 ±  5%     +22.7%       5649 ± 13%  interrupts.CPU169.CAL:Function_call_interrupts
      4920 ± 13%     -21.3%       3874 ±  2%  interrupts.CPU170.CAL:Function_call_interrupts
    463.80 ± 11%     -25.1%     347.50 ±  4%  interrupts.CPU170.RES:Rescheduling_interrupts
      3619 ±  2%     +10.2%       3990 ±  9%  interrupts.CPU36.CAL:Function_call_interrupts
      5768 ± 34%     +43.3%       8267 ±  2%  interrupts.CPU74.NMI:Non-maskable_interrupts
      5768 ± 34%     +43.3%       8267 ±  2%  interrupts.CPU74.PMI:Performance_monitoring_interrupts
      5769 ± 34%     +43.3%       8268 ±  2%  interrupts.CPU75.NMI:Non-maskable_interrupts
      5769 ± 34%     +43.3%       8268 ±  2%  interrupts.CPU75.PMI:Performance_monitoring_interrupts
      5769 ± 34%     +43.3%       8267 ±  2%  interrupts.CPU76.NMI:Non-maskable_interrupts
      5769 ± 34%     +43.3%       8267 ±  2%  interrupts.CPU76.PMI:Performance_monitoring_interrupts
      5768 ± 34%     +43.3%       8268 ±  2%  interrupts.CPU77.NMI:Non-maskable_interrupts
      5768 ± 34%     +43.3%       8268 ±  2%  interrupts.CPU77.PMI:Performance_monitoring_interrupts
      3602 ±  2%     +12.9%       4068 ± 13%  interrupts.CPU81.CAL:Function_call_interrupts
      0.05 ± 32%    +121.5%       0.10 ± 39%  perf-stat.i.MPKI
      1.06            -0.1        0.94        perf-stat.i.branch-miss-rate%
 1.107e+09           -13.3%  9.596e+08 ±  2%  perf-stat.i.branch-misses
    199835           -11.5%     176905 ±  2%  perf-stat.i.dTLB-store-misses
 9.558e+10            +9.4%  1.046e+11 ±  2%  perf-stat.i.dTLB-stores
 9.687e+08           -13.8%  8.345e+08 ±  2%  perf-stat.i.iTLB-load-misses
    535.53           +20.3%     644.34        perf-stat.i.instructions-per-iTLB-miss
      1.10 ± 12%     +79.6%       1.98 ±  2%  perf-stat.i.metric.K/sec
      1.05            -0.1        0.92        perf-stat.overall.branch-miss-rate%
      0.00 ± 13%      -0.0        0.00 ±  4%  perf-stat.overall.dTLB-load-miss-rate%
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
    528.82           +20.8%     638.59        perf-stat.overall.instructions-per-iTLB-miss
    487060           +20.3%     586095        perf-stat.overall.path-length
 1.103e+09           -13.3%  9.564e+08 ±  2%  perf-stat.ps.branch-misses
    199661           -11.5%     176727 ±  2%  perf-stat.ps.dTLB-store-misses
 9.525e+10            +9.5%  1.043e+11 ±  2%  perf-stat.ps.dTLB-stores
  9.67e+08           -13.9%   8.33e+08 ±  3%  perf-stat.ps.iTLB-load-misses


                                                                                
                             will-it-scale.per_thread_ops                       
                                                                                
   1.7e+06 +----------------------------------------------------------------+   
           |       .+.. .+.+..          .+..        .+                      |   
  1.65e+06 |.+ .+.+    +      +.+.  .+.+    +. .+.+.                        |   
           | +.                   +.          +                             |   
   1.6e+06 |-+                                                              |   
           |                                                                |   
  1.55e+06 |-+                                                              |   
           |                                                                |   
   1.5e+06 |-+                                                              |   
           |                      O    O                           O O O  O |   
  1.45e+06 |-O  O O O  O O    O O    O   O  O O O O  O O O  O O O           |   
           |               O                                                |   
   1.4e+06 |-+                                                              |   
           |                                                                |   
  1.35e+06 +----------------------------------------------------------------+   
                                                                                
                                                                                                                                                                
                                will-it-scale.workload                          
                                                                                
  3.3e+08 +-----------------------------------------------------------------+   
          |                                                                 |   
  3.2e+08 |.+   .+.+..+.+..+.+.+..   .+..+.+.    .+.+                       |   
          | +..+                  +.+        +..+                           |   
  3.1e+08 |-+                                                               |   
          |                                                                 |   
    3e+08 |-+                                                               |   
          |                                                                 |   
  2.9e+08 |-+                                                               |   
          |                                                                 |   
  2.8e+08 |-O  O O    O O    O O  O O O  O O O  O O    O O  O O O  O O O  O |   
          |        O       O                        O                       |   
  2.7e+08 |-+                                                               |   
          |                                                                 |   
  2.6e+08 +-----------------------------------------------------------------+   
                                                                                
                                                                                
[*] bisect-good sample
[O] bisect-bad  sample

***************************************************************************************************
lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/process/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/unix1/will-it-scale/0x5002f01

commit: 
  v5.8-rc3
  b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"")

        v5.8-rc3 b6509f6a8c4313c068c69785c00 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
    720286            -6.4%     674507        will-it-scale.per_process_ops
 1.383e+08            -6.4%  1.295e+08        will-it-scale.workload
     97977            +1.4%      99342        proc-vmstat.nr_anon_pages
      8.31 ± 41%     +48.5%      12.33 ±  3%  sched_debug.cpu.sched_goidle.min
      1097 ±  8%     +19.8%       1315 ±  9%  slabinfo.task_group.active_objs
      1097 ±  8%     +19.8%       1315 ±  9%  slabinfo.task_group.num_objs
     13194 ± 25%     -29.6%       9295 ± 24%  softirqs.CPU124.RCU
      9178 ±  8%      -7.8%       8467 ±  4%  softirqs.CPU48.RCU
      1229 ± 68%    +156.3%       3151 ± 70%  interrupts.CPU0.CAL:Function_call_interrupts
      8259           -37.6%       5157 ± 34%  interrupts.CPU163.NMI:Non-maskable_interrupts
      8259           -37.6%       5157 ± 34%  interrupts.CPU163.PMI:Performance_monitoring_interrupts
     15337 ± 81%    +146.4%      37788 ± 45%  numa-vmstat.node2.nr_active_anon
     15187 ± 81%    +147.5%      37594 ± 45%  numa-vmstat.node2.nr_anon_pages
     15337 ± 81%    +146.4%      37788 ± 45%  numa-vmstat.node2.nr_zone_active_anon
    127540 ±  6%      -8.0%     117311        numa-vmstat.node2.numa_other
     62308 ± 81%    +144.0%     152033 ± 45%  numa-meminfo.node2.Active
     61336 ± 81%    +146.7%     151288 ± 45%  numa-meminfo.node2.Active(anon)
     15647 ±195%    +497.8%      93536 ± 57%  numa-meminfo.node2.AnonHugePages
     60733 ± 82%    +147.8%     150509 ± 45%  numa-meminfo.node2.AnonPages
    803985 ±  8%     +10.5%     888495 ±  9%  numa-meminfo.node2.MemUsed
      1.14            -0.1        1.07        perf-stat.i.branch-miss-rate%
 1.223e+09            -6.3%  1.146e+09        perf-stat.i.branch-misses
      0.99            -1.5%       0.98        perf-stat.i.cpi
 1.619e+11            +1.4%  1.641e+11        perf-stat.i.dTLB-loads
     1e+11            +4.0%   1.04e+11        perf-stat.i.dTLB-stores
  1.06e+09            -7.8%  9.779e+08        perf-stat.i.iTLB-load-misses
 5.531e+11            +1.3%  5.603e+11        perf-stat.i.instructions
    527.69            +9.7%     578.78        perf-stat.i.instructions-per-iTLB-miss
      1.01            +1.4%       1.02        perf-stat.i.ipc
      1931            +1.4%       1958        perf-stat.i.metric.M/sec
      1.12            -0.1        1.06        perf-stat.overall.branch-miss-rate%
      0.99            -1.4%       0.97        perf-stat.overall.cpi
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
    521.71            +9.8%     572.92        perf-stat.overall.instructions-per-iTLB-miss
      1.01            +1.4%       1.03        perf-stat.overall.ipc
   1206072            +8.1%    1303735        perf-stat.overall.path-length
 1.218e+09            -6.3%  1.142e+09        perf-stat.ps.branch-misses
 1.613e+11            +1.4%  1.635e+11        perf-stat.ps.dTLB-loads
  9.97e+10            +4.0%  1.036e+11        perf-stat.ps.dTLB-stores
 1.057e+09            -7.8%  9.747e+08        perf-stat.ps.iTLB-load-misses
 5.512e+11            +1.3%  5.584e+11        perf-stat.ps.instructions
 1.668e+14            +1.2%  1.688e+14        perf-stat.total.instructions
     38.39            -2.2       36.14        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     36.80            -2.1       34.66        perf-profile.calltrace.cycles-pp.sock_write_iter.new_sync_write.vfs_write.ksys_write.do_syscall_64
     34.46            -2.0       32.44        perf-profile.calltrace.cycles-pp.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write.ksys_write
     33.29            -1.8       31.47        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     30.96            -1.8       29.19        perf-profile.calltrace.cycles-pp.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write
     31.72            -1.7       30.05        perf-profile.calltrace.cycles-pp.sock_read_iter.new_sync_read.vfs_read.ksys_read.do_syscall_64
     25.95            -1.5       24.46        perf-profile.calltrace.cycles-pp.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read
     53.97            -1.5       52.49        perf-profile.calltrace.cycles-pp.write
     24.74            -1.4       23.30        perf-profile.calltrace.cycles-pp.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read.vfs_read
     47.12            -1.0       46.10        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.write
     46.34            -1.0       45.38        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     44.29            -0.9       43.43        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     42.43            -0.8       41.64        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.write
     13.05            -0.8       12.28        perf-profile.calltrace.cycles-pp.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
     11.31            -0.7       10.65        perf-profile.calltrace.cycles-pp.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
     10.31            -0.6        9.71        perf-profile.calltrace.cycles-pp.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg
      8.93            -0.5        8.39        perf-profile.calltrace.cycles-pp.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      8.64            -0.5        8.14        perf-profile.calltrace.cycles-pp.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      8.31            -0.5        7.85        perf-profile.calltrace.cycles-pp.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic.unix_stream_recvmsg
      7.91            -0.5        7.45        perf-profile.calltrace.cycles-pp.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      6.55            -0.4        6.19        perf-profile.calltrace.cycles-pp.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.54 ±  2%      -0.3        0.25 ±100%  perf-profile.calltrace.cycles-pp.___might_sleep.__might_fault._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg
      3.81            -0.2        3.58        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.write
      3.98            -0.2        3.75        perf-profile.calltrace.cycles-pp.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      3.63            -0.2        3.41        perf-profile.calltrace.cycles-pp._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      3.88            -0.2        3.65        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.read
      3.82            -0.2        3.60        perf-profile.calltrace.cycles-pp.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      2.87            -0.2        2.66        perf-profile.calltrace.cycles-pp.security_socket_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write.vfs_write
      3.71            -0.2        3.50        perf-profile.calltrace.cycles-pp.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic.unix_stream_recvmsg
      3.62            -0.2        3.42        perf-profile.calltrace.cycles-pp._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      3.91            -0.2        3.71        perf-profile.calltrace.cycles-pp.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor.unix_stream_read_generic
      2.95            -0.2        2.76        perf-profile.calltrace.cycles-pp.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read.vfs_read
      3.50            -0.2        3.31        perf-profile.calltrace.cycles-pp.__check_object_size.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      3.33            -0.2        3.15        perf-profile.calltrace.cycles-pp.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      3.44            -0.2        3.25        perf-profile.calltrace.cycles-pp.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb.unix_stream_read_generic
      3.09            -0.2        2.91        perf-profile.calltrace.cycles-pp.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      3.65            -0.2        3.47        perf-profile.calltrace.cycles-pp.__check_object_size.simple_copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      2.38            -0.2        2.21        perf-profile.calltrace.cycles-pp.security_file_permission.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.08            -0.1        1.94        perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      2.63            -0.1        2.50        perf-profile.calltrace.cycles-pp.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all.consume_skb
      3.44            -0.1        3.31        perf-profile.calltrace.cycles-pp.sock_recvmsg.sock_read_iter.new_sync_read.vfs_read.ksys_read
      1.71            -0.1        1.59        perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_read.ksys_read.do_syscall_64
      1.69            -0.1        1.58        perf-profile.calltrace.cycles-pp.common_file_perm.security_file_permission.vfs_write.ksys_write.do_syscall_64
      1.96            -0.1        1.85        perf-profile.calltrace.cycles-pp.aa_sk_perm.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read
      1.44            -0.1        1.34        perf-profile.calltrace.cycles-pp.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      1.57            -0.1        1.48        perf-profile.calltrace.cycles-pp.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg
      1.47            -0.1        1.39        perf-profile.calltrace.cycles-pp.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter.unix_stream_read_actor
      1.20            -0.1        1.13        perf-profile.calltrace.cycles-pp.sock_def_readable.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.70            -0.1        0.63 ±  4%  perf-profile.calltrace.cycles-pp.__might_sleep.kmem_cache_alloc_node.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb
      1.10 ±  2%      -0.1        1.03 ±  2%  perf-profile.calltrace.cycles-pp.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.98            -0.1        0.91        perf-profile.calltrace.cycles-pp.skb_set_owner_w.sock_alloc_send_pskb.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      1.01 ±  2%      -0.1        0.94 ±  3%  perf-profile.calltrace.cycles-pp.__fget_light.__fdget_pos.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      1.01            -0.1        0.94        perf-profile.calltrace.cycles-pp.__might_fault._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg.sock_sendmsg
      1.13            -0.1        1.07        perf-profile.calltrace.cycles-pp.unix_write_space.sock_wfree.unix_destruct_scm.skb_release_head_state.skb_release_all
      1.05            -0.1        0.99 ±  3%  perf-profile.calltrace.cycles-pp.__ksize.__alloc_skb.alloc_skb_with_frags.sock_alloc_send_pskb.unix_stream_sendmsg
      0.96            -0.1        0.91        perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.96            -0.1        0.90        perf-profile.calltrace.cycles-pp._raw_spin_unlock_irqrestore.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.98            -0.1        0.93        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.write
      1.04            -0.1        0.98        perf-profile.calltrace.cycles-pp.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      1.00            -0.1        0.94        perf-profile.calltrace.cycles-pp.syscall_return_via_sysret.read
      0.80            -0.1        0.75        perf-profile.calltrace.cycles-pp._raw_spin_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      1.02            -0.0        0.97        perf-profile.calltrace.cycles-pp.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter.new_sync_write
      0.85            -0.0        0.80        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_unlink.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter
      0.68 ±  2%      -0.0        0.64        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.71            -0.0        0.67        perf-profile.calltrace.cycles-pp.___might_sleep.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      0.70 ±  2%      -0.0        0.66 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyin._copy_from_iter.skb_copy_datagram_from_iter.unix_stream_sendmsg
      0.85            -0.0        0.81        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.skb_queue_tail.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.56            -0.0        0.52 ±  2%  perf-profile.calltrace.cycles-pp.apparmor_socket_recvmsg.security_socket_recvmsg.sock_recvmsg.sock_read_iter.new_sync_read
      0.57            -0.0        0.54        perf-profile.calltrace.cycles-pp.___might_sleep.wait_for_unix_gc.unix_stream_sendmsg.sock_sendmsg.sock_write_iter
      0.68            -0.0        0.64 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyout._copy_to_iter.__skb_datagram_iter.skb_copy_datagram_iter
      0.57            -0.0        0.54        perf-profile.calltrace.cycles-pp.__might_sleep.__kmalloc_node_track_caller.__kmalloc_reserve.__alloc_skb.alloc_skb_with_frags
      1.68            -0.0        1.64        perf-profile.calltrace.cycles-pp.mutex_lock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.65            -0.0        0.62        perf-profile.calltrace.cycles-pp.mutex_unlock.unix_stream_read_generic.unix_stream_recvmsg.sock_read_iter.new_sync_read
      0.00            +0.6        0.57        perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.6        0.62        perf-profile.calltrace.cycles-pp.fsnotify_parent.security_file_permission.vfs_read.ksys_read.do_syscall_64
      0.00            +0.6        0.64        perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.50            +0.7        3.20        perf-profile.calltrace.cycles-pp.security_file_permission.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      0.00            +0.9        0.93        perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     48.49            +1.3       49.77        perf-profile.calltrace.cycles-pp.read
     41.56            +1.7       43.30        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.read
     40.78            +1.8       42.56        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     38.81            +1.9       40.68        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
     37.05            +1.9       38.94        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.read
      0.00            +1.9        1.90        perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     38.49            -2.2       36.24        perf-profile.children.cycles-pp.new_sync_write
     36.86            -2.1       34.71        perf-profile.children.cycles-pp.sock_write_iter
     34.57            -2.0       32.53        perf-profile.children.cycles-pp.sock_sendmsg
     33.36            -1.8       31.55        perf-profile.children.cycles-pp.new_sync_read
     31.28            -1.8       29.50        perf-profile.children.cycles-pp.unix_stream_sendmsg
     31.77            -1.7       30.10        perf-profile.children.cycles-pp.sock_read_iter
     26.00            -1.5       24.50        perf-profile.children.cycles-pp.unix_stream_recvmsg
     54.00            -1.5       52.53        perf-profile.children.cycles-pp.write
     24.99            -1.5       23.53        perf-profile.children.cycles-pp.unix_stream_read_generic
     44.34            -0.9       43.48        perf-profile.children.cycles-pp.ksys_write
     13.14            -0.8       12.36        perf-profile.children.cycles-pp.sock_alloc_send_pskb
     42.57            -0.7       41.85        perf-profile.children.cycles-pp.vfs_write
     11.37            -0.7       10.71        perf-profile.children.cycles-pp.alloc_skb_with_frags
     10.51            -0.6        9.90        perf-profile.children.cycles-pp.__alloc_skb
      9.00            -0.5        8.46        perf-profile.children.cycles-pp.unix_stream_read_actor
      8.72            -0.5        8.21        perf-profile.children.cycles-pp.skb_copy_datagram_iter
      8.00            -0.5        7.53        perf-profile.children.cycles-pp.skb_copy_datagram_from_iter
      8.38            -0.5        7.91        perf-profile.children.cycles-pp.__skb_datagram_iter
      7.43            -0.4        7.04        perf-profile.children.cycles-pp.__check_object_size
      6.57            -0.4        6.21        perf-profile.children.cycles-pp.consume_skb
      5.14            -0.3        4.83        perf-profile.children.cycles-pp.entry_SYSCALL_64
      4.60            -0.3        4.32        perf-profile.children.cycles-pp.syscall_return_via_sysret
      4.16            -0.3        3.89        perf-profile.children.cycles-pp.aa_sk_perm
      4.22            -0.3        3.96        perf-profile.children.cycles-pp.___might_sleep
      3.50            -0.2        3.26        perf-profile.children.cycles-pp.common_file_perm
      4.07            -0.2        3.83        perf-profile.children.cycles-pp.__kmalloc_reserve
      3.71            -0.2        3.48        perf-profile.children.cycles-pp._copy_from_iter
      3.85            -0.2        3.63        perf-profile.children.cycles-pp.skb_release_all
      2.95            -0.2        2.73        perf-profile.children.cycles-pp.__might_sleep
      2.91            -0.2        2.70        perf-profile.children.cycles-pp.security_socket_sendmsg
      3.72            -0.2        3.51        perf-profile.children.cycles-pp._copy_to_iter
      3.76            -0.2        3.55        perf-profile.children.cycles-pp.skb_release_head_state
      3.01            -0.2        2.81        perf-profile.children.cycles-pp.security_socket_recvmsg
      3.54            -0.2        3.33        perf-profile.children.cycles-pp.__kmalloc_node_track_caller
      3.97            -0.2        3.77        perf-profile.children.cycles-pp.simple_copy_to_iter
      3.28            -0.2        3.09        perf-profile.children.cycles-pp.kmem_cache_alloc_node
      3.54            -0.2        3.35        perf-profile.children.cycles-pp.unix_destruct_scm
      2.36 ±  2%      -0.2        2.21 ±  2%  perf-profile.children.cycles-pp.__fdget_pos
      2.64            -0.1        2.50        perf-profile.children.cycles-pp.sock_wfree
      3.50            -0.1        3.37        perf-profile.children.cycles-pp.sock_recvmsg
      2.01 ±  2%      -0.1        1.88 ±  3%  perf-profile.children.cycles-pp.__fget_light
      2.10 ±  2%      -0.1        1.97 ±  2%  perf-profile.children.cycles-pp.__might_fault
      1.97            -0.1        1.85        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      1.54            -0.1        1.44        perf-profile.children.cycles-pp.wait_for_unix_gc
      1.95            -0.1        1.85        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      1.43            -0.1        1.34        perf-profile.children.cycles-pp._cond_resched
      1.60            -0.1        1.51        perf-profile.children.cycles-pp.copyin
      1.52            -0.1        1.43        perf-profile.children.cycles-pp.__prepare_exit_to_usermode
      1.67 ±  3%      -0.1        1.58 ±  2%  perf-profile.children.cycles-pp._raw_spin_lock
      1.64            -0.1        1.55        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      1.50            -0.1        1.42        perf-profile.children.cycles-pp.copyout
      1.21            -0.1        1.13        perf-profile.children.cycles-pp.sock_def_readable
      0.98            -0.1        0.91        perf-profile.children.cycles-pp.skb_set_owner_w
      0.95            -0.1        0.89        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      1.05            -0.1        0.99 ±  2%  perf-profile.children.cycles-pp.__ksize
      0.84            -0.1        0.78 ±  2%  perf-profile.children.cycles-pp.__x86_indirect_thunk_rax
      1.13            -0.1        1.08        perf-profile.children.cycles-pp.unix_write_space
      1.05            -0.1        0.99        perf-profile.children.cycles-pp.skb_unlink
      0.74 ±  2%      -0.1        0.69        perf-profile.children.cycles-pp.apparmor_file_permission
      1.02            -0.1        0.97        perf-profile.children.cycles-pp.skb_queue_tail
      1.75            -0.0        1.70        perf-profile.children.cycles-pp.mutex_lock
      0.56            -0.0        0.52 ±  2%  perf-profile.children.cycles-pp.apparmor_socket_recvmsg
      0.71            -0.0        0.67        perf-profile.children.cycles-pp.rcu_all_qs
      0.67            -0.0        0.63        perf-profile.children.cycles-pp.__syscall_return_slowpath
      0.67            -0.0        0.64        perf-profile.children.cycles-pp.check_stack_object
      0.44            -0.0        0.41        perf-profile.children.cycles-pp.apparmor_socket_sendmsg
      0.67            -0.0        0.65        perf-profile.children.cycles-pp.mutex_unlock
      0.30            -0.0        0.28 ±  2%  perf-profile.children.cycles-pp.put_pid
      0.23            -0.0        0.21 ±  2%  perf-profile.children.cycles-pp.skb_release_data
      0.43            +0.0        0.46        perf-profile.children.cycles-pp.apparmor_socket_getpeersec_dgram
      4.96            +0.6        5.57        perf-profile.children.cycles-pp.security_file_permission
     88.79            +0.7       89.51        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     87.30            +0.8       88.11        perf-profile.children.cycles-pp.do_syscall_64
     48.55            +1.3       49.85        perf-profile.children.cycles-pp.read
      0.00            +1.9        1.87        perf-profile.children.cycles-pp.fsnotify_parent
     38.85            +1.9       40.72        perf-profile.children.cycles-pp.ksys_read
     37.17            +2.0       39.17        perf-profile.children.cycles-pp.vfs_read
      0.00            +2.9        2.90        perf-profile.children.cycles-pp.fsnotify
      4.56            -0.3        4.28        perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.10            -0.2        3.85        perf-profile.self.cycles-pp.___might_sleep
      2.96            -0.2        2.75        perf-profile.self.cycles-pp.unix_stream_read_generic
      2.65            -0.2        2.46        perf-profile.self.cycles-pp.__might_sleep
      2.94            -0.2        2.75        perf-profile.self.cycles-pp.unix_stream_sendmsg
      2.85 ±  2%      -0.2        2.67        perf-profile.self.cycles-pp.common_file_perm
      1.40 ±  4%      -0.2        1.22        perf-profile.self.cycles-pp.new_sync_read
      2.32            -0.2        2.17        perf-profile.self.cycles-pp.aa_sk_perm
      2.56            -0.1        2.41        perf-profile.self.cycles-pp.entry_SYSCALL_64
      1.90 ±  2%      -0.1        1.77 ±  3%  perf-profile.self.cycles-pp.__fget_light
      2.13            -0.1        2.00        perf-profile.self.cycles-pp.__alloc_skb
      2.29            -0.1        2.17        perf-profile.self.cycles-pp.sock_write_iter
      1.93            -0.1        1.81        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      1.79            -0.1        1.67        perf-profile.self.cycles-pp.write
      1.83            -0.1        1.72        perf-profile.self.cycles-pp.read
      1.52            -0.1        1.42        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      1.71            -0.1        1.61        perf-profile.self.cycles-pp.__kmalloc_node_track_caller
      1.93            -0.1        1.84        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      1.60 ±  3%      -0.1        1.52 ±  2%  perf-profile.self.cycles-pp._raw_spin_lock
      1.81            -0.1        1.73        perf-profile.self.cycles-pp.kmem_cache_alloc_node
      1.17            -0.1        1.09        perf-profile.self.cycles-pp.sock_def_readable
      1.55            -0.1        1.47        perf-profile.self.cycles-pp.copy_user_generic_unrolled
      1.25 ±  2%      -0.1        1.17 ±  2%  perf-profile.self.cycles-pp._copy_to_iter
      0.61            -0.1        0.54        perf-profile.self.cycles-pp.ksys_write
      1.44            -0.1        1.37        perf-profile.self.cycles-pp.sock_wfree
      0.95            -0.1        0.88        perf-profile.self.cycles-pp.skb_set_owner_w
      1.06            -0.1        0.99 ±  2%  perf-profile.self.cycles-pp._copy_from_iter
      0.58 ±  5%      -0.1        0.52 ±  5%  perf-profile.self.cycles-pp.aa_file_perm
      1.04            -0.1        0.97 ±  2%  perf-profile.self.cycles-pp.__ksize
      1.10            -0.1        1.04        perf-profile.self.cycles-pp.unix_write_space
      0.68 ±  2%      -0.1        0.62        perf-profile.self.cycles-pp.__skb_datagram_iter
      0.93            -0.1        0.87        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.52            -0.1        0.47        perf-profile.self.cycles-pp.ksys_read
      0.68 ±  2%      -0.1        0.63 ±  2%  perf-profile.self.cycles-pp.apparmor_file_permission
      0.83            -0.0        0.78        perf-profile.self.cycles-pp.unix_destruct_scm
      0.87            -0.0        0.82        perf-profile.self.cycles-pp.consume_skb
      0.83            -0.0        0.79        perf-profile.self.cycles-pp.sock_alloc_send_pskb
      0.71            -0.0        0.67        perf-profile.self.cycles-pp._cond_resched
      0.82            -0.0        0.78        perf-profile.self.cycles-pp.alloc_skb_with_frags
      0.29 ±  3%      -0.0        0.25        perf-profile.self.cycles-pp.unix_stream_read_actor
      0.66            -0.0        0.62        perf-profile.self.cycles-pp.skb_copy_datagram_from_iter
      0.53            -0.0        0.49 ±  2%  perf-profile.self.cycles-pp.apparmor_socket_recvmsg
      0.64            -0.0        0.60        perf-profile.self.cycles-pp.__syscall_return_slowpath
      0.60 ±  2%      -0.0        0.56 ±  4%  perf-profile.self.cycles-pp.__might_fault
      0.50            -0.0        0.46        perf-profile.self.cycles-pp.__kmalloc_reserve
      0.37            -0.0        0.34        perf-profile.self.cycles-pp.security_socket_recvmsg
      2.25            -0.0        2.21        perf-profile.self.cycles-pp.sock_read_iter
      0.54            -0.0        0.51        perf-profile.self.cycles-pp.rcu_all_qs
      0.37 ±  5%      -0.0        0.34 ±  4%  perf-profile.self.cycles-pp.sock_sendmsg
      0.43            -0.0        0.40        perf-profile.self.cycles-pp.__x86_indirect_thunk_rax
      0.98            -0.0        0.95        perf-profile.self.cycles-pp.unix_stream_recvmsg
      0.35            -0.0        0.32        perf-profile.self.cycles-pp.skb_copy_datagram_iter
      0.41            -0.0        0.38        perf-profile.self.cycles-pp.apparmor_socket_sendmsg
      0.56            -0.0        0.54        perf-profile.self.cycles-pp.check_stack_object
      0.44            -0.0        0.41        perf-profile.self.cycles-pp.wait_for_unix_gc
      0.29 ±  3%      -0.0        0.26 ±  2%  perf-profile.self.cycles-pp.security_socket_sendmsg
      0.42            -0.0        0.40        perf-profile.self.cycles-pp.security_socket_getpeersec_dgram
      0.63            -0.0        0.61        perf-profile.self.cycles-pp.mutex_unlock
      0.40 ±  2%      -0.0        0.38        perf-profile.self.cycles-pp.__fdget_pos
      0.31            -0.0        0.29        perf-profile.self.cycles-pp.copyin
      0.23            -0.0        0.21 ±  2%  perf-profile.self.cycles-pp.skb_release_data
      0.19            -0.0        0.17 ±  2%  perf-profile.self.cycles-pp.skb_release_head_state
      0.24            -0.0        0.23        perf-profile.self.cycles-pp.copyout
      0.21 ±  2%      -0.0        0.19 ±  2%  perf-profile.self.cycles-pp.simple_copy_to_iter
      0.26            -0.0        0.25        perf-profile.self.cycles-pp.kmalloc_slab
      0.17 ±  2%      +0.0        0.20 ±  2%  perf-profile.self.cycles-pp.__vfs_read
      0.40            +0.0        0.43        perf-profile.self.cycles-pp.apparmor_socket_getpeersec_dgram
      0.73            +0.0        0.77        perf-profile.self.cycles-pp.mutex_lock
      0.51            +0.1        0.57        perf-profile.self.cycles-pp.sock_recvmsg
      1.33            +0.2        1.57        perf-profile.self.cycles-pp.vfs_write
      0.75 ±  6%      +0.4        1.10 ±  3%  perf-profile.self.cycles-pp.security_file_permission
      0.97            +0.5        1.49 ±  2%  perf-profile.self.cycles-pp.vfs_read
      0.00            +1.7        1.65        perf-profile.self.cycles-pp.fsnotify_parent
      0.00            +2.8        2.80        perf-profile.self.cycles-pp.fsnotify



***************************************************************************************************
lkp-csl-2ap4: 192 threads Intel(R) Xeon(R) Platinum 9242 CPU @ 2.30GHz with 192G memory
=========================================================================================
compiler/cpufreq_governor/kconfig/mode/nr_task/rootfs/tbox_group/test/testcase/ucode:
  gcc-9/performance/x86_64-rhel-7.6/thread/100%/debian-x86_64-20191114.cgz/lkp-csl-2ap4/pipe1/will-it-scale/0x5002f01

commit: 
  v5.8-rc3
  b6509f6a8c ("Revert "fs: Do not check if there is a fsnotify watcher on pseudo inodes"")

        v5.8-rc3 b6509f6a8c4313c068c69785c00 
---------------- --------------------------- 
         %stddev     %change         %stddev
             \          |                \  
   1045643            -2.3%    1021331        will-it-scale.per_thread_ops
 2.008e+08            -2.3%  1.961e+08        will-it-scale.workload
  29490176 ±  3%      -9.8%   26607616 ±  3%  meminfo.DirectMap2M
    202721 ±  4%     +11.5%     225943 ±  3%  numa-numastat.node2.local_node
     25892 ± 28%     -75.5%       6341 ± 97%  numa-numastat.node2.other_node
      1843            +8.9%       2007 ±  5%  numa-vmstat.node2.nr_mapped
    130681 ±  5%     -14.7%     111436 ±  5%  numa-vmstat.node2.numa_other
      9801 ±  5%     +16.0%      11370 ±  7%  sched_debug.cpu.ttwu_count.max
      3052 ± 12%     +23.6%       3771 ± 11%  sched_debug.cpu.ttwu_local.max
    136156            +1.4%     138002        proc-vmstat.nr_active_anon
    109009            +1.1%     110229        proc-vmstat.nr_anon_pages
    136156            +1.4%     138002        proc-vmstat.nr_zone_active_anon
   1120468            +1.6%    1137854        proc-vmstat.pgalloc_normal
    337.83 ± 61%     +53.8%     519.75 ± 47%  interrupts.39:PCI-MSI.524290-edge.eth0-TxRx-1
    337.83 ± 61%     +53.8%     519.75 ± 47%  interrupts.CPU11.39:PCI-MSI.524290-edge.eth0-TxRx-1
    303.33           +45.7%     442.00 ± 44%  interrupts.CPU148.RES:Rescheduling_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU155.NMI:Non-maskable_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU155.PMI:Performance_monitoring_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU157.NMI:Non-maskable_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU157.PMI:Performance_monitoring_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU158.NMI:Non-maskable_interrupts
      5509 ± 35%     +50.1%       8271        interrupts.CPU158.PMI:Performance_monitoring_interrupts
     15130 ± 14%     -18.1%      12396 ±  8%  interrupts.CPU168.CAL:Function_call_interrupts
      4137 ± 15%     +44.3%       5970 ± 26%  interrupts.CPU48.CAL:Function_call_interrupts
    336.00 ±  6%     +60.0%     537.75 ± 31%  interrupts.CPU48.RES:Rescheduling_interrupts
      0.75            -0.0        0.73        perf-stat.i.branch-miss-rate%
 8.192e+08            -2.2%   8.01e+08        perf-stat.i.branch-misses
      0.98            -1.4%       0.97        perf-stat.i.cpi
   1.7e+11            +1.5%  1.725e+11        perf-stat.i.dTLB-loads
 1.055e+11            +2.7%  1.084e+11        perf-stat.i.dTLB-stores
 7.304e+08            -2.5%  7.123e+08        perf-stat.i.iTLB-load-misses
   5.6e+11            +1.6%  5.688e+11        perf-stat.i.instructions
    775.04            +4.2%     807.76        perf-stat.i.instructions-per-iTLB-miss
      1.02            +1.5%       1.04        perf-stat.i.ipc
      2015            +1.5%       2046        perf-stat.i.metric.M/sec
      0.73            -0.0        0.72        perf-stat.overall.branch-miss-rate%
      0.98            -1.5%       0.96        perf-stat.overall.cpi
      0.00            -0.0        0.00        perf-stat.overall.dTLB-store-miss-rate%
    764.55            +4.1%     796.24        perf-stat.overall.instructions-per-iTLB-miss
      1.02            +1.5%       1.04        perf-stat.overall.ipc
    841233            +4.0%     874479        perf-stat.overall.path-length
 8.164e+08            -2.2%  7.983e+08        perf-stat.ps.branch-misses
 1.695e+11            +1.5%   1.72e+11        perf-stat.ps.dTLB-loads
 1.052e+11            +2.7%   1.08e+11        perf-stat.ps.dTLB-stores
 7.301e+08            -2.5%  7.119e+08        perf-stat.ps.iTLB-load-misses
 5.582e+11            +1.6%  5.668e+11        perf-stat.ps.instructions
 1.689e+14            +1.5%  1.715e+14        perf-stat.total.instructions
     50.20            -1.4       48.82        perf-profile.calltrace.cycles-pp.__GI___libc_read
     40.68            -1.2       39.52        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     39.53            -1.1       38.39        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     36.75            -1.1       35.70        perf-profile.calltrace.cycles-pp.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     32.27            -0.9       31.36        perf-profile.calltrace.cycles-pp.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
     24.69            -0.6       24.14        perf-profile.calltrace.cycles-pp.new_sync_write.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     22.23            -0.5       21.72        perf-profile.calltrace.cycles-pp.pipe_write.new_sync_write.vfs_write.ksys_write.do_syscall_64
     21.04            -0.5       20.55        perf-profile.calltrace.cycles-pp.new_sync_read.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
     18.70            -0.4       18.27        perf-profile.calltrace.cycles-pp.pipe_read.new_sync_read.vfs_read.ksys_read.do_syscall_64
      6.80            -0.1        6.65        perf-profile.calltrace.cycles-pp.copy_page_from_iter.pipe_write.new_sync_write.vfs_write.ksys_write
      5.39            -0.1        5.26        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_write
      2.96            -0.1        2.84 ±  2%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_read.ksys_read.do_syscall_64.entry_SYSCALL_64_after_hwframe
      2.27            -0.1        2.15        perf-profile.calltrace.cycles-pp.copyin.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
      2.42            -0.1        2.31 ±  2%  perf-profile.calltrace.cycles-pp.copyout.copy_page_to_iter.pipe_read.new_sync_read.vfs_read
      5.38            -0.1        5.28        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64.__GI___libc_read
      2.61            -0.1        2.52        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_read.new_sync_read.vfs_read.ksys_read
      3.46            -0.1        3.38        perf-profile.calltrace.cycles-pp.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      2.62            -0.1        2.55        perf-profile.calltrace.cycles-pp.mutex_lock.pipe_write.new_sync_write.vfs_write.ksys_write
      2.08            -0.1        2.02        perf-profile.calltrace.cycles-pp.__pthread_disable_asynccancel
      0.99            -0.1        0.94        perf-profile.calltrace.cycles-pp.fput_many.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      1.04            -0.0        1.00        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
      0.97            -0.0        0.93 ±  2%  perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyin.copy_page_from_iter.pipe_write.new_sync_write
      1.23            -0.0        1.19        perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up_common_lock.pipe_write.new_sync_write.vfs_write
      1.11            -0.0        1.08        perf-profile.calltrace.cycles-pp.copy_user_generic_unrolled.copyout.copy_page_to_iter.pipe_read.new_sync_read
      1.20            -0.0        1.17        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_write.new_sync_write.vfs_write.ksys_write
      1.01            -0.0        0.98        perf-profile.calltrace.cycles-pp.__prepare_exit_to_usermode.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_read
      0.79            -0.0        0.76        perf-profile.calltrace.cycles-pp.___might_sleep.mutex_lock.pipe_read.new_sync_read.vfs_read
      1.50            -0.0        1.47        perf-profile.calltrace.cycles-pp.__might_fault.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
      1.13            -0.0        1.11        perf-profile.calltrace.cycles-pp._raw_spin_lock_irq.pipe_read.new_sync_read.vfs_read.ksys_read
      0.79            -0.0        0.77        perf-profile.calltrace.cycles-pp.___might_sleep.copy_page_from_iter.pipe_write.new_sync_write.vfs_write
      0.72            -0.0        0.70        perf-profile.calltrace.cycles-pp.__sb_end_write.pipe_write.new_sync_write.vfs_write.ksys_write
      0.99            -0.0        0.97        perf-profile.calltrace.cycles-pp.mutex_unlock.pipe_read.new_sync_read.vfs_read.ksys_read
      0.00            +0.9        0.92        perf-profile.calltrace.cycles-pp.fsnotify_parent.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     49.09            +1.4       50.48        perf-profile.calltrace.cycles-pp.__GI___libc_write
      0.00            +1.5        1.50 ±  5%  perf-profile.calltrace.cycles-pp.fsnotify.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe
     39.33            +1.6       40.95        perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     38.18            +1.7       39.84        perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     35.19            +1.8       37.00        perf-profile.calltrace.cycles-pp.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     30.77            +1.8       32.61        perf-profile.calltrace.cycles-pp.vfs_write.ksys_write.do_syscall_64.entry_SYSCALL_64_after_hwframe.__GI___libc_write
     50.18            -1.4       48.80        perf-profile.children.cycles-pp.__GI___libc_read
     36.87            -1.1       35.81        perf-profile.children.cycles-pp.ksys_read
     32.66            -0.9       31.73        perf-profile.children.cycles-pp.vfs_read
     24.84            -0.6       24.29        perf-profile.children.cycles-pp.new_sync_write
     22.51            -0.5       22.00        perf-profile.children.cycles-pp.pipe_write
     21.18            -0.5       20.69        perf-profile.children.cycles-pp.new_sync_read
     18.94            -0.4       18.50        perf-profile.children.cycles-pp.pipe_read
      5.44            -0.2        5.27        perf-profile.children.cycles-pp.mutex_lock
      7.06            -0.2        6.90        perf-profile.children.cycles-pp.copy_page_from_iter
      6.34            -0.2        6.19        perf-profile.children.cycles-pp.syscall_return_via_sysret
      7.28            -0.1        7.13        perf-profile.children.cycles-pp.entry_SYSCALL_64
      2.33            -0.1        2.21        perf-profile.children.cycles-pp.copyin
      7.14            -0.1        7.03        perf-profile.children.cycles-pp.copy_page_to_iter
      2.48            -0.1        2.36 ±  2%  perf-profile.children.cycles-pp.copyout
      4.78            -0.1        4.67        perf-profile.children.cycles-pp.___might_sleep
      2.48            -0.1        2.39        perf-profile.children.cycles-pp.copy_user_generic_unrolled
      2.23            -0.1        2.14        perf-profile.children.cycles-pp.__prepare_exit_to_usermode
      3.56            -0.1        3.48        perf-profile.children.cycles-pp.__wake_up_common_lock
      2.69            -0.1        2.63        perf-profile.children.cycles-pp.__might_sleep
      2.41            -0.1        2.35        perf-profile.children.cycles-pp._raw_spin_lock_irq
      2.08            -0.1        2.02        perf-profile.children.cycles-pp.__pthread_disable_asynccancel
      1.88            -0.1        1.82        perf-profile.children.cycles-pp.fput_many
      1.40            -0.0        1.35        perf-profile.children.cycles-pp._raw_spin_lock_irqsave
      0.34            -0.0        0.29        perf-profile.children.cycles-pp.fput
      1.38            -0.0        1.34        perf-profile.children.cycles-pp.copy_user_enhanced_fast_string
      0.24 ±  7%      -0.0        0.22        perf-profile.children.cycles-pp.tick_sched_handle
      0.92            -0.0        0.90        perf-profile.children.cycles-pp.fpregs_assert_state_consistent
      1.28            -0.0        1.26        perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
      0.75            -0.0        0.73        perf-profile.children.cycles-pp.__sb_end_write
     80.17            +0.5       80.63        perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
     77.95            +0.5       78.47        perf-profile.children.cycles-pp.do_syscall_64
      1.98            +0.9        2.88        perf-profile.children.cycles-pp.fsnotify_parent
      3.04            +1.4        4.42 ±  2%  perf-profile.children.cycles-pp.fsnotify
     49.09            +1.4       50.48        perf-profile.children.cycles-pp.__GI___libc_write
     35.45            +1.8       37.22        perf-profile.children.cycles-pp.ksys_write
     30.96            +2.0       32.93        perf-profile.children.cycles-pp.vfs_write
      6.28            -0.1        6.13        perf-profile.self.cycles-pp.syscall_return_via_sysret
      4.68            -0.1        4.58        perf-profile.self.cycles-pp.___might_sleep
      2.35            -0.1        2.26        perf-profile.self.cycles-pp.copy_user_generic_unrolled
      2.26            -0.1        2.18        perf-profile.self.cycles-pp.mutex_lock
      2.61            -0.1        2.54        perf-profile.self.cycles-pp.__GI___libc_write
      3.09            -0.1        3.02        perf-profile.self.cycles-pp.pipe_write
      2.57            -0.1        2.50        perf-profile.self.cycles-pp.__GI___libc_read
      2.26            -0.1        2.19        perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
      3.74            -0.1        3.68        perf-profile.self.cycles-pp.entry_SYSCALL_64
      0.93            -0.1        0.87        perf-profile.self.cycles-pp.ksys_write
      1.38            -0.1        1.32        perf-profile.self.cycles-pp.__prepare_exit_to_usermode
      2.31            -0.1        2.25        perf-profile.self.cycles-pp._raw_spin_lock_irq
      1.78            -0.1        1.72        perf-profile.self.cycles-pp.fput_many
      1.99            -0.1        1.94        perf-profile.self.cycles-pp.__pthread_disable_asynccancel
      2.37            -0.0        2.32        perf-profile.self.cycles-pp.__might_sleep
      1.14            -0.0        1.10        perf-profile.self.cycles-pp.do_syscall_64
      1.36            -0.0        1.31        perf-profile.self.cycles-pp._raw_spin_lock_irqsave
      2.22            -0.0        2.18        perf-profile.self.cycles-pp.__pthread_enable_asynccancel
      1.87            -0.0        1.83        perf-profile.self.cycles-pp.mutex_unlock
      1.34            -0.0        1.31        perf-profile.self.cycles-pp.copy_user_enhanced_fast_string
      0.91            -0.0        0.89        perf-profile.self.cycles-pp.file_update_time
      0.85            -0.0        0.82        perf-profile.self.cycles-pp.fpregs_assert_state_consistent
      0.44            -0.0        0.41 ±  2%  perf-profile.self.cycles-pp.touch_atime
      0.73            -0.0        0.71        perf-profile.self.cycles-pp.__sb_end_write
      1.23            -0.0        1.21        perf-profile.self.cycles-pp._raw_spin_unlock_irqrestore
      0.21            -0.0        0.20        perf-profile.self.cycles-pp.kill_fasync
      2.08 ±  2%      +0.4        2.49 ±  4%  perf-profile.self.cycles-pp.vfs_write
      1.71            +0.8        2.51        perf-profile.self.cycles-pp.fsnotify_parent
      2.94            +1.3        4.25 ±  2%  perf-profile.self.cycles-pp.fsnotify





Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.


Thanks,
Rong Chen


View attachment "config-5.8.0-rc3-00001-gb6509f6a8c431" of type "text/plain" (206273 bytes)

View attachment "job-script" of type "text/plain" (7585 bytes)

View attachment "job.yaml" of type "text/plain" (5164 bytes)

View attachment "reproduce" of type "text/plain" (340 bytes)

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ