| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4A43DB99.70602@gmail.com>
Date: Thu, 25 Jun 2009 22:18:33 +0200
From: Eric Dumazet <eric.dumazet@...il.com>
To: Paweł Staszewski <pstaszewski@...are.pl>
CC: Linux Network Development list <netdev@...r.kernel.org>
Subject: Re: weird problem
Paweł Staszewski a écrit :
> Ok
>
> After this day of observation im near 100% sure that this cpu load is
> made by route cahce flushes
> When route cache increase to its "net.ipv4.route.gc_thresh" size or is
> near that size
> system is starting to drop some routes from cache then cpu load is
> increase from 2% to near 80%
> after cleaning / flush cache when cache is filling cpu load is again
> normal 2%
>
> Someone know how to resolve this ?
> on kernels < 2.6.29 i don't see this, all start after upgrade from
> 2.6.28 to 2.6.29 - then i try 2.6.29.1 , 2.6.29.3 and 2.6.30 and on all
> this kernels >= 2.6.29 problem with cpu load is the same.
>
> I can minimize this cpu fluctuations by changing of route cache /proc
> parameters but the best result for my router was
>
> 15 sec of 2% cpu
> and after
> 15sec of 80% cpu
>
>
> Regards
> Pawel Staszewski
I believe this is known 2.6.29 regressions
Following two commits should correct the problem you have
Your best bet would be to try 2.6.31-rc1, and tell us if this recent kernel
is ok on your machine ?
commit 1ddbcb005c395518c2cd0df504cff3d4b5c85853
Author: Eric Dumazet <dada1@...mosbay.com>
Date: Tue May 19 20:14:28 2009 +0000
net: fix rtable leak in net/ipv4/route.c
Alexander V. Lukyanov found a regression in 2.6.29 and made a complete
analysis found in http://bugzilla.kernel.org/show_bug.cgi?id=13339
Quoted here because its a perfect one :
begin_of_quotation
2.6.29 patch has introduced flexible route cache rebuilding. Unfortunately the
patch has at least one critical flaw, and another problem.
rt_intern_hash calculates rthi pointer, which is later used for new entry
insertion. The same loop calculates cand pointer which is used to clean the
list. If the pointers are the same, rtable leak occurs, as first the cand is
removed then the new entry is appended to it.
This leak leads to unregister_netdevice problem (usage count > 0).
Another problem of the patch is that it tries to insert the entries in certain
order, to facilitate counting of entries distinct by all but QoS parameters.
Unfortunately, referencing an existing rtable entry moves it to list beginning,
to speed up further lookups, so the carefully built order is destroyed.
For the first problem the simplest patch it to set rthi=0 when rthi==cand, but
it will also destroy the ordering.
end_of_quotation
Problematic commit is 1080d709fb9d8cd4392f93476ee46a9d6ea05a5b
(net: implement emergency route cache rebulds when gc_elasticity is exceeded)
Trying to keep dst_entries ordered is too complex and breaks the fact that
order should depend on the frequency of use for garbage collection.
A possible fix is to make rt_intern_hash() simpler, and only makes
rt_check_expire() a litle bit smarter, being able to cope with an arbitrary
entries order. The added loop is running on cache hot data, while cpu
is prefetching next object, so should be unnoticied.
Reported-and-analyzed-by: Alexander V. Lukyanov <lav@....ru>
commit cf8da764fc6959b7efb482f375dfef9830e98205
Author: Eric Dumazet <dada1@...mosbay.com>
Date: Tue May 19 18:54:22 2009 +0000
net: fix length computation in rt_check_expire()
rt_check_expire() computes average and standard deviation of chain lengths,
but not correclty reset length to 0 at beginning of each chain.
This probably gives overflows for sum2 (and sum) on loaded machines instead
of meaningful results.
Signed-off-by: Eric Dumazet <dada1@...mosbay.com>
Acked-by: Neil Horman <nhorman@...driver.com>
Signed-off-by: David S. Miller <davem@...emloft.net>
>
>
>
> Paweł Staszewski pisze:
>> Another info
>>
>> i look at some stats from rtstats and mpstat and try to join them
>> together and here is output:
>> 19:06:50 CPU %usr %nice %sys %iowait %irq %soft
>> %steal %guest %idle rt_cache entries
>> 19:07:01 all 0.00 0.00 0.12 0.00 0.59 12.00
>> 0.00 0.00 87.29 77726
>> 19:07:02 all 0.00 0.00 0.12 0.00 0.49 6.11
>> 0.00 0.00 93.28 80127
>> 19:07:03 all 0.00 0.00 0.00 0.00 0.11 1.06
>> 0.00 0.00 98.84 82596
>> 19:07:04 all 0.00 0.00 0.16 0.00 0.00 2.07
>> 0.00 0.00 97.77 85147
>> 19:07:05 all 0.00 0.00 0.00 0.00 0.12 1.81
>> 0.00 0.00 98.07 87641
>> 19:07:06 all 0.00 0.00 0.00 0.00 0.27 1.50
>> 0.00 0.00 98.22 90085
>> 19:07:07 all 0.00 0.00 0.00 0.00 0.23 1.72
>> 0.00 0.00 98.05 93586
>> 19:07:08 all 0.00 0.00 0.00 0.00 0.00 1.56
>> 0.00 0.00 98.44 96865
>> 19:07:09 all 0.00 0.00 0.00 0.00 0.00 1.45
>> 0.00 0.00 98.55 99803
>> 19:07:10 all 0.00 0.00 0.00 0.00 0.27 1.65
>> 0.00 0.00 98.08 102123
>> 19:07:11 all 0.00 0.00 0.10 0.00 0.30 4.07
>> 0.00 0.00 95.54 104618
>> 19:07:12 all 0.00 0.00 0.00 0.00 1.30 18.96
>> 0.00 0.00 79.74 105524
>> 19:07:13 all 0.00 0.00 0.00 0.00 0.84 13.58
>> 0.00 0.00 85.58 101085
>> 19:07:14 all 0.00 0.00 0.00 0.00 1.36 20.45
>> 0.00 0.00 78.18 97518
>> 19:07:15 all 0.00 0.00 0.00 0.00 0.98 15.89
>> 0.00 0.00 83.13 94308
>> 19:07:16 all 0.00 0.00 0.00 0.00 1.32 17.00
>> 0.00 0.00 81.69 91811
>> 19:07:17 all 0.00 0.00 0.00 0.00 0.67 14.57
>> 0.00 0.00 84.75 92278
>> 19:07:18 all 0.00 0.00 0.00 0.00 1.51 17.40
>> 0.00 0.00 81.10 90936
>> 19:07:19 all 0.00 0.00 0.00 0.00 1.24 13.46
>> 0.00 0.00 85.29 87951
>> 19:07:20 all 0.00 0.00 0.14 0.00 0.99 18.75
>> 0.00 0.00 80.11 85071
>> 19:07:21 all 0.00 0.00 0.00 0.00 0.84 15.20
>> 0.00 0.00 83.96 82477
>> 19:07:22 all 0.40 0.00 0.00 0.00 1.71 17.65
>> 0.00 0.00 80.24 80301
>> 19:07:23 all 0.00 0.00 0.11 0.00 1.08 13.33
>> 0.00 0.00 85.48 78480
>> 19:07:24 all 0.00 0.00 0.00 0.00 0.87 19.25
>> 0.00 0.00 79.88 76880
>>
>> When cpu load is rising rt cache entries are decreasing
>> and when cpu load is from 1 do 3% rt cache entries are increasing
>>
>> some sysctl-s that i have and are route cache specific:
>> net.ipv4.route.gc_thresh = 190536
>> net.ipv4.route.gc_min_interval = 0
>> net.ipv4.route.gc_min_interval_ms = 0
>> net.ipv4.route.gc_timeout = 15
>> net.ipv4.route.gc_interval = 1
>> net.ipv4.route.gc_elasticity = 4
>>
>>
>>
>>
>> Paweł Staszewski pisze:
>>> oprofile output per cpu
>>> CPU: Core 2, speed 3000.08 MHz (estimated)
>>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a
>>> unit mask of 0x00 (Unhalted core cycles) count 100000
>>> Samples on CPU 0
>>> Samples on CPU 1
>>> Samples on CPU 2
>>> Samples on CPU 3
>>> Samples on CPU 4
>>> Samples on CPU 5
>>> Samples on CPU 6
>>> Samples on CPU 7
>>> Samples on CPU all
>>> samples % samples % samples % samples
>>> % samples % samples % samples %
>>> samples % samples %
>>> image name app name symbol name
>>> 29520 21.7362 0 0 0 0 0
>>> 0 0 0 0 0 9571 34.0254
>>> 0 0 4701 0.22
>>> 55 oprofiled oprofiled (no symbols)
>>> 18441 13.5785 0 0 0 0 0
>>> 0 0 0 0 0 0 0
>>> 0 0 2666 0.12
>>> 79 vmlinux vmlinux rt_worker_func
>>> 18428 13.5690 132089 2.1756 135138 1.4238 21735
>>> 42.7703 775 1.9819 893 5.3180 847 3.0111
>>> 1056 3.3130 37949 1.82
>>> 01 vmlinux vmlinux mwait_idle
>>> 15797 11.6317 212 0.0035 2879 0.0303 17
>>> 0.0335 2 0.0051 4 0.0238 14 0.0498
>>> 185 0.5804 2895 0.13
>>> 89 vmlinux vmlinux free_block
>>> 9034 6.6519 1218 0.0201 2727 0.0287 0
>>> 0 0 0 0 0 0 0
>>> 0 0 2004 0.09
>>> 61 vmlinux vmlinux dst_destroy
>>> 7113 5.2375 632 0.0104 483 0.0051 6615
>>> 13.0170 543 1.3886 461 2.7454 1951 6.9359
>>> 1274 3.9970 4073 0.19
>>> 54 vmlinux vmlinux
>>> mutex_spin_on_owner
>>> 4382 3.2266 652 0.0107 1880 0.0198 2
>>> 0.0039 0 0 0 0 3 0.0107
>>> 7 0.0220 1050 0.05
>>> 04 vmlinux vmlinux __call_rcu
>>> 4337 3.1934 418 0.0069 830 0.0087 58
>>> 0.1141 18 0.0460 8 0.0476 20 0.0711
>>> 39 0.1224 917 0.04
>>> 40 vmlinux vmlinux
>>> __rcu_process_callbacks
>>> 3991 2.9387 51 8.4e-04 325 0.0034 136
>>> 0.2676 2 0.0051 1 0.0060 1034 3.6759
>>> 205 0.6432 1110 0.05
>>> 32 vmlinux vmlinux
>>> copy_user_generic_string
>>> 3292 2.4240 275838 4.5432 391639 4.1263 869
>>> 1.7100 431 1.1022 350 2.0843 673 2.3925
>>> 1069 3.3538 89939 4.31
>>> 37 vmlinux vmlinux _raw_spin_lock
>>> 1909 1.4056 79 0.0013 376 0.0040 0
>>> 0 0 0 0 0 0 0
>>> 0 0 348 0.01
>>> 67 vmlinux vmlinux dst_rcu_free
>>> 1461 1.0758 0 0 1163 0.0123 0
>>> 0 0 0 0 0 0 0
>>> 0 0 0
>>> 0 vim vim (no symbols)
>>> 1122 0.8262 11514 0.1896 6590 0.0694 1857
>>> 3.6542 7638 19.5325 2138 12.7323 1213 4.3123
>>> 4025 12.6278 9676 0.46
>>> 41 libc-2.8.so libc-2.8.so (no symbols)
>>> 1035 0.7621 9027 0.1487 22216 0.2341 4
>>> 0.0079 0 0 0 0 5 0.0178
>>> 15 0.0471 4310 0.20
>>> 67 vmlinux vmlinux kmem_cache_free
>>> 1016 0.7481 218 0.0036 465 0.0049 0
>>> 0 0 0 0 0 0 0
>>> 0 0 255 0.01
>>> 22 vmlinux vmlinux ipv4_dst_destroy
>>> 951 0.7002 4290 0.0707 5514 0.0581 3283
>>> 6.4603 867 2.2172 673 4.0079 1096 3.8963
>>> 1316 4.1288 2593 0.12
>>> 44 vmlinux vmlinux tg_shares_up
>>> 708 0.5213 104 0.0017 279 0.0029 0
>>> 0 0 0 0 0 0 0
>>> 0 0 159 0.00
>>> 76 vmlinux vmlinux call_rcu_bh
>>> 581 0.4278 21812 0.3593 38364 0.4042 68
>>> 0.1338 23 0.0588 23 0.1370 65 0.2311
>>> 96 0.3012 8000 0.38
>>> 37 vmlinux vmlinux _raw_spin_unlock
>>> 563 0.4145 1027 0.0169 2528 0.0266 0
>>> 0 0 0 0 0 0 0 3
>>> 0.0094 545 0.02
>>> 61 vmlinux vmlinux local_bh_enable_ip
>>> 509 0.3748 0 0 3 3.2e-05 0
>>> 0 0 0 0 0 4 0.0142 3
>>> 0.0094 76 0.00
>>> 36 vmlinux vmlinux can_dirty
>>> 430 0.3166 74 0.0012 148 0.0016 25
>>> 0.0492 13 0.0332 12 0.0715 9 0.0320
>>> 12 0.0376 96 0.00
>>> 46 vmlinux vmlinux
>>> rcu_process_callbacks
>>> 372 0.2739 0 0 0 0 0
>>> 0 0 0 0 0 3 0.0107 2
>>> 0.0063 48 0.00
>>> 23 vmlinux vmlinux do_journal_end
>>> 347 0.2555 9991 0.1646 16066 0.1693 143
>>> 0.2814 66 0.1688 25 0.1489 46 0.1635
>>> 71 0.2228 3514 0.16
>>> 85 vmlinux vmlinux rb_get_reader_page
>>> 346 0.2548 952 0.0157 566 0.0060 871
>>> 1.7140 321 0.8209 174 1.0362 324 1.1518
>>> 269 0.8439 655 0.03
>>> 14 vmlinux vmlinux find_busiest_group
>>> 275 0.2025 1194 0.0197 1228 0.0129 1019
>>> 2.0052 203 0.5191 151 0.8992 290 1.0310
>>> 653 2.0487 799 0.03
>>> 83 vmlinux vmlinux find_next_bit
>>> 238 0.1752 3053 0.0503 4711 0.0496 116
>>> 0.2283 82 0.2097 54 0.3216 101 0.3591
>>> 101 0.3169 1174 0.05
>>> 63 oprofile oprofile (no symbols)
>>> 221 0.1627 0 0 0 0 0
>>> 0 0 0 0 0 0 0
>>> 0 0 18 8.6e-
>>> 04 vmlinux vmlinux ahci_interrupt
>>> 212 0.1561 38335 0.6314 60370 0.6361 329
>>> 0.6474 165 0.4220 136 0.8099 133 0.4728
>>> 205 0.6432 12821 0.61
>>> 49 vmlinux vmlinux
>>> apic_timer_interrupt
>>> 210 0.1546 117 0.0019 1001 0.0105 2
>>> 0.0039 0 0 0 0 55 0.1955
>>> 16 0.0502 270 0.01
>>> 30 vmlinux vmlinux memmove
>>> 207 0.1524 0 0 7 7.4e-05 34
>>> 0.0669 0 0 0 0 93 0.3306
>>> 62 0.1945 214 0.01
>>> 03 vmlinux vmlinux search_by_key
>>> 204 0.1502 26603 0.4382 25086 0.2643 45
>>> 0.0886 18 0.0460 19 0.1131 19 0.0675
>>> 25 0.0784 6580 0.31
>>> 56 vmlinux vmlinux __do_softirq
>>> 170 0.1252 2618 0.0431 3493 0.0368 69
>>> 0.1358 42 0.1074 51 0.3037 64 0.2275
>>> 87 0.2729 856 0.04
>>> 11 vmlinux vmlinux __schedule
>>> 151 0.1112 7590 0.1250 8909 0.0939 273
>>> 0.5372 86 0.2199 73 0.4347 99 0.3519
>>> 142 0.4455 2114 0.10
>>> 14 vmlinux vmlinux sched_clock_cpu
>>> 142 0.1046 200 0.0033 165 0.0017 65
>>> 0.1279 23 0.0588 20 0.1191 15 0.0533
>>> 36 0.1129 94 0.00
>>> 45 vmlinux vmlinux raise_softirq
>>> 141 0.1038 8048 0.1326 11386 0.1200 2
>>> 0.0039 0 0 0 0 2 0.0071
>>> 4 0.0125 2589 0.12
>>> 42 vmlinux vmlinux __phys_addr
>>> 141 0.1038 23726 0.3908 24837 0.2617 89
>>> 0.1751 58 0.1483 55 0.3275 66 0.2346
>>> 62 0.1945 5398 0.25
>>> 89 vmlinux vmlinux
>>> tick_nohz_stop_sched_tick
>>> 139 0.1023 43230 0.7120 68500 0.7217 105
>>> 0.2066 66 0.1688 86 0.5121 100 0.3555
>>> 100 0.3137 13725 0.65
>>> 83 vmlinux vmlinux getnstimeofday
>>> 136 0.1001 0 0 0 0 0
>>> 0 0 0 0 0 0 0
>>> 0 0 36 0.00
>>> 17 vmlinux vmlinux
>>> remove_journal_hash
>>> 133 0.0979 8624 0.1420 9518 0.1003 217
>>> 0.4270 68 0.1739 64 0.3811 79 0.2808
>>> 106 0.3326 2348 0.11
>>> 26 vmlinux vmlinux native_sched_clock
>>> 129 0.0950 270 0.0044 233 0.0025 355
>>> 0.6986 93 0.2378 60 0.3573 102 0.3626
>>> 105 0.3294 214 0.01
>>> 03 vmlinux vmlinux rebalance_domains
>>> 117 0.0861 0 0 36 3.8e-04 57
>>> 0.1122 0 0 10 0.0596 66 0.2346
>>> 60 0.1882 220 0.01
>>> 06 vmlinux vmlinux find_get_page
>>> 112 0.0825 287 0.0047 397 0.0042 121
>>> 0.2381 89 0.2276 63 0.3752 85 0.3022
>>> 114 0.3577 220 0.01
>>> 06 vmlinux vmlinux scheduler_tick
>>> 107 0.0788 13702 0.2257 16191 0.1706 102
>>> 0.2007 28 0.0716 44 0.2620 41 0.1458
>>> 66 0.2071 3890 0.18
>>> 66 vmlinux vmlinux
>>> tick_nohz_stop_idle
>>> 105 0.0773 4277 0.0704 7002 0.0738 53
>>> 0.1043 26 0.0665 24 0.1429 24 0.0853
>>> 38 0.1192 1561 0.07
>>> 49 vmlinux vmlinux
>>> ring_buffer_consume
>>> 103 0.0758 0 0 2 2.1e-05 5
>>> 0.0098 0 0 0 0 20 0.0711
>>> 13 0.0408 61 0.00
>>> 29 vmlinux vmlinux __find_get_block
>>> 103 0.0758 49251 0.8112 85999 0.9061 166
>>> 0.3267 105 0.2685 97 0.5777 122 0.4337
>>> 138 0.4330 17108 0.82
>>> 06 vmlinux vmlinux read_tsc
>>> 98 0.0722 17406 0.2867 40897 0.4309 15
>>> 0.0295 2 0.0051 2 0.0119 62 0.2204
>>> 51 0.1600 7913 0.37
>>> 95 vmlinux vmlinux kmem_cache_alloc
>>> 98 0.0722 14817 0.2440 21291 0.2243 41
>>> 0.0807 49 0.1253 48 0.2859 62 0.2204
>>> 64 0.2008 4582 0.21
>>> 98 vmlinux vmlinux ktime_get_ts
>>> 94 0.0692 0 0 1 1.1e-05 2
>>> 0.0039 0 0 0 0 7 0.0249
>>> 6 0.0188 16 7.7e-
>>> 04 vmlinux vmlinux
>>> __find_get_block_slow
>>> 92 0.0677 1161 0.0191 1411 0.0149 41
>>> 0.0807 39 0.0997 40 0.2382 53 0.1884
>>> 49 0.1537 408 0.01
>>> 96 vmlinux vmlinux
>>> select_nohz_load_balancer
>>> 88 0.0648 11160 0.1838 54990 0.5794 69
>>> 0.1358 43 0.1100 35 0.2084 661 2.3499
>>> 55 0.1726 4513 0.21
>>> 65 vmlinux vmlinux
>>> get_next_timer_interrupt
>>> 75 0.0552 0 0 0 0 0
>>> 0 0 0 0 0 0 0
>>> 0 0 6 2.9e-
>>> 04 vmlinux vmlinux make_request
>>> 69 0.0508 803 0.0132 1048 0.0110 35
>>> 0.0689 26 0.0665 15 0.0893 24 0.0853
>>> 58 0.1820 334 0.01
>>> --More--
>>>
>>> Paweł Staszewski pisze:
>>>> I dont know where to put this message so i start here because server
>>>> that i see this funny thing is make only networking
>>>> Routing (bgpd) / traffic management (iproute2)
>>>>
>>>> This starts on kernels >= 2.6.29
>>>>
>>>> On server i have almost constant traffic that is about 300Mbit/s RX
>>>> and 300Mbit/s TX
>>>> Server have two interfaces:
>>>> 04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit
>>>> Ethernet Controller (Copper) (rev 03)
>>>> 05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit
>>>> Ethernet Controller
>>>>
>>>> iBGP peer < - > [ eth0 - forward + shaping - eth1 ] < - > eBGP
>>>>
>>>>
>>>> And what is weird is CPU Load
>>>> it fluctuate from 10% to almost 80% - but traffic is constant :
>>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> | iface Rx
>>>> Tx Total
>>>>
>>>> ==============================================================================
>>>>
>>>> lo: 0.00 b/s 0.00
>>>> b/s 0.00 b/s
>>>> eth0: 200.22 Mb/s 266.74 Mb/s
>>>> 466.96 Mb/s
>>>> eth1: 264.00 Mb/s 192.59 Mb/s
>>>> 456.59 Mb/s
>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>> total: 464.22 Mb/s 459.34 Mb/s
>>>> 923.56 Mb/s
>>>>
>>>>
>>>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>>> input: /proc/net/dev type: rate
>>>> / iface Rx
>>>> Tx Total
>>>>
>>>> ==============================================================================
>>>>
>>>> lo: 0.00 P/s 0.00
>>>> P/s 0.00 P/s
>>>> eth0: 51314.95 P/s 49880.20 P/s
>>>> 101195.14 P/s
>>>> eth1: 43309.85 P/s 41885.79 P/s
>>>> 85195.64 P/s
>>>>
>>>> ------------------------------------------------------------------------------
>>>>
>>>> total: 94624.80 P/s 91765.99 P/s
>>>> 186390.79 P/s
>>>>
>>>>
>>>> mpstat -P ALL 1
>>>> 18:00:05 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:06 all 0.00 0.00 0.00 0.00 1.33 13.65
>>>> 0.00 0.00 85.02
>>>> 18:00:06 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:06 1 0.00 0.00 0.00 0.00 3.77 39.62
>>>> 0.00 0.00 56.60
>>>> 18:00:06 2 0.00 0.00 0.00 0.00 7.92 70.30
>>>> 0.00 0.00 21.78
>>>> 18:00:06 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:06 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:06 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:06 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:06 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:06 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:07 all 0.12 0.00 0.12 0.00 1.19 13.35
>>>> 0.00 0.00 85.22
>>>> 18:00:07 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:07 1 0.00 0.00 0.00 0.00 2.75 42.20
>>>> 0.00 0.00 55.05
>>>> 18:00:07 2 0.00 0.00 0.99 0.00 5.94 65.35
>>>> 0.00 0.00 27.72
>>>> 18:00:07 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:07 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:07 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:07 6 0.00 0.00 0.99 0.00 0.00 0.00
>>>> 0.00 0.00 99.01
>>>> 18:00:07 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:07 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:08 all 0.00 0.00 0.12 0.00 0.85 13.42
>>>> 0.00 0.00 85.61
>>>> 18:00:08 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:08 1 0.00 0.00 0.00 0.00 1.85 38.89
>>>> 0.00 0.00 59.26
>>>> 18:00:08 2 0.00 0.00 0.00 0.00 5.94 68.32
>>>> 0.00 0.00 25.74
>>>> 18:00:08 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:08 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:08 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:08 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:08 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:08 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:09 all 0.00 0.00 0.00 0.00 1.33 14.18
>>>> 0.00 0.00 84.48
>>>> 18:00:09 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:09 1 0.00 0.00 0.99 0.00 2.97 50.50
>>>> 0.00 0.00 45.54
>>>> 18:00:09 2 0.00 0.00 0.00 0.00 6.93 65.35
>>>> 0.00 0.00 27.72
>>>> 18:00:09 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:09 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:09 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:09 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:09 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:09 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:10 all 0.00 0.00 0.24 0.00 1.20 11.81
>>>> 0.00 0.00 86.75
>>>> 18:00:10 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:10 1 0.00 0.00 0.95 0.00 4.76 35.24
>>>> 0.00 0.00 59.05
>>>> 18:00:10 2 0.00 0.00 0.00 0.00 4.95 59.41
>>>> 0.00 0.00 35.64
>>>> 18:00:10 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:10 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:10 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:10 6 0.00 0.00 0.95 0.00 0.00 0.00
>>>> 0.00 0.00 99.05
>>>> 18:00:10 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:10 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:11 all 0.00 0.00 0.24 0.00 0.36 5.97
>>>> 0.00 0.00 93.43
>>>> 18:00:11 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:11 1 0.00 0.00 0.97 0.00 0.97 17.48
>>>> 0.00 0.00 80.58
>>>> 18:00:11 2 0.00 0.00 0.00 0.00 2.88 31.73
>>>> 0.00 0.00 65.38
>>>> 18:00:11 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:11 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:11 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:11 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:11 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:11 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:12 all 0.00 0.00 0.25 0.00 0.37 5.76
>>>> 0.00 0.00 93.63
>>>> 18:00:12 0 0.00 0.00 0.00 1.06 0.00 0.00
>>>> 0.00 0.00 98.94
>>>> 18:00:12 1 0.00 0.00 1.92 0.00 0.00 18.27
>>>> 0.00 0.00 79.81
>>>> 18:00:12 2 0.00 0.00 0.96 0.00 1.92 26.92
>>>> 0.00 0.00 70.19
>>>> 18:00:12 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:12 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:12 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:12 6 0.96 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 99.04
>>>> 18:00:12 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:12 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:13 all 0.12 0.00 0.25 0.00 0.12 2.34
>>>> 0.00 0.00 97.17
>>>> 18:00:13 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:13 1 0.00 0.00 0.00 0.00 0.97 7.77
>>>> 0.00 0.00 91.26
>>>> 18:00:13 2 0.00 0.00 0.98 0.00 0.98 10.78
>>>> 0.00 0.00 87.25
>>>> 18:00:13 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:13 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:13 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:13 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:13 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:13 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:14 all 0.00 0.00 0.37 0.00 0.37 2.33
>>>> 0.00 0.00 96.94
>>>> 18:00:14 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:14 1 0.00 0.00 1.92 0.00 0.96 7.69
>>>> 0.00 0.00 89.42
>>>> 18:00:14 2 0.00 0.00 0.97 0.00 0.97 10.68
>>>> 0.00 0.00 87.38
>>>> 18:00:14 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:14 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:14 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:14 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:14 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:14 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:15 all 0.00 0.00 0.61 0.00 0.25 2.45
>>>> 0.00 0.00 96.69
>>>> 18:00:15 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:15 1 0.00 0.00 3.85 0.00 0.96 7.69
>>>> 0.00 0.00 87.50
>>>> 18:00:15 2 0.00 0.00 0.96 0.00 1.92 12.50
>>>> 0.00 0.00 84.62
>>>> 18:00:15 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:15 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:15 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:15 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:15 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:15 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:16 all 0.00 0.00 0.37 0.00 0.37 1.83
>>>> 0.00 0.00 97.43
>>>> 18:00:16 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:16 1 0.00 0.00 1.96 0.00 0.98 4.90
>>>> 0.00 0.00 92.16
>>>> 18:00:16 2 0.00 0.00 1.00 0.00 1.00 10.00
>>>> 0.00 0.00 88.00
>>>> 18:00:16 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:16 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:16 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:16 6 0.98 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 99.02
>>>> 18:00:16 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:16 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:17 all 0.00 0.00 0.25 0.00 0.25 2.73
>>>> 0.00 0.00 96.78
>>>> 18:00:17 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:17 1 0.00 0.00 1.96 0.00 0.00 8.82
>>>> 0.00 0.00 89.22
>>>> 18:00:17 2 0.00 0.00 0.99 0.00 1.98 11.88
>>>> 0.00 0.00 85.15
>>>> 18:00:17 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:17 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:17 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:17 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:17 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:17 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:18 all 0.12 0.00 0.12 0.00 0.95 11.51
>>>> 0.00 0.00 87.31
>>>> 18:00:18 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:18 1 0.00 0.00 0.00 0.00 3.03 33.33
>>>> 0.00 0.00 63.64
>>>> 18:00:18 2 0.00 0.00 0.00 0.00 5.00 64.00
>>>> 0.00 0.00 31.00
>>>> 18:00:18 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:18 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:18 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:18 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:18 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:18 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:19 all 0.36 0.00 0.12 0.00 1.46 14.48
>>>> 0.00 0.00 83.58
>>>> 18:00:19 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:19 1 0.00 0.00 0.00 0.00 4.00 46.00
>>>> 0.00 0.00 50.00
>>>> 18:00:19 2 0.00 0.00 0.99 0.00 7.92 72.28
>>>> 0.00 0.00 18.81
>>>> 18:00:19 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:19 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:19 5 2.86 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 97.14
>>>> 18:00:19 6 0.95 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 99.05
>>>> 18:00:19 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:19 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:20 all 0.00 0.00 0.00 0.00 1.58 14.86
>>>> 0.00 0.00 83.56
>>>> 18:00:20 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:20 1 0.00 0.00 0.00 0.00 3.96 49.50
>>>> 0.00 0.00 46.53
>>>> 18:00:20 2 0.00 0.00 0.00 0.00 9.00 73.00
>>>> 0.00 0.00 18.00
>>>> 18:00:20 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:20 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:20 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:20 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:20 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:20 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:21 all 0.00 0.00 0.00 0.00 1.71 14.86
>>>> 0.00 0.00 83.43
>>>> 18:00:21 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:21 1 0.00 0.00 0.00 0.00 5.05 51.52
>>>> 0.00 0.00 43.43
>>>> 18:00:21 2 0.00 0.00 0.00 0.00 10.00 70.00
>>>> 0.00 0.00 20.00
>>>> 18:00:21 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:21 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:21 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:21 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:21 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:21 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:22 all 0.00 0.00 0.24 0.00 1.70 13.97
>>>> 0.00 0.00 84.08
>>>> 18:00:22 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:22 1 0.00 0.00 0.00 0.00 4.95 42.57
>>>> 0.00 0.00 52.48
>>>> 18:00:22 2 0.00 0.00 1.01 0.00 8.08 72.73
>>>> 0.00 0.00 18.18
>>>> 18:00:22 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:22 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:22 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:22 6 0.00 0.00 0.95 0.00 0.00 0.00
>>>> 0.00 0.00 99.05
>>>> 18:00:22 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:22 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:23 all 0.12 0.00 0.00 0.00 1.58 14.13
>>>> 0.00 0.00 84.17
>>>> 18:00:23 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:23 1 0.00 0.00 0.00 0.00 5.00 45.00
>>>> 0.00 0.00 50.00
>>>> 18:00:23 2 0.99 0.00 0.00 0.00 8.91 70.30
>>>> 0.00 0.00 19.80
>>>> 18:00:23 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:23 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:23 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:23 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:23 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:23 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:24 all 0.00 0.00 0.00 0.00 1.34 14.13
>>>> 0.00 0.00 84.53
>>>> 18:00:24 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:24 1 0.00 0.00 0.00 0.00 4.04 42.42
>>>> 0.00 0.00 53.54
>>>> 18:00:24 2 0.00 0.00 0.00 0.00 6.93 73.27
>>>> 0.00 0.00 19.80
>>>> 18:00:24 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:24 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:24 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:24 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:24 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:24 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:25 all 0.12 0.00 0.12 0.00 1.45 14.67
>>>> 0.00 0.00 83.64
>>>> 18:00:25 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:25 1 0.00 0.00 0.99 0.00 3.96 45.54
>>>> 0.00 0.00 49.50
>>>> 18:00:25 2 0.00 0.00 0.00 0.00 8.00 75.00
>>>> 0.00 0.00 17.00
>>>> 18:00:25 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:25 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:25 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:25 6 0.95 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 99.05
>>>> 18:00:25 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:25 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:26 all 0.00 0.00 0.00 0.00 1.22 14.60
>>>> 0.00 0.00 84.18
>>>> 18:00:26 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:26 1 0.99 0.00 0.00 0.00 2.97 50.50
>>>> 0.00 0.00 45.54
>>>> 18:00:26 2 0.00 0.00 0.00 0.00 7.07 68.69
>>>> 0.00 0.00 24.24
>>>> 18:00:26 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:26 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:26 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:26 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:26 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:26 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:27 all 0.00 0.00 0.00 0.00 1.59 15.02
>>>> 0.00 0.00 83.39
>>>> 18:00:27 0 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:27 1 0.00 0.00 0.00 0.00 6.00 52.00
>>>> 0.00 0.00 42.00
>>>> 18:00:27 2 0.00 0.00 0.00 0.00 6.93 71.29
>>>> 0.00 0.00 21.78
>>>> 18:00:27 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:27 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:27 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:27 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:27 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> 18:00:27 CPU %usr %nice %sys %iowait %irq %soft
>>>> %steal %guest %idle
>>>> 18:00:28 all 0.00 0.00 0.00 0.00 1.33 14.56
>>>> 0.00 0.00 84.10
>>>> 18:00:28 0 0.00 0.00 0.00 0.00 0.00 0.99
>>>> 0.00 0.00 99.01
>>>> 18:00:28 1 0.00 0.00 0.00 0.00 4.00 50.00
>>>> 0.00 0.00 46.00
>>>> 18:00:28 2 0.00 0.00 0.00 0.00 6.06 69.70
>>>> 0.00 0.00 24.24
>>>> 18:00:28 3 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:28 4 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:28 5 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:28 6 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>> 18:00:28 7 0.00 0.00 0.00 0.00 0.00 0.00
>>>> 0.00 0.00 100.00
>>>>
>>>> at the same time top shows:
>>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>>> 28 root 15 -5 0 0 0 S 2 0.0 0:16.60 events/1
>>>> 29 root 15 -5 0 0 0 S 2 0.0 0:15.62 events/2
>>>> 10 root 15 -5 0 0 0 S 2 0.0 0:41.01 ksoftirqd/2
>>>> 7 root 15 -5 0 0 0 S 1 0.0 0:31.17 ksoftirqd/1
>>>> 30 root 15 -5 0 0 0 S 1 0.0 0:03.00 events/3
>>>> 6545 root 20 0 11112 3244 1780 S 0 0.0 0:08.69 oprofiled
>>>> 6845 root 20 0 19084 1384 1008 R 0 0.0 0:00.07 top
>>>> 1 root 20 0 3812 612 520 S 0 0.0 0:01.00 init
>>>> 2 root 15 -5 0 0 0 S 0 0.0 0:00.00 kthreadd
>>>> 3 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/0
>>>> 4 root 15 -5 0 0 0 S 0 0.0 0:01.04 ksoftirqd/0
>>>> 5 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/0
>>>> 6 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/1
>>>> 8 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/1
>>>> 9 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/2
>>>> 11 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/2
>>>> 12 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/3
>>>> 13 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/3
>>>> 14 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/3
>>>> 15 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/4
>>>> 16 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/4
>>>> 17 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/4
>>>> 18 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/5
>>>> 19 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/5
>>>> 20 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/5
>>>> 21 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/6
>>>> 22 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/6
>>>> 23 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/6
>>>> 24 root RT -5 0 0 0 S 0 0.0 0:00.00 migration/7
>>>> 25 root 15 -5 0 0 0 S 0 0.0 0:00.00 ksoftirqd/7
>>>> 26 root RT -5 0 0 0 S 0 0.0 0:00.00 watchdog/7
>>>> 27 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/0
>>>> 31 root 15 -5 0 0 0 S 0 0.0 0:00.00 events/4
>>>> 32 root 15 -5 0 0 0 S 0 0.0 0:00.01 events/5
>>>> 33 root 15 -5 0 0 0 S 0 0.0 0:00.02 events/6
>>>> 34 root 15 -5 0 0 0 S 0 0.0 0:00.01 events/7
>>>> 35 root 15 -5 0 0 0 S 0 0.0 0:00.01 khelper
>>>> 41 root 15 -5 0 0 0 S 0 0.0 0:00.00 async/mgr
>>>> 159 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/0
>>>> 160 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/1
>>>> 161 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/2
>>>> 162 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/3
>>>> 163 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/4
>>>> 164 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/5
>>>> 165 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/6
>>>> 166 root 15 -5 0 0 0 S 0 0.0 0:00.00 kblockd/7
>>>> 167 root 15 -5 0 0 0 S 0 0.0 0:00.00 kacpid
>>>> 168 root 15 -5 0 0 0 S 0 0.0 0:00.00
>>>> kacpi_notify
>>>> 246 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/0
>>>> 247 root 15 -5 0 0 0 S 0 0.0 0:00.00 ata/1
>>>>
>>>> and oprofile output:
>>>> CPU: Core 2, speed 3000.08 MHz (estimated)
>>>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with
>>>> a unit mask of 0x00 (Unhalted core cycles) count 100000
>>>> samples % image name app name
>>>> symbol name
>>>> 11138304 12.3598 vmlinux vmlinux
>>>> rb_next
>>>> 7714260 8.5602 vmlinux vmlinux
>>>> e1000_clean
>>>> 5380466 5.9705 vmlinux vmlinux
>>>> e1000_intr_msi
>>>> 4881445 5.4168 vmlinux vmlinux
>>>> u32_classify
>>>> 3977072 4.4132 vmlinux vmlinux
>>>> _raw_spin_lock
>>>> 3471611 3.8523 vmlinux vmlinux
>>>> hfsc_dequeue
>>>> 3103493 3.4438 vmlinux vmlinux
>>>> hfsc_enqueue
>>>> 2636824 2.9260 vmlinux vmlinux
>>>> delay_tsc
>>>> 2464293 2.7345 vmlinux vmlinux
>>>> e1000_xmit_frame
>>>> 2035779 2.2590 vmlinux vmlinux
>>>> e1000_clean_tx_irq
>>>> 1947231 2.1608 vmlinux vmlinux
>>>> mwait_idle
>>>> 1919540 2.1300 vmlinux vmlinux
>>>> ipt_do_table
>>>> 1908544 2.1178 vmlinux vmlinux
>>>> e1000_clean_rx_irq
>>>> 1865996 2.0706 vmlinux vmlinux
>>>> update_vf
>>>> 1829293 2.0299 vmlinux vmlinux
>>>> rb_erase
>>>> 1777847 1.9728 vmlinux vmlinux
>>>> ip_route_input
>>>> 1430773 1.5877 vmlinux vmlinux
>>>> rb_insert_color
>>>> 1329674 1.4755 vmlinux vmlinux
>>>> __alloc_skb
>>>> 1091678 1.2114 vmlinux vmlinux
>>>> sfq_enqueue
>>>> 985105 1.0931 vmlinux vmlinux
>>>> rb_first
>>>> 944928 1.0486 vmlinux vmlinux
>>>> sfq_dequeue
>>>> 756452 0.8394 vmlinux vmlinux
>>>> read_tsc
>>>> 728703 0.8086 vmlinux vmlinux
>>>> nf_iterate
>>>> 665551 0.7385 vmlinux vmlinux
>>>> ip_forward
>>>> 662616 0.7353 vmlinux vmlinux
>>>> dev_queue_xmit
>>>> 648919 0.7201 vmlinux vmlinux
>>>> __qdisc_run
>>>> 615592 0.6831 vmlinux vmlinux
>>>> ip_rcv
>>>> 589743 0.6544 vmlinux vmlinux
>>>> apic_timer_interrupt
>>>> 550581 0.6110 vmlinux vmlinux
>>>> irq_entries_start
>>>> 536172 0.5950 vmlinux vmlinux
>>>> getnstimeofday
>>>> 527467 0.5853 vmlinux vmlinux
>>>> rb_last
>>>> 508305 0.5640 vmlinux vmlinux
>>>> skb_release_data
>>>> 506763 0.5623 vmlinux vmlinux
>>>> kfree
>>>> 506269 0.5618 vmlinux vmlinux
>>>> dst_release
>>>> 464412 0.5153 vmlinux vmlinux
>>>> __hrtimer_start_range_ns
>>>> 456354 0.5064 vmlinux vmlinux
>>>> e1000_alloc_rx_buffers
>>>> 435339 0.4831 vmlinux vmlinux
>>>> dev_hard_start_xmit
>>>> 371439 0.4122 vmlinux vmlinux
>>>> local_bh_enable
>>>> 369266 0.4098 vmlinux vmlinux
>>>> __rb_rotate_left
>>>> 361627 0.4013 vmlinux vmlinux
>>>> net_tx_action
>>>> 357797 0.3970 vmlinux vmlinux
>>>> rtsc_y2x
>>>> 350595 0.3890 vmlinux vmlinux
>>>> _raw_spin_unlock
>>>> 349359 0.3877 vmlinux vmlinux
>>>> kmem_cache_alloc
>>>> 336255 0.3731 vmlinux vmlinux
>>>> skb_dma_map
>>>> 328809 0.3649 vmlinux vmlinux
>>>> netif_receive_skb
>>>> 315826 0.3505 vmlinux vmlinux
>>>> fn_trie_lookup
>>>> 313065 0.3474 vmlinux vmlinux
>>>> __kmalloc
>>>> 307684 0.3414 vmlinux vmlinux
>>>> __do_softirq
>>>> 298676 0.3314 vmlinux vmlinux
>>>> skb_pull
>>>> 292352 0.3244 vmlinux vmlinux
>>>> __napi_complete
>>>> 288878 0.3206 vmlinux vmlinux
>>>> get_next_timer_interrupt
>>>> 280449 0.3112 vmlinux vmlinux
>>>> ip_finish_output
>>>> 264482 0.2935 vmlinux vmlinux
>>>> consume_skb
>>>> 251622 0.2792 vmlinux vmlinux
>>>> __remove_hrtimer
>>>> 227178 0.2521 vmlinux vmlinux
>>>> tc_classify_compat
>>>> 226224 0.2510 vmlinux vmlinux
>>>> handle_edge_irq
>>>>
>>>>
>>>>
>>>> proces that this machine is running is only quagga (zebra + bgpd +
>>>> oprofiled)
>>>>
>>>>
>>>> Regards
>>>> Paweł Staszewski
>>>>
>>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>>> the body of a message to majordomo@...r.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>>> the body of a message to majordomo@...r.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@...r.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
>
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists