lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <4A43B17E.6060102@itcare.pl>
Date:	Thu, 25 Jun 2009 19:18:54 +0200
From:	Paweł Staszewski <pstaszewski@...are.pl>
To:	Linux Network Development list <netdev@...r.kernel.org>
Subject: Re: weird problem

Another info

i look at some stats from rtstats and mpstat and try to join them 
together and here is output:
19:06:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
%steal  %guest   %idle        rt_cache entries
19:07:01     all    0.00    0.00    0.12    0.00    0.59   12.00    
0.00    0.00   87.29        77726
19:07:02     all    0.00    0.00    0.12    0.00    0.49    6.11    
0.00    0.00   93.28        80127
19:07:03     all    0.00    0.00    0.00    0.00    0.11    1.06    
0.00    0.00   98.84        82596
19:07:04     all    0.00    0.00    0.16    0.00    0.00    2.07    
0.00    0.00   97.77        85147
19:07:05     all    0.00    0.00    0.00    0.00    0.12    1.81    
0.00    0.00   98.07        87641
19:07:06     all    0.00    0.00    0.00    0.00    0.27    1.50    
0.00    0.00   98.22        90085
19:07:07     all    0.00    0.00    0.00    0.00    0.23    1.72    
0.00    0.00   98.05        93586
19:07:08     all    0.00    0.00    0.00    0.00    0.00    1.56    
0.00    0.00   98.44        96865
19:07:09     all    0.00    0.00    0.00    0.00    0.00    1.45    
0.00    0.00   98.55        99803
19:07:10     all    0.00    0.00    0.00    0.00    0.27    1.65    
0.00    0.00   98.08        102123
19:07:11     all    0.00    0.00    0.10    0.00    0.30    4.07    
0.00    0.00   95.54        104618
19:07:12     all    0.00    0.00    0.00    0.00    1.30   18.96    
0.00    0.00   79.74        105524
19:07:13     all    0.00    0.00    0.00    0.00    0.84   13.58    
0.00    0.00   85.58        101085
19:07:14     all    0.00    0.00    0.00    0.00    1.36   20.45    
0.00    0.00   78.18        97518
19:07:15     all    0.00    0.00    0.00    0.00    0.98   15.89    
0.00    0.00   83.13        94308
19:07:16     all    0.00    0.00    0.00    0.00    1.32   17.00    
0.00    0.00   81.69        91811
19:07:17     all    0.00    0.00    0.00    0.00    0.67   14.57    
0.00    0.00   84.75        92278
19:07:18     all    0.00    0.00    0.00    0.00    1.51   17.40    
0.00    0.00   81.10        90936
19:07:19     all    0.00    0.00    0.00    0.00    1.24   13.46    
0.00    0.00   85.29        87951
19:07:20     all    0.00    0.00    0.14    0.00    0.99   18.75    
0.00    0.00   80.11        85071
19:07:21     all    0.00    0.00    0.00    0.00    0.84   15.20    
0.00    0.00   83.96        82477
19:07:22     all    0.40    0.00    0.00    0.00    1.71   17.65    
0.00    0.00   80.24        80301
19:07:23     all    0.00    0.00    0.11    0.00    1.08   13.33    
0.00    0.00   85.48        78480
19:07:24     all    0.00    0.00    0.00    0.00    0.87   19.25    
0.00    0.00   79.88        76880

When cpu load is rising rt cache entries are decreasing
and when cpu load is from 1 do 3% rt cache entries are increasing

some sysctl-s that i have and are route cache specific:
net.ipv4.route.gc_thresh = 190536
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.gc_min_interval_ms = 0
net.ipv4.route.gc_timeout = 15
net.ipv4.route.gc_interval = 1
net.ipv4.route.gc_elasticity = 4




Paweł Staszewski pisze:
> oprofile output per cpu
> CPU: Core 2, speed 3000.08 MHz (estimated)
> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a 
> unit mask of 0x00 (Unhalted core cycles) count 100000
> Samples on CPU 0
> Samples on CPU 1
> Samples on CPU 2
> Samples on CPU 3
> Samples on CPU 4
> Samples on CPU 5
> Samples on CPU 6
> Samples on CPU 7
> Samples on CPU all
> samples  %        samples  %        samples  %        samples  
> %        samples  %        samples  %        samples  %        
> samples  %        samples  %
>    image name               app name                 symbol name
> 29520    21.7362  0              0  0              0  0              
> 0  0              0  0              0  9571     34.0254  
> 0              0  4701      0.22
> 55  oprofiled                oprofiled                (no symbols)
> 18441    13.5785  0              0  0              0  0              
> 0  0              0  0              0  0              0  
> 0              0  2666      0.12
> 79  vmlinux                  vmlinux                  rt_worker_func
> 18428    13.5690  132089    2.1756  135138    1.4238  21735    
> 42.7703  775       1.9819  893       5.3180  847       3.0111  
> 1056      3.3130  37949     1.82
> 01  vmlinux                  vmlinux                  mwait_idle
> 15797    11.6317  212       0.0035  2879      0.0303  17        
> 0.0335  2         0.0051  4         0.0238  14        0.0498  
> 185       0.5804  2895      0.13
> 89  vmlinux                  vmlinux                  free_block
> 9034      6.6519  1218      0.0201  2727      0.0287  0              
> 0  0              0  0              0  0              0  
> 0              0  2004      0.09
> 61  vmlinux                  vmlinux                  dst_destroy
> 7113      5.2375  632       0.0104  483       0.0051  6615     
> 13.0170  543       1.3886  461       2.7454  1951      6.9359  
> 1274      3.9970  4073      0.19
> 54  vmlinux                  vmlinux                  mutex_spin_on_owner
> 4382      3.2266  652       0.0107  1880      0.0198  2         
> 0.0039  0              0  0              0  3         0.0107  
> 7         0.0220  1050      0.05
> 04  vmlinux                  vmlinux                  __call_rcu
> 4337      3.1934  418       0.0069  830       0.0087  58        
> 0.1141  18        0.0460  8         0.0476  20        0.0711  
> 39        0.1224  917       0.04
> 40  vmlinux                  vmlinux                  
> __rcu_process_callbacks
> 3991      2.9387  51       8.4e-04  325       0.0034  136       
> 0.2676  2         0.0051  1         0.0060  1034      3.6759  
> 205       0.6432  1110      0.05
> 32  vmlinux                  vmlinux                  
> copy_user_generic_string
> 3292      2.4240  275838    4.5432  391639    4.1263  869       
> 1.7100  431       1.1022  350       2.0843  673       2.3925  
> 1069      3.3538  89939     4.31
> 37  vmlinux                  vmlinux                  _raw_spin_lock
> 1909      1.4056  79        0.0013  376       0.0040  0              
> 0  0              0  0              0  0              0  
> 0              0  348       0.01
> 67  vmlinux                  vmlinux                  dst_rcu_free
> 1461      1.0758  0              0  1163      0.0123  0              
> 0  0              0  0              0  0              0  
> 0              0  0
> 0  vim                      vim                      (no symbols)
> 1122      0.8262  11514     0.1896  6590      0.0694  1857      
> 3.6542  7638     19.5325  2138     12.7323  1213      4.3123  4025     
> 12.6278  9676      0.46
> 41  libc-2.8.so              libc-2.8.so              (no symbols)
> 1035      0.7621  9027      0.1487  22216     0.2341  4         
> 0.0079  0              0  0              0  5         0.0178  
> 15        0.0471  4310      0.20
> 67  vmlinux                  vmlinux                  kmem_cache_free
> 1016      0.7481  218       0.0036  465       0.0049  0              
> 0  0              0  0              0  0              0  
> 0              0  255       0.01
> 22  vmlinux                  vmlinux                  ipv4_dst_destroy
> 951       0.7002  4290      0.0707  5514      0.0581  3283      
> 6.4603  867       2.2172  673       4.0079  1096      3.8963  
> 1316      4.1288  2593      0.12
> 44  vmlinux                  vmlinux                  tg_shares_up
> 708       0.5213  104       0.0017  279       0.0029  0              
> 0  0              0  0              0  0              0  
> 0              0  159       0.00
> 76  vmlinux                  vmlinux                  call_rcu_bh
> 581       0.4278  21812     0.3593  38364     0.4042  68        
> 0.1338  23        0.0588  23        0.1370  65        0.2311  
> 96        0.3012  8000      0.38
> 37  vmlinux                  vmlinux                  _raw_spin_unlock
> 563       0.4145  1027      0.0169  2528      0.0266  0              
> 0  0              0  0              0  0              0  3         
> 0.0094  545       0.02
> 61  vmlinux                  vmlinux                  local_bh_enable_ip
> 509       0.3748  0              0  3        3.2e-05  0              
> 0  0              0  0              0  4         0.0142  3         
> 0.0094  76        0.00
> 36  vmlinux                  vmlinux                  can_dirty
> 430       0.3166  74        0.0012  148       0.0016  25        
> 0.0492  13        0.0332  12        0.0715  9         0.0320  
> 12        0.0376  96        0.00
> 46  vmlinux                  vmlinux                  
> rcu_process_callbacks
> 372       0.2739  0              0  0              0  0              
> 0  0              0  0              0  3         0.0107  2         
> 0.0063  48        0.00
> 23  vmlinux                  vmlinux                  do_journal_end
> 347       0.2555  9991      0.1646  16066     0.1693  143       
> 0.2814  66        0.1688  25        0.1489  46        0.1635  
> 71        0.2228  3514      0.16
> 85  vmlinux                  vmlinux                  rb_get_reader_page
> 346       0.2548  952       0.0157  566       0.0060  871       
> 1.7140  321       0.8209  174       1.0362  324       1.1518  
> 269       0.8439  655       0.03
> 14  vmlinux                  vmlinux                  find_busiest_group
> 275       0.2025  1194      0.0197  1228      0.0129  1019      
> 2.0052  203       0.5191  151       0.8992  290       1.0310  
> 653       2.0487  799       0.03
> 83  vmlinux                  vmlinux                  find_next_bit
> 238       0.1752  3053      0.0503  4711      0.0496  116       
> 0.2283  82        0.2097  54        0.3216  101       0.3591  
> 101       0.3169  1174      0.05
> 63  oprofile                 oprofile                 (no symbols)
> 221       0.1627  0              0  0              0  0              
> 0  0              0  0              0  0              0  
> 0              0  18       8.6e-
> 04  vmlinux                  vmlinux                  ahci_interrupt
> 212       0.1561  38335     0.6314  60370     0.6361  329       
> 0.6474  165       0.4220  136       0.8099  133       0.4728  
> 205       0.6432  12821     0.61
> 49  vmlinux                  vmlinux                  
> apic_timer_interrupt
> 210       0.1546  117       0.0019  1001      0.0105  2         
> 0.0039  0              0  0              0  55        0.1955  
> 16        0.0502  270       0.01
> 30  vmlinux                  vmlinux                  memmove
> 207       0.1524  0              0  7        7.4e-05  34        
> 0.0669  0              0  0              0  93        0.3306  
> 62        0.1945  214       0.01
> 03  vmlinux                  vmlinux                  search_by_key
> 204       0.1502  26603     0.4382  25086     0.2643  45        
> 0.0886  18        0.0460  19        0.1131  19        0.0675  
> 25        0.0784  6580      0.31
> 56  vmlinux                  vmlinux                  __do_softirq
> 170       0.1252  2618      0.0431  3493      0.0368  69        
> 0.1358  42        0.1074  51        0.3037  64        0.2275  
> 87        0.2729  856       0.04
> 11  vmlinux                  vmlinux                  __schedule
> 151       0.1112  7590      0.1250  8909      0.0939  273       
> 0.5372  86        0.2199  73        0.4347  99        0.3519  
> 142       0.4455  2114      0.10
> 14  vmlinux                  vmlinux                  sched_clock_cpu
> 142       0.1046  200       0.0033  165       0.0017  65        
> 0.1279  23        0.0588  20        0.1191  15        0.0533  
> 36        0.1129  94        0.00
> 45  vmlinux                  vmlinux                  raise_softirq
> 141       0.1038  8048      0.1326  11386     0.1200  2         
> 0.0039  0              0  0              0  2         0.0071  
> 4         0.0125  2589      0.12
> 42  vmlinux                  vmlinux                  __phys_addr
> 141       0.1038  23726     0.3908  24837     0.2617  89        
> 0.1751  58        0.1483  55        0.3275  66        0.2346  
> 62        0.1945  5398      0.25
> 89  vmlinux                  vmlinux                  
> tick_nohz_stop_sched_tick
> 139       0.1023  43230     0.7120  68500     0.7217  105       
> 0.2066  66        0.1688  86        0.5121  100       0.3555  
> 100       0.3137  13725     0.65
> 83  vmlinux                  vmlinux                  getnstimeofday
> 136       0.1001  0              0  0              0  0              
> 0  0              0  0              0  0              0  
> 0              0  36        0.00
> 17  vmlinux                  vmlinux                  remove_journal_hash
> 133       0.0979  8624      0.1420  9518      0.1003  217       
> 0.4270  68        0.1739  64        0.3811  79        0.2808  
> 106       0.3326  2348      0.11
> 26  vmlinux                  vmlinux                  native_sched_clock
> 129       0.0950  270       0.0044  233       0.0025  355       
> 0.6986  93        0.2378  60        0.3573  102       0.3626  
> 105       0.3294  214       0.01
> 03  vmlinux                  vmlinux                  rebalance_domains
> 117       0.0861  0              0  36       3.8e-04  57        
> 0.1122  0              0  10        0.0596  66        0.2346  
> 60        0.1882  220       0.01
> 06  vmlinux                  vmlinux                  find_get_page
> 112       0.0825  287       0.0047  397       0.0042  121       
> 0.2381  89        0.2276  63        0.3752  85        0.3022  
> 114       0.3577  220       0.01
> 06  vmlinux                  vmlinux                  scheduler_tick
> 107       0.0788  13702     0.2257  16191     0.1706  102       
> 0.2007  28        0.0716  44        0.2620  41        0.1458  
> 66        0.2071  3890      0.18
> 66  vmlinux                  vmlinux                  tick_nohz_stop_idle
> 105       0.0773  4277      0.0704  7002      0.0738  53        
> 0.1043  26        0.0665  24        0.1429  24        0.0853  
> 38        0.1192  1561      0.07
> 49  vmlinux                  vmlinux                  ring_buffer_consume
> 103       0.0758  0              0  2        2.1e-05  5         
> 0.0098  0              0  0              0  20        0.0711  
> 13        0.0408  61        0.00
> 29  vmlinux                  vmlinux                  __find_get_block
> 103       0.0758  49251     0.8112  85999     0.9061  166       
> 0.3267  105       0.2685  97        0.5777  122       0.4337  
> 138       0.4330  17108     0.82
> 06  vmlinux                  vmlinux                  read_tsc
> 98        0.0722  17406     0.2867  40897     0.4309  15        
> 0.0295  2         0.0051  2         0.0119  62        0.2204  
> 51        0.1600  7913      0.37
> 95  vmlinux                  vmlinux                  kmem_cache_alloc
> 98        0.0722  14817     0.2440  21291     0.2243  41        
> 0.0807  49        0.1253  48        0.2859  62        0.2204  
> 64        0.2008  4582      0.21
> 98  vmlinux                  vmlinux                  ktime_get_ts
> 94        0.0692  0              0  1        1.1e-05  2         
> 0.0039  0              0  0              0  7         0.0249  
> 6         0.0188  16       7.7e-
> 04  vmlinux                  vmlinux                  
> __find_get_block_slow
> 92        0.0677  1161      0.0191  1411      0.0149  41        
> 0.0807  39        0.0997  40        0.2382  53        0.1884  
> 49        0.1537  408       0.01
> 96  vmlinux                  vmlinux                  
> select_nohz_load_balancer
> 88        0.0648  11160     0.1838  54990     0.5794  69        
> 0.1358  43        0.1100  35        0.2084  661       2.3499  
> 55        0.1726  4513      0.21
> 65  vmlinux                  vmlinux                  
> get_next_timer_interrupt
> 75        0.0552  0              0  0              0  0              
> 0  0              0  0              0  0              0  
> 0              0  6        2.9e-
> 04  vmlinux                  vmlinux                  make_request
> 69        0.0508  803       0.0132  1048      0.0110  35        
> 0.0689  26        0.0665  15        0.0893  24        0.0853  
> 58        0.1820  334       0.01
> --More--
>
> Paweł Staszewski pisze:
>> I dont know where to put this message so i start here because server 
>> that i see this funny thing is make only networking
>> Routing (bgpd) / traffic management (iproute2)
>>
>> This starts on kernels >= 2.6.29
>>
>> On server i have almost constant traffic that is about 300Mbit/s RX 
>> and 300Mbit/s TX
>> Server have two interfaces:
>> 04:00.0 Ethernet controller: Intel Corporation 82573E Gigabit 
>> Ethernet Controller (Copper) (rev 03)
>> 05:00.0 Ethernet controller: Intel Corporation 82573L Gigabit 
>> Ethernet Controller
>>
>> iBGP peer < - > [ eth0 - forward + shaping - eth1 ] < - >  eBGP
>>
>>
>> And what is weird is CPU Load
>> it fluctuate from 10% to almost 80% - but traffic is  constant :
>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>  input: /proc/net/dev type: rate
>>  |         iface                   Rx                   
>> Tx                Total
>>  
>> ============================================================================== 
>>
>>               lo:           0.00  b/s            0.00  b/s            
>> 0.00  b/s
>>             eth0:         200.22 Mb/s          266.74 Mb/s          
>> 466.96 Mb/s
>>             eth1:         264.00 Mb/s          192.59 Mb/s          
>> 456.59 Mb/s
>>  
>> ------------------------------------------------------------------------------ 
>>
>>            total:         464.22 Mb/s          459.34 Mb/s          
>> 923.56 Mb/s
>>
>>
>> bwm-ng v0.6 (probing every 10.000s), press 'h' for help
>>  input: /proc/net/dev type: rate
>>  /         iface                   Rx                   
>> Tx                Total
>>  
>> ============================================================================== 
>>
>>               lo:            0.00 P/s             0.00 
>> P/s             0.00 P/s
>>             eth0:        51314.95 P/s         49880.20 P/s        
>> 101195.14 P/s
>>             eth1:        43309.85 P/s         41885.79 P/s         
>> 85195.64 P/s
>>  
>> ------------------------------------------------------------------------------ 
>>
>>            total:        94624.80 P/s         91765.99 P/s        
>> 186390.79 P/s
>>
>>
>> mpstat -P ALL 1
>> 18:00:05     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:06     all    0.00    0.00    0.00    0.00    1.33   13.65    
>> 0.00    0.00   85.02
>> 18:00:06       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:06       1    0.00    0.00    0.00    0.00    3.77   39.62    
>> 0.00    0.00   56.60
>> 18:00:06       2    0.00    0.00    0.00    0.00    7.92   70.30    
>> 0.00    0.00   21.78
>> 18:00:06       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:06       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:06       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:06       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:06       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:06     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:07     all    0.12    0.00    0.12    0.00    1.19   13.35    
>> 0.00    0.00   85.22
>> 18:00:07       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:07       1    0.00    0.00    0.00    0.00    2.75   42.20    
>> 0.00    0.00   55.05
>> 18:00:07       2    0.00    0.00    0.99    0.00    5.94   65.35    
>> 0.00    0.00   27.72
>> 18:00:07       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:07       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:07       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:07       6    0.00    0.00    0.99    0.00    0.00    0.00    
>> 0.00    0.00   99.01
>> 18:00:07       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:07     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:08     all    0.00    0.00    0.12    0.00    0.85   13.42    
>> 0.00    0.00   85.61
>> 18:00:08       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:08       1    0.00    0.00    0.00    0.00    1.85   38.89    
>> 0.00    0.00   59.26
>> 18:00:08       2    0.00    0.00    0.00    0.00    5.94   68.32    
>> 0.00    0.00   25.74
>> 18:00:08       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:08       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:08       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:08       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:08       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:08     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:09     all    0.00    0.00    0.00    0.00    1.33   14.18    
>> 0.00    0.00   84.48
>> 18:00:09       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:09       1    0.00    0.00    0.99    0.00    2.97   50.50    
>> 0.00    0.00   45.54
>> 18:00:09       2    0.00    0.00    0.00    0.00    6.93   65.35    
>> 0.00    0.00   27.72
>> 18:00:09       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:09       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:09       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:09       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:09       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:09     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:10     all    0.00    0.00    0.24    0.00    1.20   11.81    
>> 0.00    0.00   86.75
>> 18:00:10       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:10       1    0.00    0.00    0.95    0.00    4.76   35.24    
>> 0.00    0.00   59.05
>> 18:00:10       2    0.00    0.00    0.00    0.00    4.95   59.41    
>> 0.00    0.00   35.64
>> 18:00:10       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:10       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:10       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:10       6    0.00    0.00    0.95    0.00    0.00    0.00    
>> 0.00    0.00   99.05
>> 18:00:10       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:10     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:11     all    0.00    0.00    0.24    0.00    0.36    5.97    
>> 0.00    0.00   93.43
>> 18:00:11       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:11       1    0.00    0.00    0.97    0.00    0.97   17.48    
>> 0.00    0.00   80.58
>> 18:00:11       2    0.00    0.00    0.00    0.00    2.88   31.73    
>> 0.00    0.00   65.38
>> 18:00:11       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:11       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:11       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:11       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:11       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:11     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:12     all    0.00    0.00    0.25    0.00    0.37    5.76    
>> 0.00    0.00   93.63
>> 18:00:12       0    0.00    0.00    0.00    1.06    0.00    0.00    
>> 0.00    0.00   98.94
>> 18:00:12       1    0.00    0.00    1.92    0.00    0.00   18.27    
>> 0.00    0.00   79.81
>> 18:00:12       2    0.00    0.00    0.96    0.00    1.92   26.92    
>> 0.00    0.00   70.19
>> 18:00:12       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:12       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:12       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:12       6    0.96    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00   99.04
>> 18:00:12       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:12     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:13     all    0.12    0.00    0.25    0.00    0.12    2.34    
>> 0.00    0.00   97.17
>> 18:00:13       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:13       1    0.00    0.00    0.00    0.00    0.97    7.77    
>> 0.00    0.00   91.26
>> 18:00:13       2    0.00    0.00    0.98    0.00    0.98   10.78    
>> 0.00    0.00   87.25
>> 18:00:13       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:13       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:13       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:13       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:13       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:13     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:14     all    0.00    0.00    0.37    0.00    0.37    2.33    
>> 0.00    0.00   96.94
>> 18:00:14       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:14       1    0.00    0.00    1.92    0.00    0.96    7.69    
>> 0.00    0.00   89.42
>> 18:00:14       2    0.00    0.00    0.97    0.00    0.97   10.68    
>> 0.00    0.00   87.38
>> 18:00:14       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:14       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:14       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:14       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:14       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:14     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:15     all    0.00    0.00    0.61    0.00    0.25    2.45    
>> 0.00    0.00   96.69
>> 18:00:15       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:15       1    0.00    0.00    3.85    0.00    0.96    7.69    
>> 0.00    0.00   87.50
>> 18:00:15       2    0.00    0.00    0.96    0.00    1.92   12.50    
>> 0.00    0.00   84.62
>> 18:00:15       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:15       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:15       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:15       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:15       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:15     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:16     all    0.00    0.00    0.37    0.00    0.37    1.83    
>> 0.00    0.00   97.43
>> 18:00:16       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:16       1    0.00    0.00    1.96    0.00    0.98    4.90    
>> 0.00    0.00   92.16
>> 18:00:16       2    0.00    0.00    1.00    0.00    1.00   10.00    
>> 0.00    0.00   88.00
>> 18:00:16       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:16       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:16       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:16       6    0.98    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00   99.02
>> 18:00:16       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:16     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:17     all    0.00    0.00    0.25    0.00    0.25    2.73    
>> 0.00    0.00   96.78
>> 18:00:17       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:17       1    0.00    0.00    1.96    0.00    0.00    8.82    
>> 0.00    0.00   89.22
>> 18:00:17       2    0.00    0.00    0.99    0.00    1.98   11.88    
>> 0.00    0.00   85.15
>> 18:00:17       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:17       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:17       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:17       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:17       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:17     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:18     all    0.12    0.00    0.12    0.00    0.95   11.51    
>> 0.00    0.00   87.31
>> 18:00:18       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:18       1    0.00    0.00    0.00    0.00    3.03   33.33    
>> 0.00    0.00   63.64
>> 18:00:18       2    0.00    0.00    0.00    0.00    5.00   64.00    
>> 0.00    0.00   31.00
>> 18:00:18       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:18       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:18       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:18       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:18       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:18     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:19     all    0.36    0.00    0.12    0.00    1.46   14.48    
>> 0.00    0.00   83.58
>> 18:00:19       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:19       1    0.00    0.00    0.00    0.00    4.00   46.00    
>> 0.00    0.00   50.00
>> 18:00:19       2    0.00    0.00    0.99    0.00    7.92   72.28    
>> 0.00    0.00   18.81
>> 18:00:19       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:19       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:19       5    2.86    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00   97.14
>> 18:00:19       6    0.95    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00   99.05
>> 18:00:19       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:19     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:20     all    0.00    0.00    0.00    0.00    1.58   14.86    
>> 0.00    0.00   83.56
>> 18:00:20       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:20       1    0.00    0.00    0.00    0.00    3.96   49.50    
>> 0.00    0.00   46.53
>> 18:00:20       2    0.00    0.00    0.00    0.00    9.00   73.00    
>> 0.00    0.00   18.00
>> 18:00:20       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:20       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:20       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:20       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:20       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:20     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:21     all    0.00    0.00    0.00    0.00    1.71   14.86    
>> 0.00    0.00   83.43
>> 18:00:21       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:21       1    0.00    0.00    0.00    0.00    5.05   51.52    
>> 0.00    0.00   43.43
>> 18:00:21       2    0.00    0.00    0.00    0.00   10.00   70.00    
>> 0.00    0.00   20.00
>> 18:00:21       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:21       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:21       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:21       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:21       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:21     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:22     all    0.00    0.00    0.24    0.00    1.70   13.97    
>> 0.00    0.00   84.08
>> 18:00:22       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:22       1    0.00    0.00    0.00    0.00    4.95   42.57    
>> 0.00    0.00   52.48
>> 18:00:22       2    0.00    0.00    1.01    0.00    8.08   72.73    
>> 0.00    0.00   18.18
>> 18:00:22       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:22       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:22       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:22       6    0.00    0.00    0.95    0.00    0.00    0.00    
>> 0.00    0.00   99.05
>> 18:00:22       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:22     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:23     all    0.12    0.00    0.00    0.00    1.58   14.13    
>> 0.00    0.00   84.17
>> 18:00:23       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:23       1    0.00    0.00    0.00    0.00    5.00   45.00    
>> 0.00    0.00   50.00
>> 18:00:23       2    0.99    0.00    0.00    0.00    8.91   70.30    
>> 0.00    0.00   19.80
>> 18:00:23       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:23       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:23       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:23       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:23       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:23     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:24     all    0.00    0.00    0.00    0.00    1.34   14.13    
>> 0.00    0.00   84.53
>> 18:00:24       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:24       1    0.00    0.00    0.00    0.00    4.04   42.42    
>> 0.00    0.00   53.54
>> 18:00:24       2    0.00    0.00    0.00    0.00    6.93   73.27    
>> 0.00    0.00   19.80
>> 18:00:24       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:24       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:24       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:24       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:24       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:24     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:25     all    0.12    0.00    0.12    0.00    1.45   14.67    
>> 0.00    0.00   83.64
>> 18:00:25       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:25       1    0.00    0.00    0.99    0.00    3.96   45.54    
>> 0.00    0.00   49.50
>> 18:00:25       2    0.00    0.00    0.00    0.00    8.00   75.00    
>> 0.00    0.00   17.00
>> 18:00:25       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:25       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:25       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:25       6    0.95    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00   99.05
>> 18:00:25       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:25     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:26     all    0.00    0.00    0.00    0.00    1.22   14.60    
>> 0.00    0.00   84.18
>> 18:00:26       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:26       1    0.99    0.00    0.00    0.00    2.97   50.50    
>> 0.00    0.00   45.54
>> 18:00:26       2    0.00    0.00    0.00    0.00    7.07   68.69    
>> 0.00    0.00   24.24
>> 18:00:26       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:26       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:26       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:26       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:26       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:26     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:27     all    0.00    0.00    0.00    0.00    1.59   15.02    
>> 0.00    0.00   83.39
>> 18:00:27       0    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:27       1    0.00    0.00    0.00    0.00    6.00   52.00    
>> 0.00    0.00   42.00
>> 18:00:27       2    0.00    0.00    0.00    0.00    6.93   71.29    
>> 0.00    0.00   21.78
>> 18:00:27       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:27       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:27       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:27       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:27       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> 18:00:27     CPU    %usr   %nice    %sys %iowait    %irq   %soft  
>> %steal  %guest   %idle
>> 18:00:28     all    0.00    0.00    0.00    0.00    1.33   14.56    
>> 0.00    0.00   84.10
>> 18:00:28       0    0.00    0.00    0.00    0.00    0.00    0.99    
>> 0.00    0.00   99.01
>> 18:00:28       1    0.00    0.00    0.00    0.00    4.00   50.00    
>> 0.00    0.00   46.00
>> 18:00:28       2    0.00    0.00    0.00    0.00    6.06   69.70    
>> 0.00    0.00   24.24
>> 18:00:28       3    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:28       4    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:28       5    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:28       6    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>> 18:00:28       7    0.00    0.00    0.00    0.00    0.00    0.00    
>> 0.00    0.00  100.00
>>
>> at the same time top shows:
>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>   28 root      15  -5     0    0    0 S    2  0.0   0:16.60 events/1
>>   29 root      15  -5     0    0    0 S    2  0.0   0:15.62 events/2
>>   10 root      15  -5     0    0    0 S    2  0.0   0:41.01 ksoftirqd/2
>>    7 root      15  -5     0    0    0 S    1  0.0   0:31.17 ksoftirqd/1
>>   30 root      15  -5     0    0    0 S    1  0.0   0:03.00 events/3
>> 6545 root      20   0 11112 3244 1780 S    0  0.0   0:08.69 oprofiled
>> 6845 root      20   0 19084 1384 1008 R    0  0.0   0:00.07 top
>>    1 root      20   0  3812  612  520 S    0  0.0   0:01.00 init
>>    2 root      15  -5     0    0    0 S    0  0.0   0:00.00 kthreadd
>>    3 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/0
>>    4 root      15  -5     0    0    0 S    0  0.0   0:01.04 ksoftirqd/0
>>    5 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/0
>>    6 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/1
>>    8 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/1
>>    9 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/2
>>   11 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/2
>>   12 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/3
>>   13 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/3
>>   14 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/3
>>   15 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/4
>>   16 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/4
>>   17 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/4
>>   18 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/5
>>   19 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/5
>>   20 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/5
>>   21 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/6
>>   22 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/6
>>   23 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/6
>>   24 root      RT  -5     0    0    0 S    0  0.0   0:00.00 migration/7
>>   25 root      15  -5     0    0    0 S    0  0.0   0:00.00 ksoftirqd/7
>>   26 root      RT  -5     0    0    0 S    0  0.0   0:00.00 watchdog/7
>>   27 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/0
>>   31 root      15  -5     0    0    0 S    0  0.0   0:00.00 events/4
>>   32 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/5
>>   33 root      15  -5     0    0    0 S    0  0.0   0:00.02 events/6
>>   34 root      15  -5     0    0    0 S    0  0.0   0:00.01 events/7
>>   35 root      15  -5     0    0    0 S    0  0.0   0:00.01 khelper
>>   41 root      15  -5     0    0    0 S    0  0.0   0:00.00 async/mgr
>>  159 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/0
>>  160 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/1
>>  161 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/2
>>  162 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/3
>>  163 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/4
>>  164 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/5
>>  165 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/6
>>  166 root      15  -5     0    0    0 S    0  0.0   0:00.00 kblockd/7
>>  167 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpid
>>  168 root      15  -5     0    0    0 S    0  0.0   0:00.00 kacpi_notify
>>  246 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/0
>>  247 root      15  -5     0    0    0 S    0  0.0   0:00.00 ata/1
>>
>> and oprofile output:
>> CPU: Core 2, speed 3000.08 MHz (estimated)
>> Counted CPU_CLK_UNHALTED events (Clock cycles when not halted) with a 
>> unit mask of 0x00 (Unhalted core cycles) count 100000
>> samples  %        image name               app name                 
>> symbol name
>> 11138304 12.3598  vmlinux                  vmlinux                  
>> rb_next
>> 7714260   8.5602  vmlinux                  vmlinux                  
>> e1000_clean
>> 5380466   5.9705  vmlinux                  vmlinux                  
>> e1000_intr_msi
>> 4881445   5.4168  vmlinux                  vmlinux                  
>> u32_classify
>> 3977072   4.4132  vmlinux                  vmlinux                  
>> _raw_spin_lock
>> 3471611   3.8523  vmlinux                  vmlinux                  
>> hfsc_dequeue
>> 3103493   3.4438  vmlinux                  vmlinux                  
>> hfsc_enqueue
>> 2636824   2.9260  vmlinux                  vmlinux                  
>> delay_tsc
>> 2464293   2.7345  vmlinux                  vmlinux                  
>> e1000_xmit_frame
>> 2035779   2.2590  vmlinux                  vmlinux                  
>> e1000_clean_tx_irq
>> 1947231   2.1608  vmlinux                  vmlinux                  
>> mwait_idle
>> 1919540   2.1300  vmlinux                  vmlinux                  
>> ipt_do_table
>> 1908544   2.1178  vmlinux                  vmlinux                  
>> e1000_clean_rx_irq
>> 1865996   2.0706  vmlinux                  vmlinux                  
>> update_vf
>> 1829293   2.0299  vmlinux                  vmlinux                  
>> rb_erase
>> 1777847   1.9728  vmlinux                  vmlinux                  
>> ip_route_input
>> 1430773   1.5877  vmlinux                  vmlinux                  
>> rb_insert_color
>> 1329674   1.4755  vmlinux                  vmlinux                  
>> __alloc_skb
>> 1091678   1.2114  vmlinux                  vmlinux                  
>> sfq_enqueue
>> 985105    1.0931  vmlinux                  vmlinux                  
>> rb_first
>> 944928    1.0486  vmlinux                  vmlinux                  
>> sfq_dequeue
>> 756452    0.8394  vmlinux                  vmlinux                  
>> read_tsc
>> 728703    0.8086  vmlinux                  vmlinux                  
>> nf_iterate
>> 665551    0.7385  vmlinux                  vmlinux                  
>> ip_forward
>> 662616    0.7353  vmlinux                  vmlinux                  
>> dev_queue_xmit
>> 648919    0.7201  vmlinux                  vmlinux                  
>> __qdisc_run
>> 615592    0.6831  vmlinux                  vmlinux                  
>> ip_rcv
>> 589743    0.6544  vmlinux                  vmlinux                  
>> apic_timer_interrupt
>> 550581    0.6110  vmlinux                  vmlinux                  
>> irq_entries_start
>> 536172    0.5950  vmlinux                  vmlinux                  
>> getnstimeofday
>> 527467    0.5853  vmlinux                  vmlinux                  
>> rb_last
>> 508305    0.5640  vmlinux                  vmlinux                  
>> skb_release_data
>> 506763    0.5623  vmlinux                  vmlinux                  
>> kfree
>> 506269    0.5618  vmlinux                  vmlinux                  
>> dst_release
>> 464412    0.5153  vmlinux                  vmlinux                  
>> __hrtimer_start_range_ns
>> 456354    0.5064  vmlinux                  vmlinux                  
>> e1000_alloc_rx_buffers
>> 435339    0.4831  vmlinux                  vmlinux                  
>> dev_hard_start_xmit
>> 371439    0.4122  vmlinux                  vmlinux                  
>> local_bh_enable
>> 369266    0.4098  vmlinux                  vmlinux                  
>> __rb_rotate_left
>> 361627    0.4013  vmlinux                  vmlinux                  
>> net_tx_action
>> 357797    0.3970  vmlinux                  vmlinux                  
>> rtsc_y2x
>> 350595    0.3890  vmlinux                  vmlinux                  
>> _raw_spin_unlock
>> 349359    0.3877  vmlinux                  vmlinux                  
>> kmem_cache_alloc
>> 336255    0.3731  vmlinux                  vmlinux                  
>> skb_dma_map
>> 328809    0.3649  vmlinux                  vmlinux                  
>> netif_receive_skb
>> 315826    0.3505  vmlinux                  vmlinux                  
>> fn_trie_lookup
>> 313065    0.3474  vmlinux                  vmlinux                  
>> __kmalloc
>> 307684    0.3414  vmlinux                  vmlinux                  
>> __do_softirq
>> 298676    0.3314  vmlinux                  vmlinux                  
>> skb_pull
>> 292352    0.3244  vmlinux                  vmlinux                  
>> __napi_complete
>> 288878    0.3206  vmlinux                  vmlinux                  
>> get_next_timer_interrupt
>> 280449    0.3112  vmlinux                  vmlinux                  
>> ip_finish_output
>> 264482    0.2935  vmlinux                  vmlinux                  
>> consume_skb
>> 251622    0.2792  vmlinux                  vmlinux                  
>> __remove_hrtimer
>> 227178    0.2521  vmlinux                  vmlinux                  
>> tc_classify_compat
>> 226224    0.2510  vmlinux                  vmlinux                  
>> handle_edge_irq
>>
>>
>>
>> proces that this machine is running is only quagga (zebra + bgpd + 
>> oprofiled)
>>
>>
>> Regards
>> Paweł Staszewski
>>
>>
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe netdev" in
>> the body of a message to majordomo@...r.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
> -- 
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ