lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <PH0PR11MB5880C49AE26E587A029B43E3DAB89@PH0PR11MB5880.namprd11.prod.outlook.com>
Date:   Tue, 28 Jun 2022 04:32:58 +0000
From:   "Zhang, Qiang1" <qiang1.zhang@...el.com>
To:     "paulmck@...nel.org" <paulmck@...nel.org>
CC:     Boqun Feng <boqun.feng@...il.com>,
        "quic_neeraju@...cinc.com" <quic_neeraju@...cinc.com>,
        "frederic@...nel.org" <frederic@...nel.org>,
        "rcu@...r.kernel.org" <rcu@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: RE: [PATCH] rcu: Add exp QS check in rcu_exp_handler() for
 no-preemptible expedited RCU

On Thu, Jun 23, 2022 at 03:23:19AM +0000, Zhang, Qiang1 wrote:
> On Wed, Jun 22, 2022 at 05:34:09PM -0700, Paul E. McKenney wrote:
> > On Wed, Jun 22, 2022 at 11:34:15PM +0000, Zhang, Qiang1 wrote:
> > > 
> > > On Wed, Jun 22, 2022 at 06:35:49PM +0800, Zqiang wrote:
> > > > In CONFIG_PREEMPT=n and CONFIG_PREEMPT_COUNT=y kernel, after a exp
> > > > grace period begins, if detected current CPU enters idle in
> > > > rcu_exp_handler() IPI handler, will immediately report the exp QS of the
> > > > current cpu, at this time, maybe not being in an RCU read-side critical
> > > > section, but need wait until rcu-softirq or sched-clock irq or sched-switch
> > > > occurs on current CPU to check and report exp QS.
> > > > 
> > > 
> > > >I think the idea is OK, however, this "optimization" is based on the
> > > >implementation detail that rcu_read_lock() counts preempt_count when
> > > >CONFIG_PREEMPT_COUNT=y, right? It's a little bit dangerous because the
> > > >preempt_count when CONFIG_PREEMPT_COUNT=y and CONFIG_PREEMPT=n is mostly
> > > >for debugging purposes IIUC, and in other words, _it could be gone_.
> > > >
> > > 
> > > Yes, for CONFIG_PREEMPT_COUNT=y and CONFIG_PREEMPT=n kernel
> > > The rcu_read_lock/unlock are replaced by preempt_disbale/enable, and the 
> > > preempt-count is exists,  so can report exp QS when not being an  RCU 
> > > read-side critical(preempt_count & (PREEMPT_MASK | SOFTIRQ_MASK )return zero).
> > > in IPI handler. 
> > > 
> > > For CONFIG_PREEMPT_COUNT=n and CONFIG_PREEMPT=n kernel, 
> > > The rcu_read_lock/unlock is just barrier(). 
> > > 
> > > 
> > > So I add IS_ENABLED(CONFIG_PREEMPT_COUNT) check in code.
> > > 
> > > Of course, for CONFIG_PREEMPT_COUNT=n  kernel, in RCU softirq, the 
> > > preempt-count is also checked
> > > 
> > > /* Report any deferred quiescent states if preemption enabled. */
> > >  if (IS_ENABLED(CONFIG_PREEMPT_COUNT) && (!(preempt_count() & PREEMPT_MASK))) {
> > >                  rcu_preempt_deferred_qs(current);
> > > 
> > > but the RCU softirq may not be triggered in time and reported exp QS, for
> > > example a kernel loop exist on NO_HZ_FULL CPU
> > > 
> > > this change, It is to capture the exp QS state earlier and report it.
> > > 
> > > 
> > > >Also I'm not aware of any but there could be someone assuming that RCU
> > > >read-side critical sections can be formed without
> > > >rcu_read_{lock,unlock}() in CONFIG_PREEMPT=n kernel. For example, there
> > > >might be "creative" code like the following:
> > > >
> > > >	void do_something_only_in_nonpreempt(void)
> > > >	{
> > > >		int *p;
> > > >
> > > >		// This function only gets called in PREEMPT=n kernel,
> > > >		// which means everywhere is a RCU read-side critical
> > > >		// section, let's save some lines of code.
> > > >
> > > 		rcu_read_lock();
> > > >		p = rcu_dereference_check(gp, !IS_ENABLED(PREEMPT));
> > > >		... // of course no schedule() here.
> > > >		<access p>
> > >                              rcu_read_unlock();
> > > >	}
> > > >
> > > 
> > > Usually access to pointers of type rcu needs to be protected.
> 
> Yes, _ususally_ they are, but what about the special cases? Because
> in PREEMPT=n kernel, almost everywhere is a RCU read-side critical
> section, some one might have been "creative" enough to omit these
> rcu_read_lock() and rcu_read_unlock().
> 
> > 
> > Indeed, lockdep would normally complain about this sort of thing.
> > But in kernels built with (say) CONFIG_PREEMPT_NONE=y but without
> > CONFIG_PREEMPT_COUNT=N, can lockdep really tell the difference?
> > 
> 
> >Actually with the rcu_dereference_check() above, lockdep cannot detect
> >even CONFIG_PREEMPT_COUNT=y, that rcu_dereference_check() basically says
> >"I know I'm in a read-side critical section if it's a non-preempt
> >kernel, so don't bother to check". ;-)
> 
> > > Any thoughts?
> > 
> > It would be good to have some performance data on this change to expedited
> > grace periods.  It is adding code, so it needs some real motivation.
> 
> I used rcuscale to test it, and  count the average time of writer-duration
> 
> no applied patch(org.txt)
> writer avg  time  29690.39     29670.78     29770.65       29423.25
> 
> applied patch(new.txt)
> writer avg time 28989.99       29003.54      29281.39       28986.58
> 
> or Is there a better way to test?

>If I understand your measurments correctly, you are getting about a 2%
>improvement in expedited grace-period latency in !PREEMPT kernels.
>Do we have a situation for which that 2% improvement is important?

To be honest, it will also be affected by the work scheduling delay. Indeed, 
my current test results are not very good, but it is beneficial to detect the
QS state in advance and report it in time. of course, I will do more tests in
the future

>We would be taking some risk due to the issues Boqun points out or we
>would be adding some complexity to avoid those issues.  So this

As for the code similar to the above scenario proposed by Boqun, 
it is true that lockdep will not complain, I do not deny that someone
will write this way.

If there really is such a situation as Boqun mentioned, There is also 
a risk in the following code segment in PREEMPT=n and PREEMPT_COUNT=y

        /* Report any deferred quiescent states if preemption enabled. */
        if (IS_ENABLED(CONFIG_PREEMPT_COUNT) && (!(preempt_count() & PREEMPT_MASK)))
                rcu_preempt_deferred_qs(current);


Thanks
Zqiang

>2% has a definite cost associated with it.
>
>What am I missing here?
>
>							Thanx, Paul
>
> Thanks
> Zqiang
> 
> 
> >Agreed.
> >
> >Regards,
> >Boqun
> 
> > So, does this change make a system-level difference in (say) expedited
> > RCU grace-period latency, and if so, under what conditions?
> > 
> > 						Thanx, Paul
> > 
> > > >Again, I'm not aware of any existing code that does this but we need to
> > > >be sure.
> > > >
> > > >Regards,
> > > >Boqun
> > > >
> > > > This commit add a exp QS check in rcu_exp_handler(), when not being
> > > > in an RCU read-side critical section, report exp QS earlier.
> > > > 
> > > > Signed-off-by: Zqiang <qiang1.zhang@...el.com>
> > > > ---
> > > >  kernel/rcu/tree_exp.h | 5 ++++-
> > > >  1 file changed, 4 insertions(+), 1 deletion(-)
> > > > 
> > > > diff --git a/kernel/rcu/tree_exp.h b/kernel/rcu/tree_exp.h
> > > > index be667583a554..34f08267410f 100644
> > > > --- a/kernel/rcu/tree_exp.h
> > > > +++ b/kernel/rcu/tree_exp.h
> > > > @@ -828,11 +828,14 @@ static void rcu_exp_handler(void *unused)
> > > >  {
> > > >  	struct rcu_data *rdp = this_cpu_ptr(&rcu_data);
> > > >  	struct rcu_node *rnp = rdp->mynode;
> > > > +	bool preempt_bh_disabled =
> > > > +				!!(preempt_count() & (PREEMPT_MASK | SOFTIRQ_MASK));
> > > >  
> > > >  	if (!(READ_ONCE(rnp->expmask) & rdp->grpmask) ||
> > > >  	    __this_cpu_read(rcu_data.cpu_no_qs.b.exp))
> > > >  		return;
> > > > -	if (rcu_is_cpu_rrupt_from_idle()) {
> > > > +	if (rcu_is_cpu_rrupt_from_idle() ||
> > > > +			(IS_ENABLED(CONFIG_PREEMPT_COUNT) && !preempt_bh_disabled)) {
> > > >  		rcu_report_exp_rdp(this_cpu_ptr(&rcu_data));
> > > >  		return;
> > > >  	}
> > > > -- 
> > > > 2.25.1
> > > > 

> [   28.907109] rcu-scale: rcu_scale_writer 3 has 100 measurements
> [   28.907133] rcu-scale: rcu_scale_writer 0 has 100 measurements
> [   28.907135] rcu-scale: rcu_scale_writer 1 has 100 measurements
> [   28.907138] rcu-scale: rcu_scale_writer 2 has 100 measurements
> [   28.917241] Dumping ftrace buffer:
> [   28.917256]    (ftrace buffer empty)
> [   28.917261] rcu-scale: Test complete
> [   49.336302] rcu-scale:!!! All grace periods expedited, no normal ones to measure!
> [   49.336329] rcu-torture: Stopping rcu_scale_reader task
> [   49.336340] rcu-torture: rcu_scale_reader is stopping
> [   49.336351] rcu-torture: Stopping rcu_scale_reader task
> [   49.336361] rcu-torture: rcu_scale_reader is stopping
> [   49.336370] rcu-torture: Stopping rcu_scale_reader task
> [   49.336405] rcu-torture: rcu_scale_reader is stopping
> [   49.336426] rcu-torture: Stopping rcu_scale_reader task
> [   49.336432] rcu-torture: rcu_scale_reader is stopping
> [   49.336443] rcu-torture: Stopping rcu_scale_writer task
> [   49.336450] rcu-torture: rcu_scale_writer is stopping
> [   49.336458] rcu-scale: writer 0 gps: 102
> [   49.336460] rcu-torture: Stopping rcu_scale_writer task
> [   49.336465] rcu-torture: rcu_scale_writer is stopping
> [   49.336517] rcu-scale: writer 1 gps: 102
> [   49.336519] rcu-torture: Stopping rcu_scale_writer task
> [   49.336524] rcu-torture: rcu_scale_writer is stopping
> [   49.336532] rcu-scale: writer 2 gps: 101
> [   49.336533] rcu-torture: Stopping rcu_scale_writer task
> [   49.336539] rcu-torture: rcu_scale_writer is stopping
> [   49.336546] rcu-scale: writer 3 gps: 102
> [   49.336548] rcu-scale: start: 28841772292 end: 28844718545 duration: 2946253 gps: 407 batches: 2
> [   49.336550] rcu-scale:    0 writer-duration:     0 25552
> [   49.339264] rcu-scale:    0 writer-duration:     1 42012
> [   49.339274] rcu-scale:    0 writer-duration:     2 24377
> [   49.339280] rcu-scale:    0 writer-duration:     3 34458
> [   49.339299] rcu-scale:    0 writer-duration:     4 27007
> [   49.339305] rcu-scale:    0 writer-duration:     5 36494
> [   49.339310] rcu-scale:    0 writer-duration:     6 24278
> [   49.339315] rcu-scale:    0 writer-duration:     7 29383
> [   49.339321] rcu-scale:    0 writer-duration:     8 26494
> [   49.339326] rcu-scale:    0 writer-duration:     9 35430
> [   49.339331] rcu-scale:    0 writer-duration:    10 23986
> [   49.339337] rcu-scale:    0 writer-duration:    11 28663
> [   49.339342] rcu-scale:    0 writer-duration:    12 26329
> [   49.339347] rcu-scale:    0 writer-duration:    13 35645
> [   49.339353] rcu-scale:    0 writer-duration:    14 23761
> [   49.339358] rcu-scale:    0 writer-duration:    15 28889
> [   49.339364] rcu-scale:    0 writer-duration:    16 26519
> [   49.339369] rcu-scale:    0 writer-duration:    17 35863
> [   49.339374] rcu-scale:    0 writer-duration:    18 23984
> [   49.339380] rcu-scale:    0 writer-duration:    19 28618
> [   49.339385] rcu-scale:    0 writer-duration:    20 26547
> [   49.339390] rcu-scale:    0 writer-duration:    21 35515
> [   49.339395] rcu-scale:    0 writer-duration:    22 23913
> [   49.339400] rcu-scale:    0 writer-duration:    23 28510
> [   49.339405] rcu-scale:    0 writer-duration:    24 26746
> [   49.339410] rcu-scale:    0 writer-duration:    25 35670
> [   49.339416] rcu-scale:    0 writer-duration:    26 23963
> [   49.339421] rcu-scale:    0 writer-duration:    27 28561
> [   49.339427] rcu-scale:    0 writer-duration:    28 26380
> [   49.339432] rcu-scale:    0 writer-duration:    29 36039
> [   49.339438] rcu-scale:    0 writer-duration:    30 24215
> [   49.339443] rcu-scale:    0 writer-duration:    31 28891
> [   49.339449] rcu-scale:    0 writer-duration:    32 26578
> [   49.339454] rcu-scale:    0 writer-duration:    33 40498
> [   49.339459] rcu-scale:    0 writer-duration:    34 24245
> [   49.339464] rcu-scale:    0 writer-duration:    35 29465
> [   49.339469] rcu-scale:    0 writer-duration:    36 26946
> [   49.339475] rcu-scale:    0 writer-duration:    37 35955
> [   49.339480] rcu-scale:    0 writer-duration:    38 24059
> [   49.339486] rcu-scale:    0 writer-duration:    39 28795
> [   49.339491] rcu-scale:    0 writer-duration:    40 26536
> [   49.339496] rcu-scale:    0 writer-duration:    41 35879
> [   49.339501] rcu-scale:    0 writer-duration:    42 24181
> [   49.339506] rcu-scale:    0 writer-duration:    43 28639
> [   49.339512] rcu-scale:    0 writer-duration:    44 26546
> [   49.339517] rcu-scale:    0 writer-duration:    45 35884
> [   49.339522] rcu-scale:    0 writer-duration:    46 23986
> [   49.339527] rcu-scale:    0 writer-duration:    47 28668
> [   49.339532] rcu-scale:    0 writer-duration:    48 26454
> [   49.339537] rcu-scale:    0 writer-duration:    49 35772
> [   49.339542] rcu-scale:    0 writer-duration:    50 23942
> [   49.339547] rcu-scale:    0 writer-duration:    51 28803
> [   49.339553] rcu-scale:    0 writer-duration:    52 26545
> [   49.339558] rcu-scale:    0 writer-duration:    53 35615
> [   49.339563] rcu-scale:    0 writer-duration:    54 23569
> [   49.339568] rcu-scale:    0 writer-duration:    55 28427
> [   49.339573] rcu-scale:    0 writer-duration:    56 26489
> [   49.339578] rcu-scale:    0 writer-duration:    57 35516
> [   49.339584] rcu-scale:    0 writer-duration:    58 23722
> [   49.339589] rcu-scale:    0 writer-duration:    59 28435
> [   49.339594] rcu-scale:    0 writer-duration:    60 26401
> [   49.339599] rcu-scale:    0 writer-duration:    61 35488
> [   49.339604] rcu-scale:    0 writer-duration:    62 23855
> [   49.339610] rcu-scale:    0 writer-duration:    63 28583
> [   49.339615] rcu-scale:    0 writer-duration:    64 26629
> [   49.339620] rcu-scale:    0 writer-duration:    65 36168
> [   49.339625] rcu-scale:    0 writer-duration:    66 24191
> [   49.339630] rcu-scale:    0 writer-duration:    67 28802
> [   49.339635] rcu-scale:    0 writer-duration:    68 32436
> [   49.339640] rcu-scale:    0 writer-duration:    69 38444
> [   49.339646] rcu-scale:    0 writer-duration:    70 24417
> [   49.339651] rcu-scale:    0 writer-duration:    71 29128
> [   49.339656] rcu-scale:    0 writer-duration:    72 26480
> [   49.339661] rcu-scale:    0 writer-duration:    73 35881
> [   49.339666] rcu-scale:    0 writer-duration:    74 24123
> [   49.339671] rcu-scale:    0 writer-duration:    75 28735
> [   49.339676] rcu-scale:    0 writer-duration:    76 26359
> [   49.339681] rcu-scale:    0 writer-duration:    77 35715
> [   49.339687] rcu-scale:    0 writer-duration:    78 24092
> [   49.339692] rcu-scale:    0 writer-duration:    79 28716
> [   49.339698] rcu-scale:    0 writer-duration:    80 26448
> [   49.339703] rcu-scale:    0 writer-duration:    81 35935
> [   49.339708] rcu-scale:    0 writer-duration:    82 23966
> [   49.339713] rcu-scale:    0 writer-duration:    83 28552
> [   49.339718] rcu-scale:    0 writer-duration:    84 26487
> [   49.339723] rcu-scale:    0 writer-duration:    85 35490
> [   49.339729] rcu-scale:    0 writer-duration:    86 23976
> [   49.339734] rcu-scale:    0 writer-duration:    87 28679
> [   49.339740] rcu-scale:    0 writer-duration:    88 26525
> [   49.339745] rcu-scale:    0 writer-duration:    89 35764
> [   49.339750] rcu-scale:    0 writer-duration:    90 24050
> [   49.339756] rcu-scale:    0 writer-duration:    91 28777
> [   49.339761] rcu-scale:    0 writer-duration:    92 26412
> [   49.339767] rcu-scale:    0 writer-duration:    93 35707
> [   49.339772] rcu-scale:    0 writer-duration:    94 23913
> [   49.339777] rcu-scale:    0 writer-duration:    95 28450
> [   49.339782] rcu-scale:    0 writer-duration:    96 26665
> [   49.339788] rcu-scale:    0 writer-duration:    97 35636
> [   49.339793] rcu-scale:    0 writer-duration:    98 23953
> [   49.339798] rcu-scale:    0 writer-duration:    99 28466
> [   49.339804] rcu-scale:    0 writer-duration:   100 26654
> [   49.341227] rcu-scale:    0 writer-duration:   101 225635
> 
> 
> 
> 
> [   49.341246] rcu-scale:    1 writer-duration:     0 27277
> [   49.343227] rcu-scale:    1 writer-duration:     1 34458
> [   49.343236] rcu-scale:    1 writer-duration:     2 27340
> [   49.343242] rcu-scale:    1 writer-duration:     3 41758
> [   49.343248] rcu-scale:    1 writer-duration:     4 24161
> [   49.343254] rcu-scale:    1 writer-duration:     5 29250
> [   49.343260] rcu-scale:    1 writer-duration:     6 26996
> [   49.343266] rcu-scale:    1 writer-duration:     7 36482
> [   49.343272] rcu-scale:    1 writer-duration:     8 23782
> [   49.343277] rcu-scale:    1 writer-duration:     9 28433
> [   49.343283] rcu-scale:    1 writer-duration:    10 26577
> [   49.343289] rcu-scale:    1 writer-duration:    11 35616
> [   49.343295] rcu-scale:    1 writer-duration:    12 23821
> [   49.343301] rcu-scale:    1 writer-duration:    13 28678
> [   49.343307] rcu-scale:    1 writer-duration:    14 26517
> [   49.343313] rcu-scale:    1 writer-duration:    15 35548
> [   49.343319] rcu-scale:    1 writer-duration:    16 24196
> [   49.343325] rcu-scale:    1 writer-duration:    17 28626
> [   49.343332] rcu-scale:    1 writer-duration:    18 26772
> [   49.343338] rcu-scale:    1 writer-duration:    19 35539
> [   49.343344] rcu-scale:    1 writer-duration:    20 23897
> [   49.343350] rcu-scale:    1 writer-duration:    21 28635
> [   49.343356] rcu-scale:    1 writer-duration:    22 26453
> [   49.343362] rcu-scale:    1 writer-duration:    23 35416
> [   49.343368] rcu-scale:    1 writer-duration:    24 24096
> [   49.343373] rcu-scale:    1 writer-duration:    25 28733
> [   49.343379] rcu-scale:    1 writer-duration:    26 26538
> [   49.343385] rcu-scale:    1 writer-duration:    27 35590
> [   49.343390] rcu-scale:    1 writer-duration:    28 23739
> [   49.343396] rcu-scale:    1 writer-duration:    29 29450
> [   49.343402] rcu-scale:    1 writer-duration:    30 26452
> [   49.343408] rcu-scale:    1 writer-duration:    31 35807
> [   49.343413] rcu-scale:    1 writer-duration:    32 24145
> [   49.343419] rcu-scale:    1 writer-duration:    33 33452
> [   49.343425] rcu-scale:    1 writer-duration:    34 26767
> [   49.343431] rcu-scale:    1 writer-duration:    35 36677
> [   49.343438] rcu-scale:    1 writer-duration:    36 24102
> [   49.343444] rcu-scale:    1 writer-duration:    37 28937
> [   49.343449] rcu-scale:    1 writer-duration:    38 26622
> [   49.343454] rcu-scale:    1 writer-duration:    39 35903
> [   49.343460] rcu-scale:    1 writer-duration:    40 23956
> [   49.343465] rcu-scale:    1 writer-duration:    41 28684
> [   49.343470] rcu-scale:    1 writer-duration:    42 26901
> [   49.343475] rcu-scale:    1 writer-duration:    43 35717
> [   49.343480] rcu-scale:    1 writer-duration:    44 23957
> [   49.343485] rcu-scale:    1 writer-duration:    45 28780
> [   49.343490] rcu-scale:    1 writer-duration:    46 26747
> [   49.343495] rcu-scale:    1 writer-duration:    47 35538
> [   49.343501] rcu-scale:    1 writer-duration:    48 23963
> [   49.343506] rcu-scale:    1 writer-duration:    49 28727
> [   49.343511] rcu-scale:    1 writer-duration:    50 26519
> [   49.343516] rcu-scale:    1 writer-duration:    51 35733
> [   49.343521] rcu-scale:    1 writer-duration:    52 24098
> [   49.343527] rcu-scale:    1 writer-duration:    53 28557
> [   49.343532] rcu-scale:    1 writer-duration:    54 26208
> [   49.343537] rcu-scale:    1 writer-duration:    55 35315
> [   49.343543] rcu-scale:    1 writer-duration:    56 23932
> [   49.343548] rcu-scale:    1 writer-duration:    57 28507
> [   49.343553] rcu-scale:    1 writer-duration:    58 26443
> [   49.343559] rcu-scale:    1 writer-duration:    59 35306
> [   49.343564] rcu-scale:    1 writer-duration:    60 23868
> [   49.343570] rcu-scale:    1 writer-duration:    61 28606
> [   49.343575] rcu-scale:    1 writer-duration:    62 26308
> [   49.343580] rcu-scale:    1 writer-duration:    63 35693
> [   49.343586] rcu-scale:    1 writer-duration:    64 23934
> [   49.343591] rcu-scale:    1 writer-duration:    65 29456
> [   49.343596] rcu-scale:    1 writer-duration:    66 26600
> [   49.343602] rcu-scale:    1 writer-duration:    67 35682
> [   49.343607] rcu-scale:    1 writer-duration:    68 29849
> [   49.343612] rcu-scale:    1 writer-duration:    69 31555
> [   49.343618] rcu-scale:    1 writer-duration:    70 26942
> [   49.343623] rcu-scale:    1 writer-duration:    71 36037
> [   49.343629] rcu-scale:    1 writer-duration:    72 23965
> [   49.343634] rcu-scale:    1 writer-duration:    73 28811
> [   49.343640] rcu-scale:    1 writer-duration:    74 26734
> [   49.343645] rcu-scale:    1 writer-duration:    75 35712
> [   49.343650] rcu-scale:    1 writer-duration:    76 23845
> [   49.343656] rcu-scale:    1 writer-duration:    77 28619
> [   49.343661] rcu-scale:    1 writer-duration:    78 26637
> [   49.343666] rcu-scale:    1 writer-duration:    79 35773
> [   49.343672] rcu-scale:    1 writer-duration:    80 23910
> [   49.343677] rcu-scale:    1 writer-duration:    81 28873
> [   49.343682] rcu-scale:    1 writer-duration:    82 26581
> [   49.343688] rcu-scale:    1 writer-duration:    83 35517
> [   49.343694] rcu-scale:    1 writer-duration:    84 23928
> [   49.343699] rcu-scale:    1 writer-duration:    85 28565
> [   49.343704] rcu-scale:    1 writer-duration:    86 26464
> [   49.343710] rcu-scale:    1 writer-duration:    87 35715
> [   49.343715] rcu-scale:    1 writer-duration:    88 23968
> [   49.343721] rcu-scale:    1 writer-duration:    89 28784
> [   49.343726] rcu-scale:    1 writer-duration:    90 26576
> [   49.343731] rcu-scale:    1 writer-duration:    91 35800
> [   49.343737] rcu-scale:    1 writer-duration:    92 23918
> [   49.343742] rcu-scale:    1 writer-duration:    93 28487
> [   49.343747] rcu-scale:    1 writer-duration:    94 26491
> [   49.343753] rcu-scale:    1 writer-duration:    95 35565
> [   49.343758] rcu-scale:    1 writer-duration:    96 24092
> [   49.343763] rcu-scale:    1 writer-duration:    97 28469
> [   49.343768] rcu-scale:    1 writer-duration:    98 26865
> [   49.343774] rcu-scale:    1 writer-duration:    99 35343
> [   49.343779] rcu-scale:    1 writer-duration:   100 23995
> [   49.345221] rcu-scale:    1 writer-duration:   101 60524
> 
> 
> 
> [   49.345235] rcu-scale:    2 writer-duration:     0 56581
> [   49.347314] rcu-scale:    2 writer-duration:     1 32281
> [   49.347329] rcu-scale:    2 writer-duration:     2 29516
> [   49.347336] rcu-scale:    2 writer-duration:     3 29979
> [   49.347342] rcu-scale:    2 writer-duration:     4 36232
> [   49.347348] rcu-scale:    2 writer-duration:     5 26878
> [   49.347354] rcu-scale:    2 writer-duration:     6 29099
> [   49.347359] rcu-scale:    2 writer-duration:     7 24890
> [   49.347365] rcu-scale:    2 writer-duration:     8 35614
> [   49.347371] rcu-scale:    2 writer-duration:     9 26166
> [   49.347377] rcu-scale:    2 writer-duration:    10 28552
> [   49.347384] rcu-scale:    2 writer-duration:    11 24335
> [   49.347390] rcu-scale:    2 writer-duration:    12 35614
> [   49.347396] rcu-scale:    2 writer-duration:    13 25978
> [   49.347402] rcu-scale:    2 writer-duration:    14 28541
> [   49.347408] rcu-scale:    2 writer-duration:    15 24634
> [   49.347413] rcu-scale:    2 writer-duration:    16 35640
> [   49.347419] rcu-scale:    2 writer-duration:    17 26235
> [   49.347425] rcu-scale:    2 writer-duration:    18 28782
> [   49.347431] rcu-scale:    2 writer-duration:    19 24288
> [   49.347437] rcu-scale:    2 writer-duration:    20 35649
> [   49.347442] rcu-scale:    2 writer-duration:    21 25989
> [   49.347448] rcu-scale:    2 writer-duration:    22 28527
> [   49.347454] rcu-scale:    2 writer-duration:    23 24150
> [   49.347460] rcu-scale:    2 writer-duration:    24 35960
> [   49.347465] rcu-scale:    2 writer-duration:    25 26133
> [   49.347471] rcu-scale:    2 writer-duration:    26 28714
> [   49.347477] rcu-scale:    2 writer-duration:    27 24192
> [   49.347482] rcu-scale:    2 writer-duration:    28 35545
> [   49.347488] rcu-scale:    2 writer-duration:    29 26893
> [   49.347494] rcu-scale:    2 writer-duration:    30 28621
> [   49.347499] rcu-scale:    2 writer-duration:    31 24453
> [   49.347505] rcu-scale:    2 writer-duration:    32 35750
> [   49.347510] rcu-scale:    2 writer-duration:    33 31021
> [   49.347516] rcu-scale:    2 writer-duration:    34 29061
> [   49.347522] rcu-scale:    2 writer-duration:    35 25015
> [   49.347528] rcu-scale:    2 writer-duration:    36 36107
> [   49.347534] rcu-scale:    2 writer-duration:    37 26370
> [   49.347540] rcu-scale:    2 writer-duration:    38 28888
> [   49.347546] rcu-scale:    2 writer-duration:    39 24418
> [   49.347552] rcu-scale:    2 writer-duration:    40 35675
> [   49.347558] rcu-scale:    2 writer-duration:    41 26284
> [   49.347564] rcu-scale:    2 writer-duration:    42 28924
> [   49.347570] rcu-scale:    2 writer-duration:    43 24244
> [   49.347576] rcu-scale:    2 writer-duration:    44 35835
> [   49.347582] rcu-scale:    2 writer-duration:    45 26194
> [   49.347587] rcu-scale:    2 writer-duration:    46 29080
> [   49.347593] rcu-scale:    2 writer-duration:    47 24185
> [   49.347599] rcu-scale:    2 writer-duration:    48 35543
> [   49.347605] rcu-scale:    2 writer-duration:    49 26122
> [   49.347610] rcu-scale:    2 writer-duration:    50 28822
> [   49.347616] rcu-scale:    2 writer-duration:    51 24285
> [   49.347622] rcu-scale:    2 writer-duration:    52 35720
> [   49.347627] rcu-scale:    2 writer-duration:    53 26063
> [   49.347633] rcu-scale:    2 writer-duration:    54 28508
> [   49.347639] rcu-scale:    2 writer-duration:    55 24019
> [   49.347645] rcu-scale:    2 writer-duration:    56 35499
> [   49.347651] rcu-scale:    2 writer-duration:    57 26079
> [   49.347657] rcu-scale:    2 writer-duration:    58 28502
> [   49.347663] rcu-scale:    2 writer-duration:    59 24016
> [   49.347668] rcu-scale:    2 writer-duration:    60 35515
> [   49.347673] rcu-scale:    2 writer-duration:    61 26095
> [   49.347677] rcu-scale:    2 writer-duration:    62 28602
> [   49.347681] rcu-scale:    2 writer-duration:    63 24246
> [   49.347686] rcu-scale:    2 writer-duration:    64 35675
> [   49.347690] rcu-scale:    2 writer-duration:    65 26761
> [   49.347694] rcu-scale:    2 writer-duration:    66 28833
> [   49.347699] rcu-scale:    2 writer-duration:    67 24524
> [   49.347703] rcu-scale:    2 writer-duration:    68 43780
> [   49.347707] rcu-scale:    2 writer-duration:    69 26612
> [   49.347712] rcu-scale:    2 writer-duration:    70 29290
> [   49.347716] rcu-scale:    2 writer-duration:    71 24670
> [   49.347720] rcu-scale:    2 writer-duration:    72 35526
> [   49.347725] rcu-scale:    2 writer-duration:    73 26393
> [   49.347729] rcu-scale:    2 writer-duration:    74 28967
> [   49.347733] rcu-scale:    2 writer-duration:    75 24125
> [   49.347738] rcu-scale:    2 writer-duration:    76 35713
> [   49.347742] rcu-scale:    2 writer-duration:    77 26062
> [   49.347746] rcu-scale:    2 writer-duration:    78 28888
> [   49.347751] rcu-scale:    2 writer-duration:    79 24344
> [   49.347755] rcu-scale:    2 writer-duration:    80 35858
> [   49.347760] rcu-scale:    2 writer-duration:    81 26330
> [   49.347764] rcu-scale:    2 writer-duration:    82 28661
> [   49.347769] rcu-scale:    2 writer-duration:    83 24180
> [   49.347773] rcu-scale:    2 writer-duration:    84 35464
> [   49.347778] rcu-scale:    2 writer-duration:    85 25886
> [   49.347782] rcu-scale:    2 writer-duration:    86 28916
> [   49.347786] rcu-scale:    2 writer-duration:    87 24149
> [   49.347791] rcu-scale:    2 writer-duration:    88 35860
> [   49.347795] rcu-scale:    2 writer-duration:    89 26100
> [   49.347800] rcu-scale:    2 writer-duration:    90 28999
> [   49.347804] rcu-scale:    2 writer-duration:    91 24105
> [   49.347809] rcu-scale:    2 writer-duration:    92 35720
> [   49.347813] rcu-scale:    2 writer-duration:    93 26093
> [   49.347817] rcu-scale:    2 writer-duration:    94 28729
> [   49.347822] rcu-scale:    2 writer-duration:    95 24063
> [   49.347826] rcu-scale:    2 writer-duration:    96 35819
> [   49.347831] rcu-scale:    2 writer-duration:    97 26119
> [   49.347835] rcu-scale:    2 writer-duration:    98 28648
> [   49.347840] rcu-scale:    2 writer-duration:    99 24175
> [   49.347844] rcu-scale:    2 writer-duration:   100 34035
> 
> 
> 
> 
> [   49.349299] rcu-scale:    3 writer-duration:     0 26939
> [   49.351268] rcu-scale:    3 writer-duration:     1 27152
> [   49.351277] rcu-scale:    3 writer-duration:     2 30299
> [   49.351281] rcu-scale:    3 writer-duration:     3 36662
> [   49.351284] rcu-scale:    3 writer-duration:     4 31847
> [   49.351287] rcu-scale:    3 writer-duration:     5 29203
> [   49.351290] rcu-scale:    3 writer-duration:     6 24861
> [   49.351294] rcu-scale:    3 writer-duration:     7 36222
> [   49.351297] rcu-scale:    3 writer-duration:     8 26790
> [   49.351300] rcu-scale:    3 writer-duration:     9 28626
> [   49.351303] rcu-scale:    3 writer-duration:    10 24007
> [   49.351307] rcu-scale:    3 writer-duration:    11 35765
> [   49.351310] rcu-scale:    3 writer-duration:    12 26226
> [   49.351314] rcu-scale:    3 writer-duration:    13 28495
> [   49.351317] rcu-scale:    3 writer-duration:    14 24188
> [   49.351321] rcu-scale:    3 writer-duration:    15 35686
> [   49.351324] rcu-scale:    3 writer-duration:    16 26290
> [   49.351327] rcu-scale:    3 writer-duration:    17 28740
> [   49.351331] rcu-scale:    3 writer-duration:    18 24201
> [   49.351334] rcu-scale:    3 writer-duration:    19 36000
> [   49.351337] rcu-scale:    3 writer-duration:    20 25995
> [   49.351341] rcu-scale:    3 writer-duration:    21 28649
> [   49.351344] rcu-scale:    3 writer-duration:    22 24081
> [   49.351348] rcu-scale:    3 writer-duration:    23 35718
> [   49.351351] rcu-scale:    3 writer-duration:    24 26077
> [   49.351354] rcu-scale:    3 writer-duration:    25 28773
> [   49.351357] rcu-scale:    3 writer-duration:    26 24280
> [   49.351360] rcu-scale:    3 writer-duration:    27 35767
> [   49.351364] rcu-scale:    3 writer-duration:    28 25956
> [   49.351367] rcu-scale:    3 writer-duration:    29 28632
> [   49.351370] rcu-scale:    3 writer-duration:    30 24996
> [   49.351374] rcu-scale:    3 writer-duration:    31 35673
> [   49.351377] rcu-scale:    3 writer-duration:    32 26217
> [   49.351380] rcu-scale:    3 writer-duration:    33 28855
> [   49.351383] rcu-scale:    3 writer-duration:    34 28655
> [   49.351387] rcu-scale:    3 writer-duration:    35 36525
> [   49.351390] rcu-scale:    3 writer-duration:    36 27052
> [   49.351394] rcu-scale:    3 writer-duration:    37 28841
> [   49.351397] rcu-scale:    3 writer-duration:    38 24600
> [   49.351400] rcu-scale:    3 writer-duration:    39 35743
> [   49.351404] rcu-scale:    3 writer-duration:    40 26391
> [   49.351407] rcu-scale:    3 writer-duration:    41 28642
> [   49.351410] rcu-scale:    3 writer-duration:    42 24334
> [   49.351413] rcu-scale:    3 writer-duration:    43 36152
> [   49.351417] rcu-scale:    3 writer-duration:    44 26133
> [   49.351420] rcu-scale:    3 writer-duration:    45 28732
> [   49.351423] rcu-scale:    3 writer-duration:    46 24235
> [   49.351426] rcu-scale:    3 writer-duration:    47 36069
> [   49.351429] rcu-scale:    3 writer-duration:    48 25993
> [   49.351433] rcu-scale:    3 writer-duration:    49 28658
> [   49.351436] rcu-scale:    3 writer-duration:    50 24225
> [   49.351439] rcu-scale:    3 writer-duration:    51 35909
> [   49.351442] rcu-scale:    3 writer-duration:    52 26176
> [   49.351446] rcu-scale:    3 writer-duration:    53 28702
> [   49.351449] rcu-scale:    3 writer-duration:    54 24169
> [   49.351453] rcu-scale:    3 writer-duration:    55 35462
> [   49.351456] rcu-scale:    3 writer-duration:    56 25771
> [   49.351459] rcu-scale:    3 writer-duration:    57 28677
> [   49.351462] rcu-scale:    3 writer-duration:    58 24089
> [   49.351466] rcu-scale:    3 writer-duration:    59 35538
> [   49.351469] rcu-scale:    3 writer-duration:    60 25785
> [   49.351472] rcu-scale:    3 writer-duration:    61 28669
> [   49.351475] rcu-scale:    3 writer-duration:    62 24075
> [   49.351478] rcu-scale:    3 writer-duration:    63 35720
> [   49.351482] rcu-scale:    3 writer-duration:    64 26017
> [   49.351485] rcu-scale:    3 writer-duration:    65 28726
> [   49.351488] rcu-scale:    3 writer-duration:    66 24990
> [   49.351491] rcu-scale:    3 writer-duration:    67 35814
> [   49.351494] rcu-scale:    3 writer-duration:    68 26215
> [   49.351498] rcu-scale:    3 writer-duration:    69 34468
> [   49.351501] rcu-scale:    3 writer-duration:    70 27200
> [   49.351504] rcu-scale:    3 writer-duration:    71 36288
> [   49.351507] rcu-scale:    3 writer-duration:    72 26445
> [   49.351511] rcu-scale:    3 writer-duration:    73 28593
> [   49.351514] rcu-scale:    3 writer-duration:    74 24317
> [   49.351517] rcu-scale:    3 writer-duration:    75 36153
> [   49.351521] rcu-scale:    3 writer-duration:    76 26164
> [   49.351524] rcu-scale:    3 writer-duration:    77 28523
> [   49.351527] rcu-scale:    3 writer-duration:    78 24252
> [   49.351530] rcu-scale:    3 writer-duration:    79 35939
> [   49.351533] rcu-scale:    3 writer-duration:    80 26058
> [   49.351536] rcu-scale:    3 writer-duration:    81 28892
> [   49.351540] rcu-scale:    3 writer-duration:    82 24342
> [   49.351543] rcu-scale:    3 writer-duration:    83 35751
> [   49.351546] rcu-scale:    3 writer-duration:    84 25932
> [   49.351549] rcu-scale:    3 writer-duration:    85 28682
> [   49.351552] rcu-scale:    3 writer-duration:    86 24086
> [   49.351555] rcu-scale:    3 writer-duration:    87 35791
> [   49.351559] rcu-scale:    3 writer-duration:    88 26189
> [   49.351562] rcu-scale:    3 writer-duration:    89 28754
> [   49.351565] rcu-scale:    3 writer-duration:    90 24250
> [   49.351568] rcu-scale:    3 writer-duration:    91 35768
> [   49.351572] rcu-scale:    3 writer-duration:    92 26160
> [   49.351575] rcu-scale:    3 writer-duration:    93 28896
> [   49.351578] rcu-scale:    3 writer-duration:    94 24020
> [   49.351581] rcu-scale:    3 writer-duration:    95 35727
> [   49.351584] rcu-scale:    3 writer-duration:    96 26099
> [   49.351587] rcu-scale:    3 writer-duration:    97 28768
> [   49.351590] rcu-scale:    3 writer-duration:    98 24011
> [   49.351593] rcu-scale:    3 writer-duration:    99 35786
> [   49.351597] rcu-scale:    3 writer-duration:   100 25989
> [   49.353257] rcu-scale:    3 writer-duration:   101 46411
> 
> 

> [   47.850402] rcu-scale: rcu_scale_writer 3 has 100 measurements
> [   47.850427] rcu-scale: rcu_scale_writer 0 has 100 measurements
> [   47.850429] rcu-scale: rcu_scale_writer 1 has 100 measurements
> [   47.850432] rcu-scale: rcu_scale_writer 2 has 100 measurements
> [   47.861470] Dumping ftrace buffer:
> [   47.861486]    (ftrace buffer empty)
> [   47.861491] rcu-scale: Test complete
> [   63.668029] rcu-scale:!!! All grace periods expedited, no normal ones to measure!
> [   63.668051] rcu-torture: Stopping rcu_scale_reader task
> [   63.668074] rcu-torture: rcu_scale_reader is stopping
> [   63.668089] rcu-torture: Stopping rcu_scale_reader task
> [   63.668138] rcu-torture: rcu_scale_reader is stopping
> [   63.668151] rcu-torture: Stopping rcu_scale_reader task
> [   63.668191] rcu-torture: rcu_scale_reader is stopping
> [   63.668204] rcu-torture: Stopping rcu_scale_reader task
> [   63.668209] rcu-torture: rcu_scale_reader is stopping
> [   63.668231] rcu-torture: Stopping rcu_scale_writer task
> [   63.668241] rcu-torture: rcu_scale_writer is stopping
> [   63.668249] rcu-scale: writer 0 gps: 102
> [   63.668251] rcu-torture: Stopping rcu_scale_writer task
> [   63.668260] rcu-torture: rcu_scale_writer is stopping
> [   63.668268] rcu-scale: writer 1 gps: 102
> [   63.668270] rcu-torture: Stopping rcu_scale_writer task
> [   63.668279] rcu-torture: rcu_scale_writer is stopping
> [   63.668287] rcu-scale: writer 2 gps: 101
> [   63.668288] rcu-torture: Stopping rcu_scale_writer task
> [   63.668293] rcu-torture: rcu_scale_writer is stopping
> [   63.668310] rcu-scale: writer 3 gps: 102
> [   63.668311] rcu-scale: start: 47784890510 end: 47787901801 duration: 3011291 gps: 407 batches: 2
> [   63.668314] rcu-scale:    0 writer-duration:     0 55830
> [   63.669543] rcu-scale:    0 writer-duration:     1 40421
> [   63.669546] rcu-scale:    0 writer-duration:     2 25268
> [   63.669548] rcu-scale:    0 writer-duration:     3 32703
> [   63.669549] rcu-scale:    0 writer-duration:     4 28624
> [   63.669550] rcu-scale:    0 writer-duration:     5 36967
> [   63.669551] rcu-scale:    0 writer-duration:     6 24686
> [   63.669552] rcu-scale:    0 writer-duration:     7 29598
> [   63.669554] rcu-scale:    0 writer-duration:     8 27026
> [   63.669555] rcu-scale:    0 writer-duration:     9 38789
> [   63.669556] rcu-scale:    0 writer-duration:    10 24398
> [   63.669557] rcu-scale:    0 writer-duration:    11 29562
> [   63.669559] rcu-scale:    0 writer-duration:    12 27348
> [   63.669560] rcu-scale:    0 writer-duration:    13 36315
> [   63.669561] rcu-scale:    0 writer-duration:    14 24233
> [   63.669562] rcu-scale:    0 writer-duration:    15 29035
> [   63.669564] rcu-scale:    0 writer-duration:    16 26754
> [   63.669606] rcu-scale:    0 writer-duration:    17 36272
> [   63.669608] rcu-scale:    0 writer-duration:    18 24237
> [   63.669609] rcu-scale:    0 writer-duration:    19 28834
> [   63.669610] rcu-scale:    0 writer-duration:    20 26993
> [   63.669611] rcu-scale:    0 writer-duration:    21 36432
> [   63.669612] rcu-scale:    0 writer-duration:    22 24178
> [   63.669614] rcu-scale:    0 writer-duration:    23 28838
> [   63.669615] rcu-scale:    0 writer-duration:    24 26733
> [   63.669616] rcu-scale:    0 writer-duration:    25 36087
> [   63.669617] rcu-scale:    0 writer-duration:    26 24237
> [   63.669618] rcu-scale:    0 writer-duration:    27 28656
> [   63.669620] rcu-scale:    0 writer-duration:    28 26921
> [   63.669621] rcu-scale:    0 writer-duration:    29 36423
> [   63.669622] rcu-scale:    0 writer-duration:    30 24201
> [   63.669623] rcu-scale:    0 writer-duration:    31 28748
> [   63.669625] rcu-scale:    0 writer-duration:    32 31925
> [   63.669626] rcu-scale:    0 writer-duration:    33 36673
> [   63.669627] rcu-scale:    0 writer-duration:    34 24630
> [   63.669628] rcu-scale:    0 writer-duration:    35 29033
> [   63.669629] rcu-scale:    0 writer-duration:    36 26761
> [   63.669631] rcu-scale:    0 writer-duration:    37 36495
> [   63.669632] rcu-scale:    0 writer-duration:    38 24209
> [   63.669633] rcu-scale:    0 writer-duration:    39 29468
> [   63.669634] rcu-scale:    0 writer-duration:    40 27137
> [   63.669635] rcu-scale:    0 writer-duration:    41 36193
> [   63.669637] rcu-scale:    0 writer-duration:    42 24531
> [   63.669638] rcu-scale:    0 writer-duration:    43 29113
> [   63.669639] rcu-scale:    0 writer-duration:    44 26915
> [   63.669640] rcu-scale:    0 writer-duration:    45 36394
> [   63.669642] rcu-scale:    0 writer-duration:    46 24247
> [   63.669643] rcu-scale:    0 writer-duration:    47 28823
> [   63.669644] rcu-scale:    0 writer-duration:    48 27029
> [   63.669645] rcu-scale:    0 writer-duration:    49 36042
> [   63.669646] rcu-scale:    0 writer-duration:    50 24164
> [   63.669648] rcu-scale:    0 writer-duration:    51 28972
> [   63.669649] rcu-scale:    0 writer-duration:    52 26858
> [   63.669650] rcu-scale:    0 writer-duration:    53 36319
> [   63.669651] rcu-scale:    0 writer-duration:    54 24233
> [   63.669652] rcu-scale:    0 writer-duration:    55 28986
> [   63.669654] rcu-scale:    0 writer-duration:    56 26895
> [   63.669655] rcu-scale:    0 writer-duration:    57 36213
> [   63.669656] rcu-scale:    0 writer-duration:    58 24156
> [   63.669657] rcu-scale:    0 writer-duration:    59 28859
> [   63.669658] rcu-scale:    0 writer-duration:    60 26917
> [   63.669660] rcu-scale:    0 writer-duration:    61 36174
> [   63.669661] rcu-scale:    0 writer-duration:    62 24237
> [   63.669662] rcu-scale:    0 writer-duration:    63 28873
> [   63.669663] rcu-scale:    0 writer-duration:    64 27045
> [   63.669664] rcu-scale:    0 writer-duration:    65 36167
> [   63.669666] rcu-scale:    0 writer-duration:    66 29556
> [   63.669667] rcu-scale:    0 writer-duration:    67 28900
> [   63.669668] rcu-scale:    0 writer-duration:    68 27072
> [   63.669669] rcu-scale:    0 writer-duration:    69 36733
> [   63.669670] rcu-scale:    0 writer-duration:    70 24327
> [   63.669672] rcu-scale:    0 writer-duration:    71 28842
> [   63.669673] rcu-scale:    0 writer-duration:    72 27027
> [   63.669674] rcu-scale:    0 writer-duration:    73 36228
> [   63.669675] rcu-scale:    0 writer-duration:    74 24137
> [   63.669677] rcu-scale:    0 writer-duration:    75 29924
> [   63.669678] rcu-scale:    0 writer-duration:    76 26775
> [   63.669679] rcu-scale:    0 writer-duration:    77 36300
> [   63.669680] rcu-scale:    0 writer-duration:    78 24547
> [   63.669681] rcu-scale:    0 writer-duration:    79 28852
> [   63.669683] rcu-scale:    0 writer-duration:    80 26945
> [   63.669684] rcu-scale:    0 writer-duration:    81 36127
> [   63.669685] rcu-scale:    0 writer-duration:    82 24373
> [   63.669686] rcu-scale:    0 writer-duration:    83 28853
> [   63.669687] rcu-scale:    0 writer-duration:    84 26652
> [   63.669689] rcu-scale:    0 writer-duration:    85 35952
> [   63.669690] rcu-scale:    0 writer-duration:    86 24379
> [   63.669691] rcu-scale:    0 writer-duration:    87 28948
> [   63.669692] rcu-scale:    0 writer-duration:    88 26958
> [   63.669693] rcu-scale:    0 writer-duration:    89 36302
> [   63.669695] rcu-scale:    0 writer-duration:    90 24181
> [   63.669696] rcu-scale:    0 writer-duration:    91 29120
> [   63.669697] rcu-scale:    0 writer-duration:    92 27032
> [   63.669698] rcu-scale:    0 writer-duration:    93 36083
> [   63.669699] rcu-scale:    0 writer-duration:    94 24248
> [   63.669701] rcu-scale:    0 writer-duration:    95 29205
> [   63.669702] rcu-scale:    0 writer-duration:    96 26577
> [   63.669703] rcu-scale:    0 writer-duration:    97 36187
> [   63.669704] rcu-scale:    0 writer-duration:    98 24472
> [   63.669705] rcu-scale:    0 writer-duration:    99 28883
> [   63.669707] rcu-scale:    0 writer-duration:   100 31999
> [   63.671532] rcu-scale:    0 writer-duration:   101 48271
> 
> 
> [   63.671556] rcu-scale:    1 writer-duration:     0 53537
> [   63.673442] rcu-scale:    1 writer-duration:     1 33311
> [   63.673451] rcu-scale:    1 writer-duration:     2 28033
> [   63.673457] rcu-scale:    1 writer-duration:     3 40200
> [   63.673464] rcu-scale:    1 writer-duration:     4 25479
> [   63.673470] rcu-scale:    1 writer-duration:     5 29908
> [   63.673475] rcu-scale:    1 writer-duration:     6 27336
> [   63.673481] rcu-scale:    1 writer-duration:     7 36948
> [   63.673487] rcu-scale:    1 writer-duration:     8 24216
> [   63.673494] rcu-scale:    1 writer-duration:     9 31639
> [   63.673499] rcu-scale:    1 writer-duration:    10 27175
> [   63.673506] rcu-scale:    1 writer-duration:    11 36695
> [   63.673512] rcu-scale:    1 writer-duration:    12 24609
> [   63.673518] rcu-scale:    1 writer-duration:    13 29140
> [   63.673524] rcu-scale:    1 writer-duration:    14 26955
> [   63.673530] rcu-scale:    1 writer-duration:    15 36122
> [   63.673535] rcu-scale:    1 writer-duration:    16 24236
> [   63.673541] rcu-scale:    1 writer-duration:    17 29137
> [   63.673546] rcu-scale:    1 writer-duration:    18 26766
> [   63.673552] rcu-scale:    1 writer-duration:    19 36207
> [   63.673558] rcu-scale:    1 writer-duration:    20 24215
> [   63.673563] rcu-scale:    1 writer-duration:    21 29239
> [   63.673569] rcu-scale:    1 writer-duration:    22 26866
> [   63.673575] rcu-scale:    1 writer-duration:    23 36054
> [   63.673581] rcu-scale:    1 writer-duration:    24 24012
> [   63.673586] rcu-scale:    1 writer-duration:    25 28914
> [   63.673592] rcu-scale:    1 writer-duration:    26 26891
> [   63.673597] rcu-scale:    1 writer-duration:    27 36102
> [   63.673603] rcu-scale:    1 writer-duration:    28 24039
> [   63.673609] rcu-scale:    1 writer-duration:    29 29125
> [   63.673614] rcu-scale:    1 writer-duration:    30 26926
> [   63.673620] rcu-scale:    1 writer-duration:    31 40754
> [   63.673625] rcu-scale:    1 writer-duration:    32 24519
> [   63.673631] rcu-scale:    1 writer-duration:    33 29397
> [   63.673637] rcu-scale:    1 writer-duration:    34 27267
> [   63.673642] rcu-scale:    1 writer-duration:    35 36358
> [   63.673648] rcu-scale:    1 writer-duration:    36 24007
> [   63.673654] rcu-scale:    1 writer-duration:    37 29326
> [   63.673659] rcu-scale:    1 writer-duration:    38 26854
> [   63.673665] rcu-scale:    1 writer-duration:    39 36625
> [   63.673671] rcu-scale:    1 writer-duration:    40 24437
> [   63.673676] rcu-scale:    1 writer-duration:    41 29314
> [   63.673682] rcu-scale:    1 writer-duration:    42 26955
> [   63.673688] rcu-scale:    1 writer-duration:    43 36415
> [   63.673693] rcu-scale:    1 writer-duration:    44 24098
> [   63.673699] rcu-scale:    1 writer-duration:    45 29216
> [   63.673705] rcu-scale:    1 writer-duration:    46 26904
> [   63.673710] rcu-scale:    1 writer-duration:    47 36286
> [   63.673716] rcu-scale:    1 writer-duration:    48 24127
> [   63.673722] rcu-scale:    1 writer-duration:    49 28887
> [   63.673727] rcu-scale:    1 writer-duration:    50 26894
> [   63.673733] rcu-scale:    1 writer-duration:    51 36145
> [   63.673739] rcu-scale:    1 writer-duration:    52 24238
> [   63.673745] rcu-scale:    1 writer-duration:    53 29136
> [   63.673750] rcu-scale:    1 writer-duration:    54 26895
> [   63.673756] rcu-scale:    1 writer-duration:    55 36087
> [   63.673761] rcu-scale:    1 writer-duration:    56 24215
> [   63.673767] rcu-scale:    1 writer-duration:    57 29034
> [   63.673773] rcu-scale:    1 writer-duration:    58 26817
> [   63.673778] rcu-scale:    1 writer-duration:    59 36069
> [   63.673784] rcu-scale:    1 writer-duration:    60 24255
> [   63.673790] rcu-scale:    1 writer-duration:    61 29004
> [   63.673795] rcu-scale:    1 writer-duration:    62 26878
> [   63.673801] rcu-scale:    1 writer-duration:    63 36186
> [   63.673806] rcu-scale:    1 writer-duration:    64 24355
> [   63.673812] rcu-scale:    1 writer-duration:    65 28884
> [   63.673818] rcu-scale:    1 writer-duration:    66 32061
> [   63.673823] rcu-scale:    1 writer-duration:    67 36261
> [   63.673829] rcu-scale:    1 writer-duration:    68 24453
> [   63.673835] rcu-scale:    1 writer-duration:    69 29508
> [   63.673840] rcu-scale:    1 writer-duration:    70 27004
> [   63.673846] rcu-scale:    1 writer-duration:    71 36240
> [   63.673851] rcu-scale:    1 writer-duration:    72 23958
> [   63.673857] rcu-scale:    1 writer-duration:    73 29276
> [   63.673863] rcu-scale:    1 writer-duration:    74 26793
> [   63.673868] rcu-scale:    1 writer-duration:    75 36779
> [   63.673874] rcu-scale:    1 writer-duration:    76 24462
> [   63.673880] rcu-scale:    1 writer-duration:    77 29156
> [   63.673885] rcu-scale:    1 writer-duration:    78 27128
> [   63.673891] rcu-scale:    1 writer-duration:    79 36182
> [   63.673896] rcu-scale:    1 writer-duration:    80 24073
> [   63.673902] rcu-scale:    1 writer-duration:    81 28955
> [   63.673908] rcu-scale:    1 writer-duration:    82 27009
> [   63.673914] rcu-scale:    1 writer-duration:    83 36169
> [   63.673919] rcu-scale:    1 writer-duration:    84 23906
> [   63.673925] rcu-scale:    1 writer-duration:    85 28946
> [   63.673931] rcu-scale:    1 writer-duration:    86 26866
> [   63.673937] rcu-scale:    1 writer-duration:    87 36249
> [   63.673942] rcu-scale:    1 writer-duration:    88 24096
> [   63.673948] rcu-scale:    1 writer-duration:    89 29290
> [   63.673954] rcu-scale:    1 writer-duration:    90 26844
> [   63.673959] rcu-scale:    1 writer-duration:    91 36353
> [   63.673965] rcu-scale:    1 writer-duration:    92 24166
> [   63.673971] rcu-scale:    1 writer-duration:    93 29072
> [   63.673976] rcu-scale:    1 writer-duration:    94 26862
> [   63.673982] rcu-scale:    1 writer-duration:    95 36348
> [   63.673988] rcu-scale:    1 writer-duration:    96 23907
> [   63.673993] rcu-scale:    1 writer-duration:    97 29122
> [   63.673999] rcu-scale:    1 writer-duration:    98 27072
> [   63.674004] rcu-scale:    1 writer-duration:    99 36188
> [   63.674010] rcu-scale:    1 writer-duration:   100 29125
> [   63.675441] rcu-scale:    1 writer-duration:   101 48063
> 
> 
> 
> [   63.675458] rcu-scale:    2 writer-duration:     0 65839
> [   63.677455] rcu-scale:    2 writer-duration:     1 30102
> [   63.677470] rcu-scale:    2 writer-duration:     2 30481
> [   63.677477] rcu-scale:    2 writer-duration:     3 28129
> [   63.677483] rcu-scale:    2 writer-duration:     4 37897
> [   63.677489] rcu-scale:    2 writer-duration:     5 27420
> [   63.677496] rcu-scale:    2 writer-duration:     6 29475
> [   63.677501] rcu-scale:    2 writer-duration:     7 25128
> [   63.677507] rcu-scale:    2 writer-duration:     8 36449
> [   63.677512] rcu-scale:    2 writer-duration:     9 29093
> [   63.677518] rcu-scale:    2 writer-duration:    10 29369
> [   63.677524] rcu-scale:    2 writer-duration:    11 24938
> [   63.677530] rcu-scale:    2 writer-duration:    12 36634
> [   63.677535] rcu-scale:    2 writer-duration:    13 26628
> [   63.677541] rcu-scale:    2 writer-duration:    14 29116
> [   63.677547] rcu-scale:    2 writer-duration:    15 24320
> [   63.677552] rcu-scale:    2 writer-duration:    16 36351
> [   63.677558] rcu-scale:    2 writer-duration:    17 26613
> [   63.677564] rcu-scale:    2 writer-duration:    18 29039
> [   63.677570] rcu-scale:    2 writer-duration:    19 24353
> [   63.677575] rcu-scale:    2 writer-duration:    20 36335
> [   63.677581] rcu-scale:    2 writer-duration:    21 26706
> [   63.677587] rcu-scale:    2 writer-duration:    22 29200
> [   63.677592] rcu-scale:    2 writer-duration:    23 24211
> [   63.677598] rcu-scale:    2 writer-duration:    24 36036
> [   63.677604] rcu-scale:    2 writer-duration:    25 26464
> [   63.677610] rcu-scale:    2 writer-duration:    26 29181
> [   63.677616] rcu-scale:    2 writer-duration:    27 24021
> [   63.677621] rcu-scale:    2 writer-duration:    28 36169
> [   63.677627] rcu-scale:    2 writer-duration:    29 26697
> [   63.677633] rcu-scale:    2 writer-duration:    30 29213
> [   63.677638] rcu-scale:    2 writer-duration:    31 24181
> [   63.677644] rcu-scale:    2 writer-duration:    32 41287
> [   63.677650] rcu-scale:    2 writer-duration:    33 27016
> [   63.677656] rcu-scale:    2 writer-duration:    34 29455
> [   63.677662] rcu-scale:    2 writer-duration:    35 24571
> [   63.677667] rcu-scale:    2 writer-duration:    36 36144
> [   63.677673] rcu-scale:    2 writer-duration:    37 26673
> [   63.677678] rcu-scale:    2 writer-duration:    38 29031
> [   63.677684] rcu-scale:    2 writer-duration:    39 25127
> [   63.677690] rcu-scale:    2 writer-duration:    40 36453
> [   63.677695] rcu-scale:    2 writer-duration:    41 26455
> [   63.677701] rcu-scale:    2 writer-duration:    42 29460
> [   63.677707] rcu-scale:    2 writer-duration:    43 24698
> [   63.677712] rcu-scale:    2 writer-duration:    44 36093
> [   63.677718] rcu-scale:    2 writer-duration:    45 26573
> [   63.677723] rcu-scale:    2 writer-duration:    46 29157
> [   63.677729] rcu-scale:    2 writer-duration:    47 24356
> [   63.677735] rcu-scale:    2 writer-duration:    48 36504
> [   63.677741] rcu-scale:    2 writer-duration:    49 26157
> [   63.677746] rcu-scale:    2 writer-duration:    50 29103
> [   63.677752] rcu-scale:    2 writer-duration:    51 24577
> [   63.677758] rcu-scale:    2 writer-duration:    52 36218
> [   63.677763] rcu-scale:    2 writer-duration:    53 26553
> [   63.677769] rcu-scale:    2 writer-duration:    54 29224
> [   63.677775] rcu-scale:    2 writer-duration:    55 24290
> [   63.677780] rcu-scale:    2 writer-duration:    56 36322
> [   63.677786] rcu-scale:    2 writer-duration:    57 26509
> [   63.677792] rcu-scale:    2 writer-duration:    58 28951
> [   63.677797] rcu-scale:    2 writer-duration:    59 24316
> [   63.677803] rcu-scale:    2 writer-duration:    60 36402
> [   63.677809] rcu-scale:    2 writer-duration:    61 26453
> [   63.677814] rcu-scale:    2 writer-duration:    62 29023
> [   63.677820] rcu-scale:    2 writer-duration:    63 24538
> [   63.677826] rcu-scale:    2 writer-duration:    64 36279
> [   63.677831] rcu-scale:    2 writer-duration:    65 26277
> [   63.677837] rcu-scale:    2 writer-duration:    66 34442
> [   63.677843] rcu-scale:    2 writer-duration:    67 24564
> [   63.677849] rcu-scale:    2 writer-duration:    68 36429
> [   63.677855] rcu-scale:    2 writer-duration:    69 27018
> [   63.677861] rcu-scale:    2 writer-duration:    70 29207
> [   63.677866] rcu-scale:    2 writer-duration:    71 24488
> [   63.677872] rcu-scale:    2 writer-duration:    72 36187
> [   63.677878] rcu-scale:    2 writer-duration:    73 26564
> [   63.677883] rcu-scale:    2 writer-duration:    74 29127
> [   63.677889] rcu-scale:    2 writer-duration:    75 25213
> [   63.677894] rcu-scale:    2 writer-duration:    76 36202
> [   63.677900] rcu-scale:    2 writer-duration:    77 26709
> [   63.677905] rcu-scale:    2 writer-duration:    78 29292
> [   63.677911] rcu-scale:    2 writer-duration:    79 24338
> [   63.677917] rcu-scale:    2 writer-duration:    80 36322
> [   63.677922] rcu-scale:    2 writer-duration:    81 26289
> [   63.677928] rcu-scale:    2 writer-duration:    82 29386
> [   63.677934] rcu-scale:    2 writer-duration:    83 24300
> [   63.677939] rcu-scale:    2 writer-duration:    84 36011
> [   63.677945] rcu-scale:    2 writer-duration:    85 26219
> [   63.677951] rcu-scale:    2 writer-duration:    86 29250
> [   63.677957] rcu-scale:    2 writer-duration:    87 24484
> [   63.677963] rcu-scale:    2 writer-duration:    88 36254
> [   63.677969] rcu-scale:    2 writer-duration:    89 26631
> [   63.677975] rcu-scale:    2 writer-duration:    90 29105
> [   63.677980] rcu-scale:    2 writer-duration:    91 24502
> [   63.677986] rcu-scale:    2 writer-duration:    92 36409
> [   63.677992] rcu-scale:    2 writer-duration:    93 26414
> [   63.677997] rcu-scale:    2 writer-duration:    94 29130
> [   63.678003] rcu-scale:    2 writer-duration:    95 24586
> [   63.678009] rcu-scale:    2 writer-duration:    96 36159
> [   63.678014] rcu-scale:    2 writer-duration:    97 26263
> [   63.678021] rcu-scale:    2 writer-duration:    98 29479
> [   63.678027] rcu-scale:    2 writer-duration:    99 24352
> [   63.678032] rcu-scale:    2 writer-duration:   100 39478
> 
> 
> 
> 
> [   63.679533] rcu-scale:    3 writer-duration:     0 31164
> [   63.681440] rcu-scale:    3 writer-duration:     1 31690
> [   63.681451] rcu-scale:    3 writer-duration:     2 28398
> [   63.681457] rcu-scale:    3 writer-duration:     3 37524
> [   63.681462] rcu-scale:    3 writer-duration:     4 30331
> [   63.681468] rcu-scale:    3 writer-duration:     5 30588
> [   63.681473] rcu-scale:    3 writer-duration:     6 25430
> [   63.681479] rcu-scale:    3 writer-duration:     7 36628
> [   63.681485] rcu-scale:    3 writer-duration:     8 27195
> [   63.681490] rcu-scale:    3 writer-duration:     9 29267
> [   63.681496] rcu-scale:    3 writer-duration:    10 26967
> [   63.681503] rcu-scale:    3 writer-duration:    11 36610
> [   63.681509] rcu-scale:    3 writer-duration:    12 27017
> [   63.681514] rcu-scale:    3 writer-duration:    13 29493
> [   63.681520] rcu-scale:    3 writer-duration:    14 24647
> [   63.681526] rcu-scale:    3 writer-duration:    15 36099
> [   63.681531] rcu-scale:    3 writer-duration:    16 26574
> [   63.681537] rcu-scale:    3 writer-duration:    17 28995
> [   63.681542] rcu-scale:    3 writer-duration:    18 24627
> [   63.681548] rcu-scale:    3 writer-duration:    19 36099
> [   63.681554] rcu-scale:    3 writer-duration:    20 26580
> [   63.681559] rcu-scale:    3 writer-duration:    21 29133
> [   63.681565] rcu-scale:    3 writer-duration:    22 24684
> [   63.681570] rcu-scale:    3 writer-duration:    23 36089
> [   63.681576] rcu-scale:    3 writer-duration:    24 26508
> [   63.681582] rcu-scale:    3 writer-duration:    25 28821
> [   63.681587] rcu-scale:    3 writer-duration:    26 24382
> [   63.681593] rcu-scale:    3 writer-duration:    27 36151
> [   63.681599] rcu-scale:    3 writer-duration:    28 26336
> [   63.681605] rcu-scale:    3 writer-duration:    29 28946
> [   63.681610] rcu-scale:    3 writer-duration:    30 24735
> [   63.681616] rcu-scale:    3 writer-duration:    31 36237
> [   63.681622] rcu-scale:    3 writer-duration:    32 26239
> [   63.681627] rcu-scale:    3 writer-duration:    33 34242
> [   63.681633] rcu-scale:    3 writer-duration:    34 24917
> [   63.681639] rcu-scale:    3 writer-duration:    35 36459
> [   63.681645] rcu-scale:    3 writer-duration:    36 26774
> [   63.681650] rcu-scale:    3 writer-duration:    37 28911
> [   63.681656] rcu-scale:    3 writer-duration:    38 24797
> [   63.681662] rcu-scale:    3 writer-duration:    39 36099
> [   63.681667] rcu-scale:    3 writer-duration:    40 27206
> [   63.681673] rcu-scale:    3 writer-duration:    41 29201
> [   63.681678] rcu-scale:    3 writer-duration:    42 24676
> [   63.681684] rcu-scale:    3 writer-duration:    43 36260
> [   63.681690] rcu-scale:    3 writer-duration:    44 26859
> [   63.681695] rcu-scale:    3 writer-duration:    45 29087
> [   63.681701] rcu-scale:    3 writer-duration:    46 24531
> [   63.681707] rcu-scale:    3 writer-duration:    47 36144
> [   63.681712] rcu-scale:    3 writer-duration:    48 26415
> [   63.681718] rcu-scale:    3 writer-duration:    49 29230
> [   63.681724] rcu-scale:    3 writer-duration:    50 24371
> [   63.681729] rcu-scale:    3 writer-duration:    51 36045
> [   63.681735] rcu-scale:    3 writer-duration:    52 26707
> [   63.681740] rcu-scale:    3 writer-duration:    53 29079
> [   63.681746] rcu-scale:    3 writer-duration:    54 24622
> [   63.681752] rcu-scale:    3 writer-duration:    55 36202
> [   63.681757] rcu-scale:    3 writer-duration:    56 26489
> [   63.681763] rcu-scale:    3 writer-duration:    57 29016
> [   63.681769] rcu-scale:    3 writer-duration:    58 24632
> [   63.681775] rcu-scale:    3 writer-duration:    59 35968
> [   63.681780] rcu-scale:    3 writer-duration:    60 26480
> [   63.681786] rcu-scale:    3 writer-duration:    61 29171
> [   63.681792] rcu-scale:    3 writer-duration:    62 24542
> [   63.681797] rcu-scale:    3 writer-duration:    63 35972
> [   63.681803] rcu-scale:    3 writer-duration:    64 26459
> [   63.681809] rcu-scale:    3 writer-duration:    65 29248
> [   63.681814] rcu-scale:    3 writer-duration:    66 24403
> [   63.681820] rcu-scale:    3 writer-duration:    67 41353
> [   63.681826] rcu-scale:    3 writer-duration:    68 26759
> [   63.681831] rcu-scale:    3 writer-duration:    69 29167
> [   63.681837] rcu-scale:    3 writer-duration:    70 25051
> [   63.681843] rcu-scale:    3 writer-duration:    71 36298
> [   63.681848] rcu-scale:    3 writer-duration:    72 26557
> [   63.681854] rcu-scale:    3 writer-duration:    73 28940
> [   63.681860] rcu-scale:    3 writer-duration:    74 24726
> [   63.681865] rcu-scale:    3 writer-duration:    75 36109
> [   63.681871] rcu-scale:    3 writer-duration:    76 27216
> [   63.681877] rcu-scale:    3 writer-duration:    77 29170
> [   63.681882] rcu-scale:    3 writer-duration:    78 24606
> [   63.681888] rcu-scale:    3 writer-duration:    79 36459
> [   63.681893] rcu-scale:    3 writer-duration:    80 26470
> [   63.681899] rcu-scale:    3 writer-duration:    81 28939
> [   63.681905] rcu-scale:    3 writer-duration:    82 24758
> [   63.681910] rcu-scale:    3 writer-duration:    83 36139
> [   63.681916] rcu-scale:    3 writer-duration:    84 26535
> [   63.681922] rcu-scale:    3 writer-duration:    85 28824
> [   63.681927] rcu-scale:    3 writer-duration:    86 24272
> [   63.681933] rcu-scale:    3 writer-duration:    87 36187
> [   63.681939] rcu-scale:    3 writer-duration:    88 26565
> [   63.681944] rcu-scale:    3 writer-duration:    89 29050
> [   63.681950] rcu-scale:    3 writer-duration:    90 24686
> [   63.681955] rcu-scale:    3 writer-duration:    91 36216
> [   63.681961] rcu-scale:    3 writer-duration:    92 26589
> [   63.681967] rcu-scale:    3 writer-duration:    93 29076
> [   63.681973] rcu-scale:    3 writer-duration:    94 24683
> [   63.681978] rcu-scale:    3 writer-duration:    95 36081
> [   63.681984] rcu-scale:    3 writer-duration:    96 26710
> [   63.681989] rcu-scale:    3 writer-duration:    97 28881
> [   63.681995] rcu-scale:    3 writer-duration:    98 24494
> [   63.682001] rcu-scale:    3 writer-duration:    99 36297
> [   63.682006] rcu-scale:    3 writer-duration:   100 26487
> 
> 
> [   63.683440] rcu-scale:    3 writer-duration:   101 48416

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ