lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <20200505183402.2021-1-sjpark@amazon.com>
Date:   Tue, 5 May 2020 20:34:02 +0200
From:   SeongJae Park <sjpark@...zon.com>
To:     "Paul E. McKenney" <paulmck@...nel.org>
CC:     SeongJae Park <sjpark@...zon.com>,
        Eric Dumazet <eric.dumazet@...il.com>,
        Eric Dumazet <edumazet@...gle.com>,
        David Miller <davem@...emloft.net>,
        "Al Viro" <viro@...iv.linux.org.uk>,
        Jakub Kicinski <kuba@...nel.org>,
        "Greg Kroah-Hartman" <gregkh@...uxfoundation.org>,
        <sj38.park@...il.com>, netdev <netdev@...r.kernel.org>,
        LKML <linux-kernel@...r.kernel.org>,
        SeongJae Park <sjpark@...zon.de>, <snu@...zon.com>,
        <amit@...nel.org>, <stable@...r.kernel.org>
Subject: Re: Re: Re: Re: [PATCH net v2 0/2] Revert the 'socket_alloc' life cycle change

On Tue, 5 May 2020 11:17:07 -0700 "Paul E. McKenney" <paulmck@...nel.org> wrote:

> On Tue, May 05, 2020 at 07:56:05PM +0200, SeongJae Park wrote:
> > On Tue, 5 May 2020 10:30:36 -0700 "Paul E. McKenney" <paulmck@...nel.org> wrote:
> > 
> > > On Tue, May 05, 2020 at 07:05:53PM +0200, SeongJae Park wrote:
> > > > On Tue, 5 May 2020 09:37:42 -0700 Eric Dumazet <eric.dumazet@...il.com> wrote:
> > > > 
> > > > > 
> > > > > 
> > > > > On 5/5/20 9:31 AM, Eric Dumazet wrote:
> > > > > > 
> > > > > > 
> > > > > > On 5/5/20 9:25 AM, Eric Dumazet wrote:
> > > > > >>
> > > > > >>
> > > > > >> On 5/5/20 9:13 AM, SeongJae Park wrote:
> > > > > >>> On Tue, 5 May 2020 09:00:44 -0700 Eric Dumazet <edumazet@...gle.com> wrote:
> > > > > >>>
> > > > > >>>> On Tue, May 5, 2020 at 8:47 AM SeongJae Park <sjpark@...zon.com> wrote:
> > > > > >>>>>
> > > > > >>>>> On Tue, 5 May 2020 08:20:50 -0700 Eric Dumazet <eric.dumazet@...il.com> wrote:
> > > > > >>>>>
> > > > > >>>>>>
> > > > > >>>>>>
> > > > > >>>>>> On 5/5/20 8:07 AM, SeongJae Park wrote:
> > > > > >>>>>>> On Tue, 5 May 2020 07:53:39 -0700 Eric Dumazet <edumazet@...gle.com> wrote:
> > > > > >>>>>>>
> > > > > >>>>>>
> > > > [...]
> > > > > >>
> > > > > >> I would ask Paul opinion on this issue, because we have many objects
> > > > > >> being freed after RCU grace periods.
> > > > > >>
> > > > > >> If RCU subsystem can not keep-up, I guess other workloads will also suffer.
> > > > > >>
> > > > > >> Sure, we can revert patches there and there trying to work around the issue,
> > > > > >> but for objects allocated from process context, we should not have these problems.
> > > > > >>
> > > > > > 
> > > > > > I wonder if simply adjusting rcu_divisor to 6 or 5 would help 
> > > > > > 
> > > > > > diff --git a/kernel/rcu/tree.c b/kernel/rcu/tree.c
> > > > > > index d9a49cd6065a20936edbda1b334136ab597cde52..fde833bac0f9f81e8536211b4dad6e7575c1219a 100644
> > > > > > --- a/kernel/rcu/tree.c
> > > > > > +++ b/kernel/rcu/tree.c
> > > > > > @@ -427,7 +427,7 @@ module_param(qovld, long, 0444);
> > > > > >  static ulong jiffies_till_first_fqs = ULONG_MAX;
> > > > > >  static ulong jiffies_till_next_fqs = ULONG_MAX;
> > > > > >  static bool rcu_kick_kthreads;
> > > > > > -static int rcu_divisor = 7;
> > > > > > +static int rcu_divisor = 6;
> > > > > >  module_param(rcu_divisor, int, 0644);
> > > > > >  
> > > > > >  /* Force an exit from rcu_do_batch() after 3 milliseconds. */
> > > > > > 
> > > > > 
> > > > > To be clear, you can adjust the value without building a new kernel.
> > > > > 
> > > > > echo 6 >/sys/module/rcutree/parameters/rcu_divisor
> > > > 
> > > > I tried value 6, 5, and 4, but none of those removed the problem.
> > > 
> > > Thank you for checking this!
> > > 
> > > Was your earlier discussion on long RCU readers speculation, or do you
> > > have measurements?
> > 
> > It was just a guess without any measurement or dedicated investigation.
> 
> OK, another thing to check is the duration of the low-memory episode.
> Does this duration exceed the RCU CPU stall warning time?  (21 seconds
> in mainline, 60 in many distros, but check rcupdate.rcu_cpu_stall_timeout
> to be sure.)

The benchmark takes about 36 seconds for 10,000 repeats of the test.

The value on the test machine is 60.

So the duration would not exceeded the warning time and therefore I haven't
seen the warning message.

As told in other mail, I will also adjust this value to shorter one.

> 
> Also, any chance of a .config?  Or at least the RCU portions?  I am
> guessing CONFIG_PREEMPT=n, for example.

I guess this would be ok.

    # CONFIG_PREEMPT is not set
    
    #
    # RCU Subsystem
    #
    CONFIG_TREE_RCU=y
    CONFIG_RCU_EXPERT=y
    CONFIG_SRCU=y
    CONFIG_TREE_SRCU=y
    CONFIG_RCU_STALL_COMMON=y
    CONFIG_RCU_NEED_SEGCBLIST=y
    CONFIG_RCU_FANOUT=64
    CONFIG_RCU_FANOUT_LEAF=16
    # CONFIG_RCU_FAST_NO_HZ is not set
    CONFIG_RCU_NOCB_CPU=y
    # end of RCU Subsystem


Thanks,
SeongJae Park

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ