netdev - Re: Long delay on estimation_timer causes packet latency

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Fri, 4 Dec 2020 21:59:02 +0800
From:   "dust.li" <dust.li@...ux.alibaba.com>
To:     Julian Anastasov <ja@....bg>
Cc:     yunhong-cgl jiang <xintian1976@...il.com>, horms@...ge.net.au,
        netdev@...r.kernel.org, lvs-devel@...r.kernel.org,
        Yunhong Jiang <yunhjiang@...y.com>
Subject: Re: Long delay on estimation_timer causes packet latency

On Fri, Dec 04, 2020 at 07:42:56AM +0200, Julian Anastasov wrote:
>
>	Hello,
>
>On Fri, 4 Dec 2020, dust.li wrote:
>
>> 
>> On 12/3/20 4:48 PM, Julian Anastasov wrote:
>> >
>> > - work will use spin_lock_bh(&s->lock) to protect the
>> > entries, we do not want delays between /proc readers and
>> > the work if using mutex. But _bh locks stop timers and
>> > networking for short time :( Not sure yet if just spin_lock
>> > is safe for both /proc and estimator's work.
>
>	Here stopping BH is may be not so fatal if some
>CPUs are used for networking and others for workqueues.
>
>> Thanks for sharing your thoughts !
>> 
>> 
>> I think it's a good idea to split the est_list into different
>> 
>> slots, I believe it will dramatically reduce the delay brought
>> 
>> by estimation.
>
>	268ms/64 => 4ms average. As the estimation with single
>work does not utilize many CPUs simultaneously, this can be a
>problem for 300000-400000 services but this looks crazy.
Yes. Consider the largest server we use now, which has 256 HT
servers with 4 NUMA nodes. Even that should not be a big problem.

>
>> My only concern is the cost of the estimation when the number of
>> 
>> services is large. Splitting the est_list won't reduce the real
>> 
>> work to do.
>> 
>> In our case, each estimation cost at most 268ms/2000ms, which is
>> 
>> about 13% of one CPU hyper-thread, and this should be a common case
>> 
>> in a large K8S cluster with lots of services.
>> 
>> Since the estimation is not needed in our environment at all, it's
>> 
>> just a waste of CPU resource. Have you ever consider add a switch to
>> 
>> let the user turn the estimator off ?
>
>	No problem to add sysctl var for this, we usually add function
>to check which can be used in ip_vs_in_stats, ip_vs_out_stats,
>ip_vs_conn_stats. If switch can be changed at any time, what should
>we do? Add/Del est entries as normally but do not start the
>delayed work if flag disables stats. When flag is enabled counters
>will increase and we will start delayed work.

Yes, this would be perfect for me !