netdev - Re: High Cpu load when run smartdns : __ipv6_dev_get

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date: Mon, 28 Aug 2023 08:06:15 +0300
From: Martin Zaharinov <micron10@...il.com>
To: David Ahern <dsahern@...nel.org>
Cc: netdev <netdev@...r.kernel.org>,
 Eric Dumazet <edumazet@...gle.com>,
 pymumu@...il.com
Subject: Re: High Cpu load when run smartdns : __ipv6_dev_get_saddr

Hi David


> On 28 Aug 2023, at 5:42, David Ahern <dsahern@...nel.org> wrote:
> 
> On 8/27/23 1:17 PM, Martin Zaharinov wrote:
>> Hi David,
>> 
>> 
>> 
>>> On 27 Aug 2023, at 19:51, David Ahern <dsahern@...nel.org> wrote:
>>> 
>>> On 8/27/23 7:20 AM, Martin Zaharinov wrote:
>>>> Hi Eric 
>>>> 
>>>> 
>>>> i need you help to find is this bug or no.
>>>> 
>>>> I talk with smartdns team and try to research in his code but for the moment not found ..
>>>> 
>>>> test system have 5k ppp users on pppoe device
>>>> 
>>>> after run smartdns  
>>>> 
>>>> service got to 100% load 
>>>> 
>>>> in normal case when run other 2 type of dns server (isc bind or knot ) all is fine .
>>>> 
>>>> but when run smartdns  see perf : 
>>>> 
>>>> 
>>>> PerfTop:    4223 irqs/sec  kernel:96.9%  exact: 100.0% lost: 0/0 drop: 0/0 [4000Hz cycles],  (target_pid: 1208268)
>>>> ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>>> 
>>>>   28.48%  [kernel]        [k] __ipv6_dev_get_saddr
>>>>   12.31%  [kernel]        [k] l3mdev_master_ifindex_rcu
>>>>    6.63%  [pppoe]         [k] pppoe_rcv
>>>>    3.82%  [kernel]        [k] ipv6_dev_get_saddr
>>>>    2.07%  [kernel]        [k] __dev_queue_xmit
>>> 
>>> Can you post stack traces for the top 5 symbols?
>> 
>> If write how i will get.
> 
> While running traffic load:
>    perf record -a -g -- sleep 5
>    perf report --stdio
> 


Here is perf.data file : https://easyupload.io/k3ep8l



>> 
>>> 
>>> What is the packet rate when the above is taken?
>> 
>> its normal rate of dns query… with both other dns server all is fine 
> 
> That means nothing to me. You will need to post packet rates.

I honestly don't know how to measure it, but I don't think they are more than 10k QPS - in system have 5-5.5k users 

> 
>> 
>>> 
>>> 4,223 irqs/sec is not much of a load; can you add some details on the
>>> hardware and networking setup (e.g., l3mdev reference suggests you are
>>> using VRF)?
>> No system is very simple:
>> 
>> eth0 (Internet) Router (smartDNS + pppoe server ) - eth1 ( User side with pppoe server ) here have 5000 ppp interface .
>> 
>> with both other service i dont see all work fine.
> 
> ip link sh type vrf
> --> that does not show any devices? It should because the majority of
> work done in l3mdev_master_ifindex_rcu is for vrf port devices. ie., it
> should not appear in the perf-top data you posted unless vrf devices are
> in play.

VRF is disable in kernel config .