netdev - Re: Stacks leading into skb:kfree

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date: Tue, 18 Jul 2023 16:43:04 -0600
From: David Ahern <dsahern@...nel.org>
To: Ivan Babrou <ivan@...udflare.com>
Cc: Linux Kernel Network Developers <netdev@...r.kernel.org>,
 kernel-team <kernel-team@...udflare.com>, Eric Dumazet
 <edumazet@...gle.com>, "David S. Miller" <davem@...emloft.net>,
 Paolo Abeni <pabeni@...hat.com>, Steven Rostedt <rostedt@...dmis.org>,
 Masami Hiramatsu <mhiramat@...nel.org>, Jakub Kicinski <kuba@...nel.org>
Subject: Re: Stacks leading into skb:kfree_skb

On 7/18/23 4:33 PM, Ivan Babrou wrote:
> On Fri, Jul 14, 2023 at 5:54 PM David Ahern <dsahern@...nel.org> wrote:
>>
>> On 7/14/23 4:13 PM, Ivan Babrou wrote:
>>> As requested by Jakub Kicinski and David Ahern here:
>>>
>>> * https://lore.kernel.org/netdev/20230713201427.2c50fc7b@kernel.org/
>>>
>>> I made some aggregations for the stacks we see leading into
>>> skb:kfree_skb endpoint. There's a lot of data that is not easily
>>> digestible, so I lightly massaged the data and added flamegraphs in
>>> addition to raw stack counts. Here's the gist link:
>>>
>>> * https://gist.github.com/bobrik/0e57671c732d9b13ac49fed85a2b2290
>>
>> I see a lot of packet_rcv as the tip before kfree_skb. How many packet
>> sockets do you have running on that box? Can you accumulate the total
>> packet_rcv -> kfree_skb_reasons into 1 count -- regardless of remaining
>> stacktrace?
> 
> Yan will respond regarding the packet sockets later in the day, he
> knows this stuff better than I do.
> 
> In the meantime, here are the aggregations you requested:
> 
> * Normal: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.normal.packet_rcv.aggregated.svg
> * Spike: https://gist.githubusercontent.com/bobrik/0e57671c732d9b13ac49fed85a2b2290/raw/ae8aa1bc3b22fad6cf541afeb51aa8049d122d02/flamegraph.spike.packet_rcv.aggregated.svg

For the spike, 97% are drops in packet_rcv. Each raw packet socket
causes every packet to be cloned which makes an N-factor on the number
of skbs to be freed. If this is tcpdump or lldp with a filter that would
be what Jakub mentioned in his response.

> 
> I just realized that Github links make flamegraphs non-interactive. If
> you download them and open a local copy, they should work better:

Firefox shows the graphs just fine.

> 
> * Expand to your screen width
> * Working search with highlights
> * Tooltips with counts and percentages
> * Working zoom