netdev - Re: [PATCH v4 net-next 2/2] net: introduce budget

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20230316172020.5af40fe8@kernel.org>
Date:   Thu, 16 Mar 2023 17:20:20 -0700
From:   Jakub Kicinski <kuba@...nel.org>
To:     Jason Xing <kerneljasonxing@...il.com>
Cc:     jbrouer@...hat.com, davem@...emloft.net, edumazet@...gle.com,
        pabeni@...hat.com, ast@...nel.org, daniel@...earbox.net,
        hawk@...nel.org, john.fastabend@...il.com,
        stephen@...workplumber.org, simon.horman@...igine.com,
        sinquersw@...il.com, bpf@...r.kernel.org, netdev@...r.kernel.org,
        Jason Xing <kernelxing@...cent.com>
Subject: Re: [PATCH v4 net-next 2/2] net: introduce budget_squeeze to help
 us tune rx behavior

On Wed, 15 Mar 2023 17:20:41 +0800 Jason Xing wrote:
> In our production environment, there're hundreds of machines hitting the
> old time_squeeze limit often from which we cannot tell what exactly causes
> such issues. Hitting limits aranged from 400 to 2000 times per second,
> Especially, when users are running on the guest OS with veth policy
> configured, it is relatively easier to hit the limit. After several tries
> without this patch, I found it is only real time_squeeze not including
> budget_squeeze that hinders the receive process.

That is the common case, and can be understood from the napi trace
point and probing the kernel with bpftrace. We should only add
uAPI for statistics which must be maintained contiguously. For
investigations tracing will always be orders of magnitude more
powerful :(

On the time squeeze BTW, have you found out what the problem was?
In workloads I've seen the time problems are often because of noise 
in how jiffies are accounted (cgroup code disables interrupts
for long periods of time, for example, making jiffies increment 
by 2, 3 or 4 rather than by 1).

> So when we encounter some related performance issue and then get lost on
> how to tune the budget limit and time limit in net_rx_action() function,
> we can separately counting both of them to avoid the confusion.