lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  linux-cve-announce  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <dfc33064-f99f-4728-858f-95c80300bcff@huawei.com>
Date: Sun, 4 Jan 2026 10:18:50 +0800
From: Yunsheng Lin <linyunsheng@...wei.com>
To: Jesper Dangaard Brouer <hawk@...nel.org>, Leon Hwang
	<leon.hwang@...ux.dev>, <netdev@...r.kernel.org>
CC: Ilias Apalodimas <ilias.apalodimas@...aro.org>, Steven Rostedt
	<rostedt@...dmis.org>, Masami Hiramatsu <mhiramat@...nel.org>, Mathieu
 Desnoyers <mathieu.desnoyers@...icios.com>, "David S . Miller"
	<davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>, Jakub Kicinski
	<kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>, Simon Horman
	<horms@...nel.org>, <kerneljasonxing@...il.com>, <lance.yang@...ux.dev>,
	<jiayuan.chen@...ux.dev>, <linux-kernel@...r.kernel.org>,
	<linux-trace-kernel@...r.kernel.org>, Leon Huang Fu
	<leon.huangfu@...pee.com>, Dragos Tatulea <dtatulea@...dia.com>, kernel-team
	<kernel-team@...udflare.com>, Yan Zhai <yan@...udflare.com>
Subject: Re: [PATCH net-next v3] page_pool: Add page_pool_release_stalled
 tracepoint

On 2026/1/2 19:43, Jesper Dangaard Brouer wrote:
> 
> 
> On 02/01/2026 08.17, Leon Hwang wrote:
>> Introduce a new tracepoint to track stalled page pool releases,
>> providing better observability for page pool lifecycle issues.
>>
> 
> In general I like/support adding this tracepoint for "debugability" of
> page pool lifecycle issues.
> 
> For "observability" @Kuba added a netlink scheme[1][2] for page_pool[3], which gives us the ability to get events and list page_pools from userspace.
> I've not used this myself (yet) so I need input from others if this is something that others have been using for page pool lifecycle issues?
> 
> Need input from @Kuba/others as the "page-pool-get"[4] state that "Only Page Pools associated with a net_device can be listed".  Don't we want the ability to list "invisible" page_pool's to allow debugging issues?
> 
>  [1] https://docs.kernel.org/userspace-api/netlink/intro-specs.html
>  [2] https://docs.kernel.org/userspace-api/netlink/index.html
>  [3] https://docs.kernel.org/netlink/specs/netdev.html
>  [4] https://docs.kernel.org/netlink/specs/netdev.html#page-pool-get
> 
> Looking at the code, I see that NETDEV_CMD_PAGE_POOL_CHANGE_NTF netlink
> notification is only generated once (in page_pool_destroy) and not when
> we retry in page_pool_release_retry (like this patch).  In that sense,
> this patch/tracepoint is catching something more than netlink provides.
> First I though we could add a netlink notification, but I can imagine
> cases this could generate too many netlink messages e.g. a netdev with
> 128 RX queues generating these every second for every RX queue.
> 
> Guess, I've talked myself into liking this change, what do other
> maintainers think?  (e.g. netlink scheme and debugging balance)
> 
> 
>> Problem:
>> Currently, when a page pool shutdown is stalled due to inflight pages,
>> the kernel only logs a warning message via pr_warn(). This has several
>> limitations:
>>
>> 1. The warning floods the kernel log after the initial DEFER_WARN_INTERVAL,
>>     making it difficult to track the progression of stalled releases
>> 2. There's no structured way to monitor or analyze these events
>> 3. Debugging tools cannot easily capture and correlate stalled pool
>>     events with other network activity
>>
>> Solution:
>> Add a new tracepoint, page_pool_release_stalled, that fires when a page
>> pool shutdown is stalled. The tracepoint captures:
>> - pool: pointer to the stalled page_pool
>> - inflight: number of pages still in flight
>> - sec: seconds since the release was deferred
>>
>> The implementation also modifies the logging behavior:
>> - pr_warn() is only emitted during the first warning interval
>>    (DEFER_WARN_INTERVAL to DEFER_WARN_INTERVAL*2)
>> - The tracepoint is fired always, reducing log noise while still
>>    allowing monitoring tools to track the issue

If the initial log is still present, I don't really see what's the benefit
of re-triggering logs or tracepoints when the first two fields are unchanged
and the last two fields can be inspected using some tool? If there are none,
perhaps we only need to print the first trigger log and a log upon completion
of page_pool destruction.

>>
>> This allows developers and system administrators to:
>> - Use tools like perf, ftrace, or eBPF to monitor stalled releases
>> - Correlate page pool issues with network driver behavior
>> - Analyze patterns without parsing kernel logs
>> - Track the progression of inflight page counts over time
>>
>> Signed-off-by: Leon Huang Fu <leon.huangfu@...pee.com>
>> Signed-off-by: Leon Hwang <leon.hwang@...ux.dev>

Powered by blists - more mailing lists

Powered by Openwall GNU/*/Linux Powered by OpenVZ