Message-ID: <130d8c90-6285-41b0-926e-d6df9791bcd4@linux.dev>
Date: Mon, 5 Jan 2026 14:23:27 +0800
From: Leon Hwang <leon.hwang@...ux.dev>
To: Yunsheng Lin <linyunsheng@...wei.com>,
 Jesper Dangaard Brouer <hawk@...nel.org>, netdev@...r.kernel.org
Cc: Ilias Apalodimas <ilias.apalodimas@...aro.org>,
 Steven Rostedt <rostedt@...dmis.org>, Masami Hiramatsu
 <mhiramat@...nel.org>, Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
 "David S . Miller" <davem@...emloft.net>, Eric Dumazet
 <edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>,
 Paolo Abeni <pabeni@...hat.com>, Simon Horman <horms@...nel.org>,
 kerneljasonxing@...il.com, lance.yang@...ux.dev, jiayuan.chen@...ux.dev,
 linux-kernel@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 Leon Huang Fu <leon.huangfu@...pee.com>, Dragos Tatulea
 <dtatulea@...dia.com>, kernel-team <kernel-team@...udflare.com>,
 Yan Zhai <yan@...udflare.com>
Subject: Re: [PATCH net-next v3] page_pool: Add page_pool_release_stalled
 tracepoint



On 4/1/26 10:18, Yunsheng Lin wrote:
> On 2026/1/2 19:43, Jesper Dangaard Brouer wrote:
>>
>>
>> On 02/01/2026 08.17, Leon Hwang wrote:
>>> Introduce a new tracepoint to track stalled page pool releases,
>>> providing better observability for page pool lifecycle issues.
>>>
>>
>> In general I like/support adding this tracepoint for "debugability" of
>> page pool lifecycle issues.
>>
>> For "observability" @Kuba added a netlink scheme[1][2] for page_pool[3], which gives us the ability to get events and list page_pools from userspace.
>> I've not used this myself (yet) so I need input from others if this is something that others have been using for page pool lifecycle issues?
>>
>> Need input from @Kuba/others as the "page-pool-get"[4] state that "Only Page Pools associated with a net_device can be listed".  Don't we want the ability to list "invisible" page_pool's to allow debugging issues?
>>
>>  [1] https://docs.kernel.org/userspace-api/netlink/intro-specs.html
>>  [2] https://docs.kernel.org/userspace-api/netlink/index.html
>>  [3] https://docs.kernel.org/netlink/specs/netdev.html
>>  [4] https://docs.kernel.org/netlink/specs/netdev.html#page-pool-get
>>
>> Looking at the code, I see that NETDEV_CMD_PAGE_POOL_CHANGE_NTF netlink
>> notification is only generated once (in page_pool_destroy) and not when
>> we retry in page_pool_release_retry (like this patch).  In that sense,
>> this patch/tracepoint is catching something more than netlink provides.
>> First I though we could add a netlink notification, but I can imagine
>> cases this could generate too many netlink messages e.g. a netdev with
>> 128 RX queues generating these every second for every RX queue.
>>
>> Guess, I've talked myself into liking this change, what do other
>> maintainers think?  (e.g. netlink scheme and debugging balance)
>>
>>
>>> Problem:
>>> Currently, when a page pool shutdown is stalled due to inflight pages,
>>> the kernel only logs a warning message via pr_warn(). This has several
>>> limitations:
>>>
>>> 1. The warning floods the kernel log after the initial DEFER_WARN_INTERVAL,
>>>     making it difficult to track the progression of stalled releases
>>> 2. There's no structured way to monitor or analyze these events
>>> 3. Debugging tools cannot easily capture and correlate stalled pool
>>>     events with other network activity
>>>
>>> Solution:
>>> Add a new tracepoint, page_pool_release_stalled, that fires when a page
>>> pool shutdown is stalled. The tracepoint captures:
>>> - pool: pointer to the stalled page_pool
>>> - inflight: number of pages still in flight
>>> - sec: seconds since the release was deferred
>>>
>>> The implementation also modifies the logging behavior:
>>> - pr_warn() is only emitted during the first warning interval
>>>    (DEFER_WARN_INTERVAL to DEFER_WARN_INTERVAL*2)
>>> - The tracepoint is fired always, reducing log noise while still
>>>    allowing monitoring tools to track the issue
> 
> If the initial log is still present, I don't really see what's the benefit
> of re-triggering logs or tracepoints when the first two fields are unchanged
> and the last two fields can be inspected using some tool? If there are none,
> perhaps we only need to print the first trigger log and a log upon completion
> of page_pool destruction.
> 

Even though it is possible to inspect the last two fields via the
workqueue (e.g., by tracing page_pool_release_retry with BPF tools),
this is not a practical approach for routine monitoring or debugging.

With the proposed tracepoint, obtaining these fields becomes
straightforward and lightweight, making it much easier to observe and
reason about stalled page pool releases in real systems.

In the issue I encountered, it was crucial to notice that the inflight
count was gradually decreasing over time. This gave me confidence that
some orphaned pages were eventually being returned to the page pool.
Based on that signal, I was then able to capture the call stack of
page_pool_put_defragged_page (kernel v6.6) to identify the code path
responsible for returning those pages.

Without repeated pr_warn logs or tracepoint events, it would have been
significantly harder to observe this progression and correlate it with
the eventual page returns.

Thanks,
Leon

Powered by blists - more mailing lists