netdev - Re: [PATCH net-next v1] page_pool: avoid infinite loop to schedule delayed worker

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <CAL+tcoBa9uz7i-9_-wtakpQkmeiX55RpQn2zkpDnXFBAXutkYw@mail.gmail.com>
Date: Thu, 13 Feb 2025 08:43:26 +0800
From: Jason Xing <kerneljasonxing@...il.com>
To: Mina Almasry <almasrymina@...gle.com>
Cc: davem@...emloft.net, edumazet@...gle.com, kuba@...nel.org, 
	pabeni@...hat.com, hawk@...nel.org, ilias.apalodimas@...aro.org, 
	horms@...nel.org, netdev@...r.kernel.org
Subject: Re: [PATCH net-next v1] page_pool: avoid infinite loop to schedule
 delayed worker

On Thu, Feb 13, 2025 at 8:38 AM Mina Almasry <almasrymina@...gle.com> wrote:
>
> On Wed, Feb 12, 2025 at 3:39 PM Jason Xing <kerneljasonxing@...il.com> wrote:
> >
> > On Thu, Feb 13, 2025 at 3:24 AM Mina Almasry <almasrymina@...gle.com> wrote:
> > >
> > > On Tue, Feb 11, 2025 at 7:14 PM Jason Xing <kerneljasonxing@...il.com> wrote:
> > > >
> > > > On Wed, Feb 12, 2025 at 10:37 AM Mina Almasry <almasrymina@...gle.com> wrote:
> > > > >
> > > > > On Mon, Feb 10, 2025 at 5:10 AM Jason Xing <kerneljasonxing@...il.com> wrote:
> > > > > >
> > > > > > If the buggy driver causes the inflight less than 0 [1] and warns
> > > > >
> > > > > How does a buggy driver trigger this?
> > > >
> > > > We're still reproducing and investigating. With a certain version of
> > > > driver + XDP installed, we have a very slight chance to see this
> > > > happening.
> > > >
> > > > >
> > > > > > us in page_pool_inflight(), it means we should not expect the
> > > > > > whole page_pool to get back to work normally.
> > > > > >
> > > > > > We noticed the kworker is waken up repeatedly and infinitely[1]
> > > > > > in production. If the page pool detect the error happening,
> > > > > > probably letting it go is a better way and do not flood the
> > > > > > var log messages. This patch mitigates the adverse effect.
> > > > > >
> > > > > > [1]
> > > > > > [Mon Feb 10 20:36:11 2025] ------------[ cut here ]------------
> > > > > > [Mon Feb 10 20:36:11 2025] Negative(-51446) inflight packet-pages
> > > > > > ...
> > > > > > [Mon Feb 10 20:36:11 2025] Call Trace:
> > > > > > [Mon Feb 10 20:36:11 2025]  page_pool_release_retry+0x23/0x70
> > > > > > [Mon Feb 10 20:36:11 2025]  process_one_work+0x1b1/0x370
> > > > > > [Mon Feb 10 20:36:11 2025]  worker_thread+0x37/0x3a0
> > > > > > [Mon Feb 10 20:36:11 2025]  kthread+0x11a/0x140
> > > > > > [Mon Feb 10 20:36:11 2025]  ? process_one_work+0x370/0x370
> > > > > > [Mon Feb 10 20:36:11 2025]  ? __kthread_cancel_work+0x40/0x40
> > > > > > [Mon Feb 10 20:36:11 2025]  ret_from_fork+0x35/0x40
> > > > > > [Mon Feb 10 20:36:11 2025] ---[ end trace ebffe800f33e7e34 ]---
> > > > > >
> > > > > > Signed-off-by: Jason Xing <kerneljasonxing@...il.com>
> > > > > > ---
> > > > > >  net/core/page_pool.c | 2 +-
> > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/net/core/page_pool.c b/net/core/page_pool.c
> > > > > > index 1c6fec08bc43..8e9f5801aabb 100644
> > > > > > --- a/net/core/page_pool.c
> > > > > > +++ b/net/core/page_pool.c
> > > > > > @@ -1167,7 +1167,7 @@ void page_pool_destroy(struct page_pool *pool)
> > > > > >         page_pool_disable_direct_recycling(pool);
> > > > > >         page_pool_free_frag(pool);
> > > > > >
> > > > > > -       if (!page_pool_release(pool))
> > > > > > +       if (page_pool_release(pool) <= 0)
> > > > > >                 return;
> > > > >
> > > > > Isn't it the condition in page_pool_release_retry() that you want. to
> > > > > modify? That is the one that handles whether the worker keeps spinning
> > > > > no?
> > > >
> > > > Right, do you mean this patch?
> > > > diff --git a/net/core/page_pool.c b/net/core/page_pool.c
> > > > index 8e9f5801aabb..7dde3bd5f275 100644
> > > > --- a/net/core/page_pool.c
> > > > +++ b/net/core/page_pool.c
> > > > @@ -1112,7 +1112,7 @@ static void page_pool_release_retry(struct
> > > > work_struct *wq)
> > > >         int inflight;
> > > >
> > > >         inflight = page_pool_release(pool);
> > > > -       if (!inflight)
> > > > +       if (inflight < 0)
> > > >                 return;
> > > >
> > > > It has the same behaviour as the current patch does. I thought we
> > > > could stop it earlier.
> > > >
> > >
> > > Yes I mean this.
> >
> > I'm going to post the above diff patch in V2. Am I understanding right?
> >
>
> Please also add Jakub's request, i.e. a code comment indicating why we
> return early.

Got it.

>
> Also, now that I look more closely, we want to make sure we get at
> least one warning when inflight goes negative, so, maybe something
> like (untested, may need some iteration):

Good suggestion.

>
> ```
> diff --git a/net/core/page_pool.c b/net/core/page_pool.c
> index 2ea8041aba7e..6d62ea45571b 100644
> --- a/net/core/page_pool.c
> +++ b/net/core/page_pool.c
> @@ -1113,13 +1113,12 @@ static void page_pool_release_retry(struct
> work_struct *wq)
>         int inflight;
>
>         inflight = page_pool_release(pool);
> -       if (!inflight)
> -               return;
>
>         /* Periodic warning for page pools the user can't see */
>         netdev = READ_ONCE(pool->slow.netdev);
>         if (time_after_eq(jiffies, pool->defer_warn) &&
> -           (!netdev || netdev == NET_PTR_POISON)) {
> +           (!netdev || netdev == NET_PTR_POISON) &&
> +           inflight != 0) {
>                 int sec = (s32)((u32)jiffies - (u32)pool->defer_start) / HZ;
>
>                 pr_warn("%s() stalled pool shutdown: id %u, %d
> inflight %d sec devmem=%d\n",
> @@ -1128,7 +1127,15 @@ static void page_pool_release_retry(struct
> work_struct *wq)
>                 pool->defer_warn = jiffies + DEFER_WARN_INTERVAL;
>         }
>
> -       /* Still not ready to be disconnected, retry later */
> +       /* In rare cases, a driver bug may cause inflight to go negative. Don't
> +        * reschedule release if inflight is 0 or negative.
> +        *      - If 0, the page_pool has been destroyed
> +        *      - if negative, we will never recover
> +        * in both cases no reschedule necessary.
> +        */
> +       if (inflight < 1)

Maybe I would change the above to 'inflight <= 0' which looks more
obvious at the first glance? :)

> +               return;
> +
>         schedule_delayed_work(&pool->release_dw, DEFER_TIME);
>  }
> ```

I will test it. Thanks, Mina.

Thanks,
Jason

>
> --
> Thanks,
> Mina