netdev - Re: [PATCH] qed: avoid spin loops in _qed_mcp_cmd_and

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CADUfDZqx6EjOY=JcQuC6hfPjGgTZCk6BcV5_D1Dp+WQJiXmEnQ@mail.gmail.com>
Date:   Wed, 27 Oct 2021 17:20:04 -0700
From:   Caleb Sander <csander@...estorage.com>
To:     Eric Dumazet <eric.dumazet@...il.com>
Cc:     netdev@...r.kernel.org, Ariel Elior <aelior@...vell.com>,
        GR-everest-linux-l2@...vell.com,
        Joern Engel <joern@...estorage.com>
Subject: Re: [PATCH] qed: avoid spin loops in _qed_mcp_cmd_and_union()

> Here you might sleep/schedule, while CAN_SLEEP was not set ?

I also do not know this driver, just trying to fix an observed latency issue.
As far as I can tell, the CAN_SLEEP flag is set/unset depending on
which function called qed_mcp_cmd_and_union();
it does not indicate whether the function is running in atomic context.
For example, qed_mcp_cmd() calls it without CAN_SLEEP,
yet qed_mcp_drain() calls msleep() immediately after qed_mcp_cmd().

We were concerned that this function might be called in atomic context,
so we added a WARN_ON_ONCE(in_atomic()). We never saw the warning fire
during two weeks of testing, so we believe sleeping is possible here.

> I would suggest using usleep_range() instead, because cond_resched()
> can be a NOP under some circumstances.
> Then perhaps not count against max_retries, but based on total elapsed time ?

I agree these would both be improvements to the current code.
I was trying to provide a minimal change that would allow these loops
to yield the CPU,
but will happily do this refactoring if the driver authors think it
would be beneficial.

On Wed, Oct 27, 2021 at 3:25 PM Eric Dumazet <eric.dumazet@...il.com> wrote:
>
>
>
> On 10/27/21 2:45 PM, Caleb Sander wrote:
> > By default, qed_mcp_cmd_and_union() sets max_retries to 500K and
> > usecs to 10, so these loops can together delay up to 5s.
> > We observed thread scheduling delays of over 700ms in production,
> > with stacktraces pointing to this code as the culprit.
> >
> > Add calls to cond_resched() in both loops to yield the CPU if necessary.
> >
> > Signed-off-by: Caleb Sander <csander@...estorage.com>
> > Reviewed-by: Joern Engel <joern@...estorage.com>
> > ---
> >  drivers/net/ethernet/qlogic/qed/qed_mcp.c | 12 ++++++++----
> >  1 file changed, 8 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/net/ethernet/qlogic/qed/qed_mcp.c b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
> > index 24cd41567..d6944f020 100644
> > --- a/drivers/net/ethernet/qlogic/qed/qed_mcp.c
> > +++ b/drivers/net/ethernet/qlogic/qed/qed_mcp.c
> > @@ -485,10 +485,12 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
> >
> >               spin_unlock_bh(&p_hwfn->mcp_info->cmd_lock);
> >
> > -             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP))
> > +             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP)) {
>
> I do not know this driver, but apparently, there is this CAN_SLEEP test
> hinting about being able to sleep.
>
> >                       msleep(msecs);
> > -             else
> > +             } else {
> > +                     cond_resched();
>
> Here you might sleep/schedule, while CAN_SLEEP was not set ?
>
> >                       udelay(usecs);
>
>
> I would suggest using usleep_range() instead, because cond_resched()
> can be a NOP under some circumstances.
>
> > +             }
> >       } while (++cnt < max_retries);
>
> Then perhaps not count against max_retries, but based on total elapsed time ?
>
> >
> >       if (cnt >= max_retries) {
> > @@ -517,10 +519,12 @@ _qed_mcp_cmd_and_union(struct qed_hwfn *p_hwfn,
> >                * The spinlock stays locked until the list element is removed.
> >                */
> >
> > -             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP))
> > +             if (QED_MB_FLAGS_IS_SET(p_mb_params, CAN_SLEEP)) {
> >                       msleep(msecs);
> > -             else
> > +             } else {
> > +                     cond_resched();
> >                       udelay(usecs);
> > +             }
> >
> >               spin_lock_bh(&p_hwfn->mcp_info->cmd_lock);
> >
> >