netdev - Re: tg3 dropping packets at high packet rates

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CALs4sv2M+9N1joECMQrOGKHQ_YjMqzeF1gPD_OBQ2_r+SJwOwQ@mail.gmail.com>
Date:   Thu, 19 May 2022 20:05:49 +0530
From:   Pavan Chebbi <pavan.chebbi@...adcom.com>
To:     David Laight <David.Laight@...lab.com>
Cc:     Paolo Abeni <pabeni@...hat.com>,
        Michael Chan <michael.chan@...adcom.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "mchan@...adcom.com" <mchan@...adcom.com>,
        David Miller <davem@...emloft.net>
Subject: Re: tg3 dropping packets at high packet rates

On Thu, May 19, 2022 at 7:41 PM David Laight <David.Laight@...lab.com> wrote:
>
> From: Paolo Abeni
> > Sent: 19 May 2022 14:29
> ....
> > If the packet processing is 'bursty', you can have idle time and still
> > hit now and the 'rx ring is [almost] full' condition. If pause frames
> > are enabled, that will cause the peer to stop sending frames: drop can
> > happen in the switch, and the local NIC will not notice (unless there
> > are counters avaialble for pause frames sent).
>
> The test program sending the data does spread it out.
> So it isn't sending 2000 packets with minimal IPG every 10ms.
> (I'm sending from 2 systems.)
>
> I don't know if pause frames are enabled (ethtool might suggest they are).
> But detecting whether they are sent is another matter.
>
> In any case sending pause frames doesn't fix anything.
> They are largely entirely useless unless you have a cable
> that directly connects two computers.
>
> > AFAICS the packet processing is bursty, because enqueuing packets to a
> > remote CPU in considerably faster then full network stack processing.
>
> I have taken restricted ftrace traces of the receiving system.
> Not often seen more than 4 frames processed in one napi callback
> Certainly didn't spot blocks of 100+ that you might expect
> to see if the driver code was the bottleneck.
>
> > Side note: on a not-to-obsolete H/W the kernel should be able to
> > process >1mpps per cpu.
>
> Yes, and, IIRC, a 33Mhz 486 can saturate 10MHz ethernet with
> small packets.
>
> In this case the cpu are almost twiddling their thumbs.
>   model name      : Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz
>   stepping        : 2
>   microcode       : 0x43
>   cpu MHz         : 1300.000
> cpu 14 (the one taking the interrupts) is running at full speed.
>
> cpu doesn't seem to be the bottleneck.
> The problem seems to be the hardware not using all the buffers
> it has been given.
>

When this happens, can you provide the register dump that you can
obtain using ethtool -d ?

>         David
>
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)

Download attachment "smime.p7s" of type "application/pkcs7-signature" (4209 bytes)