netdev - Re: Requeues

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <CAA93jw4eCG19v5Pt_2o-W7OBQ=66O126xi-Gb8gObakhQabMTw@mail.gmail.com>
Date:	Fri, 9 Mar 2012 17:54:48 +0000
From:	Dave Taht <dave.taht@...il.com>
To:	"John A. Sullivan III" <jsullivan@...nsourcedevel.com>
Cc:	John Fastabend <john.r.fastabend@...el.com>, netdev@...r.kernel.org
Subject: Re: Requeues

On Sat, Mar 3, 2012 at 10:13 AM, John A. Sullivan III
<jsullivan@...nsourcedevel.com> wrote:
> On Fri, 2012-03-02 at 22:54 -0800, John Fastabend wrote:
>> On 3/2/2012 9:48 PM, John A. Sullivan III wrote:
>> > Hello, all.  I am seeing a small but significant number of requeues on
>> > our pfifo_fast qdiscs.  I've not been able to find much on what this
>> > means but the little I have implies it may be a problem with the
>> > physical interfaces.  However, these are fairly high end systems with
>> > Intel e1000 quad port cards.


>From what you describe it sounds like this is not a 'problem', but a
sign that your environment is capable of saturating e1000 quad port
cards.

Adding a little active queue management (red or sfqred) will reduce
the impact of the extra buffering and lead to more responsive streams
with more accurate estimates of the BDP.

if you are already running bql, you will generally see more requeues.

>> >
>> > We thought it might have something to do with the bonded interfaces so
>> > we checked some other high end systems without bonded interfaces but the
>> > same quad port cards and, lo and behold, the same small but significant
>> > number of requeues.
>> >
>> > Is this normal or does it indicate a problem somewhere? Thanks - John
>> >
>> > --
>> > To unsubscribe from this list: send the line "unsubscribe netdev" in
>> > the body of a message to majordomo@...r.kernel.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>> One of two things can happen to cause the requeue counter to increment.
>>
>> When the qdisc dequeues a packet it then gets enqueued in yet another
>> queue in the e1000 driver. If the qdisc is dequeueing packets faster
>> than the hardware can consume them the driver will return a
>> NETDEV_TX_BUSY error. This causes the qdisc to 'requeue' the packet and
>> in the process increments this counter.
>>
>> The gist is pfifo_fast dequeued a packet tried to give it to e1000 which
>> pushed back so pfifo_fast put the packet back on the queue.
>>
>> By the way you can tune the size of the e1000 queues manually by playing
>> with:
>>
>>       /sys/class/net/ethx/tx_queue_len
>>
>> But you likely don't want to make them any larger else you'll hit the
>> buffer bloat problem. I'm guessing we should add byte queue limit support
>> here.
>>
>> The second way to trigger this is multiple cpus contending for a lock.
>>
>> In short requeue counts as long as its not excessive are just part
>> of normal operation. So shouldn't be a problem.
> <snip>
> Thank you very much.  I was not aware of CPU contention being a cause.
> We did notice the problem primarily on systems with 8 to 16 cores - John
>
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht
SKYPE: davetaht
US Tel: 1-239-829-5608
http://www.bufferbloat.net
--
To unsubscribe from this list: send the line "unsubscribe netdev" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html