netdev - Re: [PATCH net-next 0/5] virtio-net: support dynamic coalescing moderation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <d3b9e9e8-1ef4-48ac-8a2f-4fa647ae4372@linux.alibaba.com>
Date: Wed, 1 Nov 2023 19:03:41 +0800
From: Heng Qi <hengqi@...ux.alibaba.com>
To: "Michael S. Tsirkin" <mst@...hat.com>, Jason Wang <jasowang@...hat.com>
Cc: netdev@...r.kernel.org, virtualization@...ts.linux-foundation.org,
 Xuan Zhuo <xuanzhuo@...ux.alibaba.com>, Eric Dumazet <edumazet@...gle.com>,
 "David S. Miller" <davem@...emloft.net>, Paolo Abeni <pabeni@...hat.com>,
 Jesper Dangaard Brouer <hawk@...nel.org>,
 John Fastabend <john.fastabend@...il.com>,
 Alexei Starovoitov <ast@...nel.org>, Jakub Kicinski <kuba@...nel.org>,
 Simon Horman <horms@...nel.org>, "Liu, Yujie" <yujie.liu@...el.com>
Subject: Re: [PATCH net-next 0/5] virtio-net: support dynamic coalescing
 moderation



在 2023/10/25 下午1:53, Michael S. Tsirkin 写道:
> On Wed, Oct 25, 2023 at 09:18:27AM +0800, Jason Wang wrote:
>> On Tue, Oct 24, 2023 at 8:03 PM Heng Qi <hengqi@...ux.alibaba.com> wrote:
>>>
>>>
>>> 在 2023/10/12 下午4:29, Jason Wang 写道:
>>>> On Thu, Oct 12, 2023 at 3:44 PM Heng Qi <hengqi@...ux.alibaba.com> wrote:
>>>>> Now, virtio-net already supports per-queue moderation parameter
>>>>> setting. Based on this, we use the netdim library of linux to support
>>>>> dynamic coalescing moderation for virtio-net.
>>>>>
>>>>> Due to hardware scheduling issues, we only tested rx dim.
>>>> Do you have PPS numbers? And TX numbers are also important as the
>>>> throughput could be misleading due to various reasons.
>>> Hi Jason!
>>>
>>> The comparison of rx netdim performance is as follows:
>>> (the backend supporting tx dim is not yet ready)
>> Thanks a lot for the numbers.
>>
>> I'd still expect the TX result as I did play tx interrupt coalescing
>> about 10 years ago.
>>
>> I will start to review the series but let's try to have some TX numbers as well.
>>
>> Btw, it would be more convenient to have a raw PPS benchmark. E.g you
>> can try to use a software or hardware packet generator.
>>
>> Thanks
> Latency results are also kind of interesting.

I test the latency using sockperf pp:

@Rx cmd
taskset -c 0 sockperf sr -p 8989

@Tx cmd
taskset -c 0 sockperf pp -i ${ip} -p 8989 -t 10

After running this cmd 5 times and averaging the results,
we get the following data:

dim off: 17.7735 usec
dim on: 18.0110 usec

Thanks!

>
>
>>>
>>> I. Sockperf UDP
>>> =================================================
>>> 1. Env
>>> rxq_0 is affinity to cpu_0
>>>
>>> 2. Cmd
>>> client:  taskset -c 0 sockperf tp -p 8989 -i $IP -t 10 -m 16B
>>> server: taskset -c 0 sockperf sr -p 8989
>>>
>>> 3. Result
>>> dim off: 1143277.00 rxpps, throughput 17.844 MBps, cpu is 100%.
>>> dim on: 1124161.00 rxpps, throughput 17.610 MBps, cpu is 83.5%.
>>> =================================================
>>>
>>>
>>> II. Redis
>>> =================================================
>>> 1. Env
>>> There are 8 rxqs and rxq_i is affinity to cpu_i.
>>>
>>> 2. Result
>>> When all cpus are 100%, ops/sec of memtier_benchmark client is
>>> dim off:   978437.23
>>> dim on: 1143638.28
>>> =================================================
>>>
>>>
>>> III. Nginx
>>> =================================================
>>> 1. Env
>>> There are 8 rxqs and rxq_i is affinity to cpu_i.
>>>
>>> 2. Result
>>> When all cpus are 100%, requests/sec of wrk client is
>>> dim off:   877931.67
>>> dim on: 1019160.31
>>> =================================================
>>>
>>> Thanks!
>>>
>>>> Thanks
>>>>
>>>>> @Test env
>>>>> rxq0 has affinity to cpu0.
>>>>>
>>>>> @Test cmd
>>>>> client: taskset -c 0 sockperf tp -i ${IP} -t 30 --tcp -m ${msg_size}
>>>>> server: taskset -c 0 sockperf sr --tcp
>>>>>
>>>>> @Test res
>>>>> The second column is the ratio of the result returned by client
>>>>> when rx dim is enabled to the result returned by client when
>>>>> rx dim is disabled.
>>>>>           --------------------------------------
>>>>>           | msg_size |  rx_dim=on / rx_dim=off |
>>>>>           --------------------------------------
>>>>>           |   14B    |         + 3%            |
>>>>>           --------------------------------------
>>>>>           |   100B   |         + 16%           |
>>>>>           --------------------------------------
>>>>>           |   500B   |         + 25%           |
>>>>>           --------------------------------------
>>>>>           |   1400B  |         + 28%           |
>>>>>           --------------------------------------
>>>>>           |   2048B  |         + 22%           |
>>>>>           --------------------------------------
>>>>>           |   4096B  |         + 5%            |
>>>>>           --------------------------------------
>>>>>
>>>>> ---
>>>>> This patch set was part of the previous netdim patch set[1].
>>>>> [1] was split into a merged bugfix set[2] and the current set.
>>>>> The previous relevant commentators have been Cced.
>>>>>
>>>>> [1] https://lore.kernel.org/all/20230811065512.22190-1-hengqi@linux.alibaba.com/
>>>>> [2] https://lore.kernel.org/all/cover.1696745452.git.hengqi@linux.alibaba.com/
>>>>>
>>>>> Heng Qi (5):
>>>>>     virtio-net: returns whether napi is complete
>>>>>     virtio-net: separate rx/tx coalescing moderation cmds
>>>>>     virtio-net: extract virtqueue coalescig cmd for reuse
>>>>>     virtio-net: support rx netdim
>>>>>     virtio-net: support tx netdim
>>>>>
>>>>>    drivers/net/virtio_net.c | 394 ++++++++++++++++++++++++++++++++-------
>>>>>    1 file changed, 322 insertions(+), 72 deletions(-)
>>>>>
>>>>> --
>>>>> 2.19.1.6.gb485710b
>>>>>
>>>>>
>>>