Date: Wed, 31 Jan 2024 10:32:19 -0700
From: Jens Axboe <axboe@...nel.dk>
To: Olivier Langlois <olivier@...llion01.com>,
 Stefan Roesch <shr@...kernel.io>, io-uring@...r.kernel.org,
 kernel-team@...com
Cc: ammarfaizi2@...weeb.org, netdev@...r.kernel.org, kuba@...nel.org
Subject: Re: [PATCH v15 0/7] io_uring: add napi busy polling support

On 1/31/24 10:22 AM, Olivier Langlois wrote:
> On Tue, 2024-01-30 at 15:59 -0700, Jens Axboe wrote:
>> On 1/30/24 2:20 PM, Olivier Langlois wrote:
>>> Hi,
>>>
>>> I was wondering what did happen to this patch submission...
>>>
>>> It seems like Stefan did put a lot of effort in addressing every
>>> reported issue for several weeks/months...
>>>
>>> and then nothing... as if this patch has never been reviewed by
>>> anyone...
>>>
>>> has it been decided to not integrate NAPI busy looping in io_uring
>>> privately finally?
>>
>> It's really just waiting for testing, I want to ensure it's working
>> as
>> we want it to before committing. But the production bits I wanted to
>> test on have been dragging out, hence I have not made any moves
>> towards
>> merging this for upstream just yet.
>>
>> FWIW, I have been maintaining the patchset, you can find the current
>> series here:
>>
>> https://git.kernel.dk/cgit/linux/log/?h=io_uring-napi
>>
> 
> test setup:
> -----------
> - kernel 6.7.2 with Jens patchset applied (It did almost work as-is
> except for modifs in io_uring/register.c that was in
> io_uring/io_uring.c in 6.7.2)
> - liburing 2.5 patched with Stefan patch after having carefully make
> sure that IORING_REGISTER_NAPI,IORING_UNREGISTER_NAPI values match the
> ones found in the kernel. (It was originally 26,27 and it is now 27,28)
> - 3 threads each having their own private io_uring ring.
> 
> thread 1:
> - use SQ_POLL kernel thread
> - reads data stream from 15-20 TCP connections
> - enable NAPI busy polling by calling io_uring_register_napi()
> 
> [2024-01-31 08:59:55] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 3(fd 43), napi_id:31
> [2024-01-31 08:59:55] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 8(fd 38), napi_id:30
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 10(fd 36), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 14(fd 32), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 12(fd 34), napi_id:28
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 2(fd 44), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 16(fd 30), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 9(fd 37), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 20(fd 26), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 1(fd 45), napi_id:30
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 6(fd 40), napi_id:28
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 13(fd 33), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 22(fd 22), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 7(fd 39), napi_id:30
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 18(fd 28), napi_id:28
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 19(fd 27), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1028
> LWS_CALLBACK_CLIENT_ESTABLISHED client 23(fd 21), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 4(fd 42), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 5(fd 41), napi_id:25
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 21(fd 24), napi_id:31
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 17(fd 29), napi_id:30
> [2024-01-31 08:59:56] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 15(fd 31), napi_id:28
> [2024-01-31 08:59:57] INFO WSBASE/client_established 1010
> LWS_CALLBACK_CLIENT_ESTABLISHED client 11(fd 35), napi_id:30
> [2024-01-31 09:00:14] INFO WSBASE/client_established 1031
> LWS_CALLBACK_CLIENT_ESTABLISHED client 24(fd 25), napi_id:30
> 
> thread 2:
> - No SQ_POLL
> - reads data stream from 1 TCP socket
> - enable NAPI busy polling by calling io_uring_register_napi()
> 
> [2024-01-31 09:01:45] INFO WSBASE/client_established 1031
> LWS_CALLBACK_CLIENT_ESTABLISHED client 25(fd 23), napi_id:31
> 
> thread 3:
> - No SQ_POLL
> - No NAPI busy polling
> - read data stream from 1 TCP socket
> 
> Outcome:
> --------
> 
> I did not measure latency to make sure that NAPI polling was effective
> but I did ensure the stability of running the patchset by letting the
> program run for 5+ hours non stop without experiencing any glitches

Thanks for testing!

Any chance that you could run some tests with and without NAPI that help
validate that it actually works? That part is what I'm most interested
in, not too worried about the stability of it as I have scrutinized it
pretty close already.

-- 
Jens Axboe

Powered by blists - more mailing lists