linux-kernel - Re: Phoronix pts fio io_uring test regression report on upstream v6.1 and v5.15

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <15BBDF79-8063-40BE-AC19-52FA69C98492@oracle.com>
Date:   Tue, 14 Feb 2023 18:58:45 +0000
From:   Saeed Mirzamohammadi <saeed.mirzamohammadi@...cle.com>
To:     Jens Axboe <axboe@...nel.dk>
CC:     "io-uring@...r.kernel.org" <io-uring@...r.kernel.org>,
        "asml.silence@...il.com" <asml.silence@...il.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: Phoronix pts fio io_uring test regression report on upstream v6.1
 and v5.15

Hi Jens,

> On Jan 26, 2023, at 10:35 AM, Jens Axboe <axboe@...nel.dk> wrote:
> 
> On 1/26/23 11:04 AM, Saeed Mirzamohammadi wrote:
>> Hi Jens,
>> 
>>> On Jan 25, 2023, at 4:28 PM, Jens Axboe <axboe@...nel.dk> wrote:
>>> 
>>> On 1/25/23 5:22?PM, Saeed Mirzamohammadi wrote:
>>>> Hi Jens,
>>>> 
>>>> I applied your patch (with a minor conflict in xfs_file_open() since FMODE_BUF_WASYNC isn't in v5.15) and did the same series of tests on the v5.15 kernel. All the io_uring benchmarks regressed 20-45% after it. I haven't tested on v6.1 yet.
>>> 
>>> It should basically make the behavior the same as before once you apply
>>> the patch, so please pass on the patch that you applied for 5.15 so we
>>> can take a closer look.
>> 
>> Attached the patch.
> 
> I tested the upstream variant, and it does what it's supposed to and
> gets parallel writes on O_DIRECT. Unpatched, any dio write results in:
> 
>             fio-566     [000] .....   131.071108: io_uring_queue_async_work: ring 00000000706cb6c0, request 00000000b21691c4, user_data 0xaaab0e8e4c00, opcode WRITE, flags 0xe0040000, hashed queue, work 000000002c5aeb79
> 
> and after the patch:
> 
>             fio-376     [000] .....    24.590994: io_uring_queue_async_work: ring 000000007bdb650a, request 000000006b5350e0, user_data 0xaaab1b3e3c00, opcode WRITE, flags 0xe0040000, normal queue, work 00000000e3e81955
> 

Thanks for looking into this.

> where the hashed queued is serialized based on the inode, and the normal
> queue is not (eg they run in parallel).
> 
> As mentioned, the fio job being used isn't representative of anything
> that should actually be run, the async flag really only exists for
> experimentation. Do you have a real workload that is seeing a regression?
> If yes, does that real workload change performance with the patch?

I tested without the async flag but didn’t see any change in the performance.

I haven’t tested any real workload yet. I’ll share with you if I noticed anything.

Thanks,
Saeed

p.s. I experienced multipathd issues with the patch that I had to work through. Never without the patch.

> 
> -- 
> Jens Axboe