linux-kernel - Re: [External] : Re: [bug-report] 5-9% FIO randomwrite ext4 perf regression on 6.12.y kernel

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite for Android: free password hash cracker in your pocket
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <14F95ECE-2AB7-4A50-AE45-C98EC426312C@oracle.com>
Date: Fri, 22 Nov 2024 18:26:46 +0000
From: Saeed Mirzamohammadi <saeed.mirzamohammadi@...cle.com>
To: Paul Webb <paul.x.webb@...cle.com>
CC: Christoph Hellwig <hch@....de>, Jens Axboe <axboe@...nel.dk>,
        Phil Auld
	<pauld@...hat.com>,
        Chaitanya Kulkarni <chaitanyak@...dia.com>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
        "linux-nvme@...ts.infradead.org" <linux-nvme@...ts.infradead.org>,
        Ramanan
 Govindarajan <ramanan.govindarajan@...cle.com>,
        Sagi Grimberg
	<sagi@...mberg.me>, Keith Busch <kbusch@...nel.org>,
        Nicky Veitch
	<nicky.veitch@...cle.com>
Subject: Re: [External] : Re: [bug-report] 5-9% FIO randomwrite ext4 perf
 regression on 6.12.y kernel

FYI, Tried disabling write zeros but still getting the same errors:
[ 326.097275] operation not supported error, dev nvme2n1, sector 10624 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
[ 338.496217] nvme0n1: Dataset Management(0x9) @ LBA 10928, 256 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
… 

diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index d3bde17c818d5..ad2ce6008062e 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -3425,7 +3425,8 @@ static const struct pci_device_id nvme_id_table[] = {
                .driver_data = NVME_QUIRK_STRIPE_SIZE |
                                NVME_QUIRK_DEALLOCATE_ZEROES |
                                NVME_QUIRK_IGNORE_DEV_SUBNQN |
-                               NVME_QUIRK_BOGUS_NID, },
+                               NVME_QUIRK_BOGUS_NID |
+                               NVME_QUIRK_DISABLE_WRITE_ZEROES, },
        { PCI_VDEVICE(INTEL, 0x0a55),   /* Dell Express Flash P4600 */
                .driver_data = NVME_QUIRK_STRIPE_SIZE |
                                NVME_QUIRK_DEALLOCATE_ZEROES, },


This is for:
        { PCI_VDEVICE(INTEL, 0x0a54),   /* Intel P4500/P4600 */

And as we can see from lspci:
$ lspci -nn | grep -i nvme 19:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
20:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
94:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]
9b:00.0 Non-Volatile memory controller [0108]: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] [8086:0a54]


> On Nov 22, 2024, at 9:18 AM, Paul Webb <paul.x.webb@...cle.com> wrote:
> 
> 
> On 22/11/2024 12:13, Christoph Hellwig wrote:
>> On Thu, Nov 21, 2024 at 09:07:32PM +0000, Paul Webb wrote:
>>> Christoph:
>>> To check for weird lazy init code using write zeroes
>>> 
>>> Values in the 5.15 kernel baseline prior to the commit:
>>> $ cat /sys/block/nvme*n1/queue/write_zeroes_max_bytes
>>> 0
>>> 0
>>> 0
>>> 0
>>> 
>>> Values in the 6.11 kernel that contains the commit:
>>> $ cat /sys/block/nvme*n1/queue/write_zeroes_max_bytes
>>> 2199023255040
>>> 2199023255040
>>> 2199023255040
>>> 2199023255040
>> Thanks!  So 6.11 actually enables write zeroes for your controller.
>> 
>>> Another interesting datapoint is that while performing some runs I am
>>> seeing the following output on the console in the 6.11/6.12 kernels that
>>> contain the commit:
>>> 
>>> [  473.398188] operation not supported error, dev nvme2n1, sector 13952 op 0x9:(WRITE_ZEROES) flags 0x800 phys_seg 0 prio class 0
>> .. which it doesn't handle well.
>> 
>>> [  473.534550] nvme0n1: Dataset Management(0x9) @ LBA 14000, 256 blocks, Invalid Command Opcode (sct 0x0 / sc 0x1) DNR
>> .. and interesting this is for a Deallocate, which should only happen
>> with the quirk for certain Intel controllers from the very first days of
>> nvme.
>> 
>> What controller do you have?  Can you post the output of lspci and
>> "nvme list"?
> 
> Hi Christoph,
> 
> The nvme related output from lspci is as follows:
> $ lspci | grep -i nvme
> 19:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 20:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 94:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 9b:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
> 
> 
> $ sudo nvme list
> Node                  Generic               SN Model                                    Namespace Usage                      Format           FW Rev
> --------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
> /dev/nvme0n1          /dev/ng0n1            PHLN942100EQ6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme1n1          /dev/ng1n1            PHLN942100PE6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme2n1          /dev/ng2n1            PHLN9415002B6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
> /dev/nvme3n1          /dev/ng3n1            PHLN942100DQ6P4CGN 7361456_ICRPC2DD2ORA6.4T                 0x1          6.40  TB / 6.40  TB    512   B +  0 B   VDV1RL06
>