linux-kernel - Re: [PATCH] ena: Speed up initialization 90x by reducing poll delays

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Tue, 3 Mar 2020 00:53:54 +0000
From:   "Machulsky, Zorik" <zorik@...zon.com>
To:     Josh Triplett <josh@...htriplett.org>
CC:     "Belgazal, Netanel" <netanel@...zon.com>,
        "Kiyanovski, Arthur" <akiyano@...zon.com>,
        "Tzalik, Guy" <gtzalik@...zon.com>,
        "Bshara, Saeed" <saeedb@...zon.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] ena: Speed up initialization 90x by reducing poll delays



On 3/2/20, 4:40 PM, "Josh Triplett" <josh@...htriplett.org> wrote:
    
    
    On Mon, Mar 02, 2020 at 11:16:32PM +0000, Machulsky, Zorik wrote:
    >
    > On 2/28/20, 4:29 PM, "Josh Triplett" <josh@...htriplett.org> wrote:
    >
    >     Before initializing completion queue interrupts, the ena driver uses
    >     polling to wait for responses on the admin command queue. The ena driver
    >     waits 5ms between polls, but the hardware has generally finished long
    >     before that. Reduce the poll time to 10us.
    >
    >     On a c5.12xlarge, this improves ena initialization time from 173.6ms to
    >     1.920ms, an improvement of more than 90x. This improves server boot time
    >     and time to network bringup.
    >
    > Thanks Josh,
    > We agree that polling rate should be increased, but prefer not to do it aggressively and blindly.
    > For example linear backoff approach might be a better choice. Please let us re-work a little this
    > patch and bring it to review. Thanks!
    
    That's fine, as long as it has the same net improvement on boot time.
    
    I'd appreciate the opportunity to test any alternate approach you might
    have.
    
    (Also, as long as you're working on this, you might wish to make a
    similar change to the EFA driver, and to the FreeBSD drivers.) 

Absolutely! Already forwarded this to the owners of these drivers.  Thanks!
    
    >     Before:
    >     [    0.531722] calling  ena_init+0x0/0x63 @ 1
    >     [    0.531722] ena: Elastic Network Adapter (ENA) v2.1.0K
    >     [    0.531751] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K
    >     [    0.531946] PCI Interrupt Link [LNKD] enabled at IRQ 11
    >     [    0.547425] ena: ena device version: 0.10
    >     [    0.547427] ena: ena controller version: 0.0.1 implementation version 1
    >     [    0.709497] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency
    >     [    0.709508] initcall ena_init+0x0/0x63 returned 0 after 173616 usecs
    >
    >     After:
    >     [    0.526965] calling  ena_init+0x0/0x63 @ 1
    >     [    0.526966] ena: Elastic Network Adapter (ENA) v2.1.0K
    >     [    0.527056] ena 0000:00:05.0: Elastic Network Adapter (ENA) v2.1.0K
    >     [    0.527196] PCI Interrupt Link [LNKD] enabled at IRQ 11
    >     [    0.527211] ena: ena device version: 0.10
    >     [    0.527212] ena: ena controller version: 0.0.1 implementation version 1
    >     [    0.528925] ena 0000:00:05.0: Elastic Network Adapter (ENA) found at mem febf4000, mac addr 06:c4:22:0e:dc:da, Placement policy: Low Latency
    >     [    0.528934] initcall ena_init+0x0/0x63 returned 0 after 1920 usecs