netdev - Re: [PATCH net-next] net/rds: Use DMA memory pool allocation for rds

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Date:   Wed, 2 Oct 2019 09:16:33 +0200
From:   Håkon Bugge <haakon.bugge@...cle.com>
To:     David Miller <davem@...emloft.net>
Cc:     netdev@...r.kernel.org,
        Santosh Shilimkar <santosh.shilimkar@...cle.com>,
        rds-devel@....oracle.com,
        Ka-Cheong Poon <ka-cheong.poon@...cle.com>
Subject: Re: [PATCH net-next] net/rds: Use DMA memory pool allocation for
 rds_header



> On 2 Oct 2019, at 07:20, Ka-Cheong Poon <ka-cheong.poon@...cle.com> wrote:
> 
> On 10/2/19 1:16 AM, David Miller wrote:
>> From: Ka-Cheong Poon <ka-cheong.poon@...cle.com>
>> Date: Mon, 30 Sep 2019 02:08:00 -0700
>>> Currently, RDS calls ib_dma_alloc_coherent() to allocate a large piece
>>> of contiguous DMA coherent memory to store struct rds_header for
>>> sending/receiving packets.  The memory allocated is then partitioned
>>> into struct rds_header.  This is not necessary and can be costly at
>>> times when memory is fragmented.  Instead, RDS should use the DMA
>>> memory pool interface to handle this.
>>> 
>>> Suggested-by: Håkon Bugge <haakon.bugge@...cle.com>
>>> Signed-off-by: Ka-Cheong Poon <ka-cheong.poon@...cle.com>
>> This is trading a one-time overhead for extra levels of dereferencing
>> on every single descriptor access in the fast paths.
>> I do not agree with this tradeoff, please implement this more
>> reasonably.
> 
> 
> The problem with the existing way of pre-allocation is
> that when there are a lot of RDS connections, the call to
> ib_dma_alloc_coherent() can fail because there are not
> enough contiguous memory pages available.  It is causing
> problems in production systems.
> 
> And the i_{recv|send|_hdrs_dma array dereferencing is done
> at send/receive ring initialization and refill.  It is not
> done at every access of the header.

The commit removes costly order:4 allocations (with the default 1024 #entries in the recv work-queue). These allocations may need reclaims and/or compaction. At worst, they may fail.

Consider a switch failure, thousands of RDS IB connections, and their corresponding QPs, need to be resurrected. Each QP needs this order:4 allocation, and they are all created in close proximity in time, leading to an immense memory hog.


Thxs, Håkon




> Thanks.
> 
> 
> -- 
> K. Poon
> ka-cheong.poon@...cle.com
> 
> 
>