netdev - Re: [PATCH net-next RFC 1/2] page_pool: add page recycling support based on elevated refcnt

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <YOQRxS+MUFIRubsf@enceladus>
Date:   Tue, 6 Jul 2021 11:18:13 +0300
From:   Ilias Apalodimas <ilias.apalodimas@...aro.org>
To:     Yunsheng Lin <linyunsheng@...wei.com>
Cc:     Jesper Dangaard Brouer <jbrouer@...hat.com>, davem@...emloft.net,
        kuba@...nel.org, linuxarm@...neuler.org, yisen.zhuang@...wei.com,
        salil.mehta@...wei.com, thomas.petazzoni@...tlin.com,
        mw@...ihalf.com, linux@...linux.org.uk, hawk@...nel.org,
        ast@...nel.org, daniel@...earbox.net, john.fastabend@...il.com,
        akpm@...ux-foundation.org, peterz@...radead.org, will@...nel.org,
        willy@...radead.org, vbabka@...e.cz, fenghua.yu@...el.com,
        guro@...com, peterx@...hat.com, feng.tang@...el.com, jgg@...pe.ca,
        mcroce@...rosoft.com, hughd@...gle.com, jonathan.lemon@...il.com,
        alobakin@...me, willemb@...gle.com, wenxu@...oud.cn,
        cong.wang@...edance.com, haokexin@...il.com, nogikh@...gle.com,
        elver@...gle.com, netdev@...r.kernel.org,
        linux-kernel@...r.kernel.org, bpf@...r.kernel.org,
        Alexander Duyck <alexander.duyck@...il.com>
Subject: Re: [PATCH net-next RFC 1/2] page_pool: add page recycling support
 based on elevated refcnt

> >>

[...]

> >>>
> >>>
> >>>> So add elevated refcnt support in page pool, and support
> >>>> allocating page frag to enable multi-frames-per-page based
> >>>> on the elevated refcnt support.
> >>>>
> >>>> As the elevated refcnt is per page, and there is no space
> >>>> for that in "struct page" now, so add a dynamically allocated
> >>>> "struct page_pool_info" to record page pool ptr and refcnt
> >>>> corrsponding to a page for now. Later, we can recycle the
> >>>> "struct page_pool_info" too, or use part of page memory to
> >>>> record pp_info.
> >>>
> >>> I'm not happy with allocating a memory (slab) object "struct page_pool_info" per page.
> >>>
> >>> This also gives us an extra level of indirection.
> >>
> >> I'm not happy with that either, if there is better way to
> >> avoid that, I will be happy to change it:)
> > 
> > I think what we have to answer here is, do we want and does it make sense
> > for page_pool to do the housekeeping of the buffer splitting or are we
> > better of having each driver do that.  IIRC your previous patch on top of
> > the original recycling patchset was just 'atomic' refcnts on top of page pool.
> 
> You are right that driver was doing the the buffer splitting in previous
> patch.
> 
> The reason why I abandoned that is:
> 1. Currently the meta-data of page in the driver is per desc, which means
>    it might not be able to use first half of a page for a desc, and the
>    second half of the same page for another desc, this ping-pong way of
>    reusing the whole page for only one desc in the driver seems unnecessary
>    and waste a lot of memory when there is already reusing in the page pool.
> 
> 2. Easy use of API for the driver too, which means the driver uses
>    page_pool_dev_alloc_frag() and page_pool_put_full_page() for elevated
>    refcnt case, corresponding to page_pool_dev_alloc_pages() and
>    page_pool_put_full_page() for non-elevated refcnt case, the driver does
>    not need to worry about the meta-data of a page.
> 

Ok that makes sense.  We'll need the complexity anyway and I said I don't
have any strong opinions yet, we might as well make page_pool responsible
for it.
What we need to keep in mind is that page_pool was primarily used for XDP
packets.  We need to make sure we have no performance regressions there.
However I don't have access to > 10gbit NICs with XDP support. Can anyone
apply the patchset and check the performance?

> > 
> >>

[...]

> >> Aside from the performance improvement, there is memory usage
> >> decrease for 64K page size kernel, which means a 64K page can
> >> be used by 32 description with 2k buffer size, and that is a
> >> lot of memory saving for 64 page size kernel comparing to the
> >> current split page reusing implemented in the driver.
> >>
> > 
> > Whether the driver or page_pool itself keeps the meta-data, the outcome
> > here won't change.  We'll still be able to use page frags.
> 
> As above, it is the ping-pong way of reusing when the driver keeps the
> meta-data, and it is page-frag way of reusing when the page pool keeps
> the meta-data.
> 
> I am not sure if the page-frag way of reusing is possible when we still
> keep the meta-data in the driver, which seems very complex at the initial
> thinking.
> 

Fair enough. It's complex in both scenarios so if people think it's useful
I am not against adding it in the API.


Thanks
/Ilias
> > 
> > 
> > Cheers
> > /Ilias
> >>
> >>>
> >>>  __page_frag_cache_refill() + __page_frag_cache_drain() + page_frag_alloc_align()
> >>>
> >>>
> >>
> >> [...]
> > .
> >