Message-ID: <CACGkMEtYE1OY+okxHAj=cVfW-Qz45an28oO=Wv15yOtpD6UqdQ@mail.gmail.com>
Date: Thu, 8 Aug 2024 10:58:07 +0800
From: Jason Wang <jasowang@...hat.com>
To: Yongji Xie <xieyongji@...edance.com>
Cc: "Michael S. Tsirkin" <mst@...hat.com>, Xuan Zhuo <xuanzhuo@...ux.alibaba.com>, 
	Eugenio Perez Martin <eperezma@...hat.com>, Maxime Coquelin <maxime.coquelin@...hat.com>, 
	virtualization@...ts.linux.dev, linux-kernel <linux-kernel@...r.kernel.org>, 
	21cnbao@...il.com, penguin-kernel@...ove.sakura.ne.jp, linux-mm@...ck.org, 
	Andrew Morton <akpm@...ux-foundation.org>
Subject: Re: [PATCH] vduse: avoid using __GFP_NOFAIL

On Wed, Aug 7, 2024 at 2:52 PM Yongji Xie <xieyongji@...edance.com> wrote:
>
> On Mon, Aug 5, 2024 at 4:21 PM Jason Wang <jasowang@...hat.com> wrote:
> >
> > Barry said [1]:
> >
> > """
> > mm doesn't support non-blockable __GFP_NOFAIL allocation. Because
> > __GFP_NOFAIL without direct reclamation may just result in a busy
> > loop within non-sleepable contexts.
> > ""“
> >
> > Unfortuantely, we do that under read lock. A possible way to fix that
> > is to move the pages allocation out of the lock into the caller, but
> > having to allocate a huge number of pages and auxiliary page array
> > seems to be problematic as well per Tetsuon [2]:
> >
> > """
> > You should implement proper error handling instead of using
> > __GFP_NOFAIL if count can become large.
> > """
> >
> > So I choose another way, which does not release kernel bounce pages
> > when user tries to register usersapce bounce pages. Then we don't need
> > to do allocation in the path which is not expected to be fail (e.g in
> > the release). We pay this for more memory usage but further
> > optimizations could be done on top.
> >
> > [1] https://lore.kernel.org/all/CACGkMEtcOJAA96SF9B8m-nZ1X04-XZr+nq8ZQ2saLnUdfOGOLg@mail.gmail.com/T/#m3caef86a66ea6318ef94f9976ddb3a0ccfe6fcf8
> > [2] https://lore.kernel.org/all/CACGkMEtcOJAA96SF9B8m-nZ1X04-XZr+nq8ZQ2saLnUdfOGOLg@mail.gmail.com/T/#m7ad10eaba48ade5abf2d572f24e185d9fb146480
> >
> > Fixes: 6c77ed22880d ("vduse: Support using userspace pages as bounce buffer")
> > Signed-off-by: Jason Wang <jasowang@...hat.com>
> > ---
>
> Reviewed-by: Xie Yongji <xieyongji@...edance.com>
> Tested-by: Xie Yongji <xieyongji@...edance.com>

Thanks.

>
> Have tested it with qemu-storage-daemon [1]:
>
> $ qemu-storage-daemon \
>     --chardev socket,id=charmonitor,path=/tmp/qmp.sock,server=on,wait=off \
>     --monitor chardev=charmonitor \
>     --blockdev driver=host_device,cache.direct=on,aio=native,filename=/dev/nullb0,node-name=disk0
> \
>     --export type=vduse-blk,id=vduse-test,name=vduse-test,node-name=disk0,writable=on
>
> [1] https://github.com/bytedance/qemu/tree/vduse-umem

Great, would you want to post them to the Qemu?

>
> >  drivers/vdpa/vdpa_user/iova_domain.c | 18 ++++++++++--------
> >  drivers/vdpa/vdpa_user/iova_domain.h |  1 +
> >  2 files changed, 11 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c
> > index 791d38d6284c..933d2f7cd49a 100644
> > --- a/drivers/vdpa/vdpa_user/iova_domain.c
> > +++ b/drivers/vdpa/vdpa_user/iova_domain.c
> > @@ -162,6 +162,7 @@ static void vduse_domain_bounce(struct vduse_iova_domain *domain,
> >                                 enum dma_data_direction dir)
> >  {
> >         struct vduse_bounce_map *map;
> > +       struct page *page;
> >         unsigned int offset;
> >         void *addr;
> >         size_t sz;
> > @@ -178,7 +179,10 @@ static void vduse_domain_bounce(struct vduse_iova_domain *domain,
> >                             map->orig_phys == INVALID_PHYS_ADDR))
> >                         return;
> >
> > -               addr = kmap_local_page(map->bounce_page);
> > +               page = domain->user_bounce_pages ?
> > +                      map->user_bounce_page : map->bounce_page;
> > +
> > +               addr = kmap_local_page(page);
> >                 do_bounce(map->orig_phys + offset, addr + offset, sz, dir);
> >                 kunmap_local(addr);
> >                 size -= sz;
> > @@ -270,9 +274,8 @@ int vduse_domain_add_user_bounce_pages(struct vduse_iova_domain *domain,
> >                                 memcpy_to_page(pages[i], 0,
> >                                                page_address(map->bounce_page),
> >                                                PAGE_SIZE);
> > -                       __free_page(map->bounce_page);
> >                 }
> > -               map->bounce_page = pages[i];
> > +               map->user_bounce_page = pages[i];
> >                 get_page(pages[i]);
> >         }
> >         domain->user_bounce_pages = true;
> > @@ -297,17 +300,16 @@ void vduse_domain_remove_user_bounce_pages(struct vduse_iova_domain *domain)
> >                 struct page *page = NULL;
> >
> >                 map = &domain->bounce_maps[i];
> > -               if (WARN_ON(!map->bounce_page))
> > +               if (WARN_ON(!map->user_bounce_page))
> >                         continue;
> >
> >                 /* Copy user page to kernel page if it's in use */
> >                 if (map->orig_phys != INVALID_PHYS_ADDR) {
> > -                       page = alloc_page(GFP_ATOMIC | __GFP_NOFAIL);
> > +                       page = map->bounce_page;
> >                         memcpy_from_page(page_address(page),
> > -                                        map->bounce_page, 0, PAGE_SIZE);
> > +                                        map->user_bounce_page, 0, PAGE_SIZE);
> >                 }
> > -               put_page(map->bounce_page);
> > -               map->bounce_page = page;
> > +               put_page(map->user_bounce_page);
>
> map->user_bounce_page = NULL?

For safety, it might be better. Let me add this and post a V2/

Thanks

>
> Thanks,
> Yongji
>

Powered by blists - more mailing lists