[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <tencent_A4E83BA6071B2204B6F5D4E69A50D21C1A09@qq.com>
Date: Fri, 22 Apr 2022 21:15:01 +0800
From: "zhangfei.gao@...mail.com" <zhangfei.gao@...mail.com>
To: Jean-Philippe Brucker <jean-philippe@...aro.org>
Cc: Fenghua Yu <fenghua.yu@...el.com>,
Dave Hansen <dave.hansen@...el.com>,
Joerg Roedel <joro@...tes.org>,
Ravi V Shankar <ravi.v.shankar@...el.com>,
Tony Luck <tony.luck@...el.com>,
Ashok Raj <ashok.raj@...el.com>,
Peter Zijlstra <peterz@...radead.org>,
Dave Hansen <dave.hansen@...ux.intel.com>,
x86 <x86@...nel.org>,
linux-kernel <linux-kernel@...r.kernel.org>,
iommu <iommu@...ts.linux-foundation.org>,
Ingo Molnar <mingo@...hat.com>, Borislav Petkov <bp@...en8.de>,
Andy Lutomirski <luto@...nel.org>,
Josh Poimboeuf <jpoimboe@...hat.com>,
Thomas Gleixner <tglx@...utronix.de>, will@...nel.org,
robin.murphy@....com, zhangfei.gao@...aro.org
Subject: Re: Re: [PATCH v4 05/11] iommu/sva: Assign a PASID to mm on PASID
allocation and free it on mm exit
Hi, Jean
On 2022/4/22 下午6:11, Jean-Philippe Brucker wrote:
> On Fri, Apr 22, 2022 at 05:03:10PM +0800, zhangfei.gao@...mail.com wrote:
> [...]
>>> Have tested, still got some issue with our openssl-engine.
>>>
>>> 1. If openssl-engine does not register rsa, nginx works well.
>>>
>>> 2. If openssl-engine register rsa, nginx also works, but ioasid is not
>>> freed when nginx stop.
>>>
>>> IMPLEMENT_DYNAMIC_BIND_FN(bind_fn)
>>> bind_fn
>>> ENGINE_set_RSA(e, rsa_methods())
>>>
>>> destroy_fn
>>>
>>> If ENGINE_set_RSA is set, nginx start and stop will NOT call destroy_fn.
>>> Even rsa_methods is almost new via RSA_meth_new.
>>>
>>> In 5.18-rcx, this caused ioasid not freed in nginx start and stop.
>>> In 5.17, though destroy_fn is not called, but ioasid is freed when nginx
>>> stop, so not noticed this issue before.
>> 1. uacce_fops_release
>> In 5.16 or 5.17
>> In fact, we aslo has the issue: openssl engine does not call destroy_fn ->
>> close(uacce_fd)
>> But system will automatically close all opened fd,
>> so uacce_fops_release is also called and free ioasid.
>>
>> Have one experiment, not call close fd
>>
>> log: open uacce fd but no close
>> [ 2583.471225] dump_backtrace+0x0/0x1a0
>> [ 2583.474876] show_stack+0x20/0x30
>> [ 2583.478178] dump_stack_lvl+0x8c/0xb8
>> [ 2583.481825] dump_stack+0x18/0x34
>> [ 2583.485126] uacce_fops_release+0x44/0xdc
>> [ 2583.489117] __fput+0x78/0x240
>> [ 2583.492159] ____fput+0x18/0x28
>> [ 2583.495288] task_work_run+0x88/0x160
>> [ 2583.498936] do_notify_resume+0x214/0x490
>> [ 2583.502927] el0_svc+0x58/0x70
>> [ 2583.505968] el0t_64_sync_handler+0xb0/0xb8
>> [ 2583.510132] el0t_64_sync+0x1a0/0x1a4
>> [ 2583.582292] uacce_fops_release q=00000000d6674128
>>
>> In 5.18, since refcount was add.
>> The opened uacce fd was not closed automatically by system
>> So we see the issue.
>>
>> log: open uacce fd but no close
>> [ 106.360140] uacce_fops_open q=00000000ccc38d74
>> [ 106.364929] ioasid_alloc ioasid=1
>> [ 106.368585] iommu_sva_alloc_pasid pasid=1
>> [ 106.372943] iommu_sva_bind_device handle=000000006cca298a
>> // ioasid is not free
> I'm trying to piece together what happens from the kernel point of view.
>
> * master process with mm A opens a queue fd through uacce, which calls
> iommu_sva_bind_device(dev, A) -> PASID 1
>
> * master forks and exits. Child (daemon) gets mm B, inherits the queue fd.
> The device is still bound to mm A with PASID 1, since the queue fd is
> still open.
> We discussed this before, but I don't remember where we left off. The
> child can't use the queue because its mappings are not copied on fork(),
> and the queue is still bound to the parent mm A. The child either needs to
> open a new queue or take ownership of the old one with a new uacce ioctl.
Yes, currently nginx aligned with the case.
Child process (worker process) reopen uacce,
Master process (do init) open uacce, iommu_sva_bind_device(dev, A) ->
PASID 1
Master process fork Child (daemon) and exit.
Child (daemon) does not use PASID 1 any more, only fork and manage
worker process.
Worker process reopen uacce, iommu_sva_bind_device(dev, B) PASID 2
So it is expected.
> Is that the "IMPLEMENT_DYNAMIC_BIND_FN()" you mention, something out of
> tree? This operation should unbind from A before binding to B, no?
> Otherwise we leak PASID 1.
In 5.16 PASID 1 from master is hold until nginx service stop.
nginx start
master:
iommu_sva_alloc_pasid mm->pasid=1 // master process
lynx https start:
iommu_sva_alloc_pasid mm->pasid=2 //worker process
nginx stop: from fops_release
iommu_sva_free_pasid mm->pasid=2 // worker process
iommu_sva_free_pasid mm->pasid=1 // master process
Have one silly question.
kerne driver
fops_open
iommu_sva_bind_device
fops_release
iommu_sva_unbind_device
application
main()
fd = open
return;
Application exit but not close(fd), is it expected fops_release will be
called automatically by system?
On 5.17
fops_release is called automatically, as well as iommu_sva_unbind_device.
On 5.18-rc1.
fops_release is not called, have to manually call close(fd)
Since nginx may have a issue, it does not call close(fd) when nginx -s quit.
Thanks
Powered by blists - more mailing lists