| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 23 Jul 2014 11:35:59 +0300 From: Oded Gabbay <oded.gabbay@....com> To: Daniel Vetter <daniel.vetter@...ll.ch> CC: Jerome Glisse <j.glisse@...il.com>, Christian König <deathsimple@...afone.de>, David Airlie <airlied@...ux.ie>, Alex Deucher <alexdeucher@...il.com>, Andrew Morton <akpm@...ux-foundation.org>, "John Bridgman" <John.Bridgman@....com>, Joerg Roedel <joro@...tes.org>, "Andrew Lewycky" <Andrew.Lewycky@....com>, Michel Dänzer <michel.daenzer@....com>, Ben Goz <Ben.Goz@....com>, Alexey Skidanov <Alexey.Skidanov@....com>, "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>, "dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>, linux-mm <linux-mm@...ck.org>, "Sellek, Tom" <Tom.Sellek@....com> Subject: Re: [PATCH v2 00/25] AMDKFD kernel driver On 23/07/14 10:05, Daniel Vetter wrote: > On Wed, Jul 23, 2014 at 8:50 AM, Oded Gabbay <oded.gabbay@....com> wrote: >> On 22/07/14 14:15, Daniel Vetter wrote: >>> >>> On Tue, Jul 22, 2014 at 12:52:43PM +0300, Oded Gabbay wrote: >>>> >>>> On 22/07/14 12:21, Daniel Vetter wrote: >>>>> >>>>> On Tue, Jul 22, 2014 at 10:19 AM, Oded Gabbay <oded.gabbay@....com> >>>>> wrote: >>>>>>> >>>>>>> Exactly, just prevent userspace from submitting more. And if you have >>>>>>> misbehaving userspace that submits too much, reset the gpu and tell it >>>>>>> that you're sorry but won't schedule any more work. >>>>>> >>>>>> >>>>>> I'm not sure how you intend to know if a userspace misbehaves or not. >>>>>> Can >>>>>> you elaborate ? >>>>> >>>>> >>>>> Well that's mostly policy, currently in i915 we only have a check for >>>>> hangs, and if userspace hangs a bit too often then we stop it. I guess >>>>> you can do that with the queue unmapping you've describe in reply to >>>>> Jerome's mail. >>>>> -Daniel >>>>> >>>> What do you mean by hang ? Like the tdr mechanism in Windows (checks if a >>>> gpu job takes more than 2 seconds, I think, and if so, terminates the >>>> job). >>> >>> >>> Essentially yes. But we also have some hw features to kill jobs quicker, >>> e.g. for media workloads. >>> -Daniel >>> >> >> Yeah, so this is what I'm talking about when I say that you and Jerome come >> from a graphics POV and amdkfd come from a compute POV, no offense intended. >> >> For compute jobs, we simply can't use this logic to terminate jobs. Graphics >> are mostly Real-Time while compute jobs can take from a few ms to a few >> hours!!! And I'm not talking about an entire application runtime but on a >> single submission of jobs by the userspace app. We have tests with jobs that >> take between 20-30 minutes to complete. In theory, we can even imagine a >> compute job which takes 1 or 2 days (on larger APUs). >> >> Now, I understand the question of how do we prevent the compute job from >> monopolizing the GPU, and internally here we have some ideas that we will >> probably share in the next few days, but my point is that I don't think we >> can terminate a compute job because it is running for more than x seconds. >> It is like you would terminate a CPU process which runs more than x seconds. >> >> I think this is a *very* important discussion (detecting a misbehaved >> compute process) and I would like to continue it, but I don't think moving >> the job submission from userspace control to kernel control will solve this >> core problem. > > Well graphics gets away with cooperative scheduling since usually > people want to see stuff within a few frames, so we can legitimately > kill jobs after a fairly short timeout. Imo if you want to allow > userspace to submit compute jobs that are atomic and take a few > minutes to hours with no break-up in between and no hw means to > preempt then that design is screwed up. We really can't tell the core > vm that "sorry we will hold onto these gobloads of memory you really > need now for another few hours". Pinning memory like that essentially > without a time limit is restricted to root. > -Daniel > First of all, I don't see the relation to memory pinning here. I already said on this thread that amdkfd does NOT pin local memory. The only memory we allocate is system memory, and we map it to the gart, and we can limit that memory by limiting max # of queues and max # of process through kernel parameters. Most of the memory used is allocated via regular means by the userspace, which is usually pageable. Second, it is important to remember that this problem only exists in KV. In CZ, the GPU can context switch between waves (by doing mid-wave preemption). So even long running waves are getting switched on and off constantly and there is no monopolizing of GPU resources. Third, even in KV, we can kill waves. The question is when and how to recognize it. I think it would be sufficient for now if we expose this ability to the kernel. Oded -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists