Message-ID: <dd99213b-3caf-4fc0-1bf5-314297e3d450@amazon.com>
Date:   Thu, 23 Jul 2020 21:21:51 +0300
From:   "Paraschiv, Andra-Irina" <andraprs@...zon.com>
To:     Greg KH <gregkh@...uxfoundation.org>
CC:     <linux-kernel@...r.kernel.org>,
        Anthony Liguori <aliguori@...zon.com>,
        Benjamin Herrenschmidt <benh@...nel.crashing.org>,
        Colm MacCarthaigh <colmmacc@...zon.com>,
        David Duncan <davdunc@...zon.com>,
        Bjoern Doebel <doebel@...zon.de>,
        David Woodhouse <dwmw@...zon.co.uk>,
        Frank van der Linden <fllinden@...zon.com>,
        Alexander Graf <graf@...zon.de>, Karen Noel <knoel@...hat.com>,
        Martin Pohlack <mpohlack@...zon.de>,
        Matt Wilson <msw@...zon.com>,
        Paolo Bonzini <pbonzini@...hat.com>,
        Balbir Singh <sblbir@...zon.com>,
        Stefano Garzarella <sgarzare@...hat.com>,
        "Stefan Hajnoczi" <stefanha@...hat.com>,
        Stewart Smith <trawets@...zon.com>,
        "Uwe Dannowski" <uwed@...zon.de>,
        Vitaly Kuznetsov <vkuznets@...hat.com>, <kvm@...r.kernel.org>,
        <ne-devel-upstream@...zon.com>, Alexander Graf <graf@...zon.com>
Subject: Re: [PATCH v5 01/18] nitro_enclaves: Add ioctl interface definition



On 23/07/2020 13:54, Greg KH wrote:
> On Thu, Jul 23, 2020 at 12:23:56PM +0300, Paraschiv, Andra-Irina wrote:
>>
>> On 22/07/2020 12:57, Greg KH wrote:
>>> On Wed, Jul 22, 2020 at 11:27:29AM +0300, Paraschiv, Andra-Irina wrote:
>>>>>> +#ifndef _UAPI_LINUX_NITRO_ENCLAVES_H_
>>>>>> +#define _UAPI_LINUX_NITRO_ENCLAVES_H_
>>>>>> +
>>>>>> +#include <linux/types.h>
>>>>>> +
>>>>>> +/* Nitro Enclaves (NE) Kernel Driver Interface */
>>>>>> +
>>>>>> +#define NE_API_VERSION (1)
>>>>> Why do you need this version?  It shouldn't be needed, right?
>>>> The version is used as a way for the user space tooling to sync on the
>>>> features set provided by the driver e.g. in case an older version of the
>>>> driver is available on the system and the user space tooling expects a set
>>>> of features that is not included in that driver version.
>>> That is guaranteed to get out of sync instantly with different distro
>>> kernels backporting random things, combined with stable kernel patch
>>> updates and the like.
>>>
>>> Just use the normal api interfaces instead, don't try to "version"
>>> anything, it will not work, trust us :)
>>>
>>> If an ioctl returns -ENOTTY then hey, it's not present and your
>>> userspace code can handle it that way.
>> Correct, there could be a variety of kernel versions and user space tooling
>> either in the original form, customized or written from scratch. And ENOTTY
>> signals an ioctl not available or e.g. EINVAL (or custom error) if the
>> parameter field value is not valid within a certain version. We have these
>> in place, that's good. :)
>>
>> However, I was thinking, for example, of an ioctl flow usage where a certain
>> order needs to be followed e.g. create a VM, add resources to a VM, start a
>> VM.
>>
>> Let's say, for an use case wrt new features, ioctl A (create a VM) succeeds,
>> ioctl B (add memory to the VM) succeeds, ioctl C (add CPU to the VM)
>> succeeds and ioctl D (add any other type of resource before starting the VM)
>> fails because it is not supported.
>>
>> Would not need to call ioctl A to C and go through their underneath logic to
>> realize ioctl D support is not there and rollback all the changes done till
>> then within ioctl A to C logic. Of course, there could be ioctl A followed
>> by ioctl D, and would need to rollback ioctl A changes, but I shared a more
>> lengthy call chain that can be an option as well.
> I think you are overthinking this.
>
> If your interface is this complex, you have much larger issues as you
> ALWAYS have to be able to handle error conditions properly, even if the
> API is "supported".

True, the error paths need to handled correctly on the kernel driver and 
on the user space logic side, independent of supported features or not. 
Cannot assume that all ioctl callers are behaving correctly or there are 
no errors in the system.

What I wanted to cover with that example is more towards the user space 
logic using new features, either early exiting before even trying the 
ioctl call flow path or going through part of the flow till getting the 
error e.g. ENOTTY for one of the ioctl calls.

>
> Perhaps your API is showing to be too complex?
>
> Also, where is the userspace code for all of this?  Did I miss a link to
> it in the patches somewhere?

Nope, you didn't miss any references to it. The codebase for the user 
space code is not publicly available for now, but it will be available 
on GitHub once the whole project is GA. And I'll include the refs, once 
available, in the NE kernel driver documentation.

I can summarize here the ioctl interface usage flow, let me know if I 
can help with more clarifications:

Enclave creation

* Open the misc device (/dev/nitro_enclaves) and get the device fd.
* Using the device fd, call NE_GET_API_VERSION to check the API version.
* Using the device fd, call NE_CREATE_VM and get an enclave fd.
* Using the enclave fd, call NE_GET_IMAGE_LOAD_INFO to get the offset in 
the enclave memory where to place the enclave image. Enclave memory 
regions consist of hugetlbfs huge pages. Place the enclave image in 
enclave memory.
* Using the enclave fd, call NE_SET_USER_MEMORY_REGION to set a memory 
region for an enclave. Repeat this step for all enclave memory regions.
* Using the enclave fd, call NE_ADD_VCPU to add a vCPU for an enclave. 
Repeat this step for all enclave vCPUs. The CPUs are part of the NE CPU 
pool, set using a sysfs file before starting to create an enclave.
* Using the enclave fd, call NE_START_ENCLAVE to start an enclave.


Enclave termination

* Close the enclave fd.


Thanks,
Andra

>
> good luck!
>
> greg k-h




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

Powered by blists - more mailing lists