Message-ID: <CABi2SkUrvCcfydP9ejbeCxNms4BcgbhjOHgmvQyPBa_kgdUMPw@mail.gmail.com>
Date: Mon, 30 Sep 2024 17:10:50 -0700
From: Jeff Xu <jeffxu@...omium.org>
To: Randy Dunlap <rdunlap@...radead.org>
Cc: akpm@...ux-foundation.org, keescook@...omium.org, corbet@....net, 
	jorgelo@...omium.org, groeck@...omium.org, linux-kernel@...r.kernel.org, 
	linux-kselftest@...r.kernel.org, linux-mm@...ck.org, jannh@...gle.com, 
	sroettger@...gle.com, pedro.falcato@...il.com, 
	linux-hardening@...r.kernel.org, willy@...radead.org, 
	gregkh@...uxfoundation.org, torvalds@...ux-foundation.org, 
	deraadt@...nbsd.org, usama.anjum@...labora.com, surenb@...gle.com, 
	merimus@...gle.com, lorenzo.stoakes@...cle.com, Liam.Howlett@...cle.com, 
	enh@...gle.com
Subject: Re: [PATCH v1 1/1] mseal: update mseal.rst

Hi Randy,

On Sat, Sep 28, 2024 at 11:28 AM Randy Dunlap <rdunlap@...radead.org> wrote:
>
>
>
> On 9/27/24 11:52 AM, jeffxu@...omium.org wrote:
> > From: Jeff Xu <jeffxu@...omium.org>
> >
> > Update doc after in-loop change: mprotect/madvise can have
> > partially updated and munmap is atomic.
> >
> > Fix indentation and clarify some sections to improve readability.
> >
> > Signed-off-by: Jeff Xu <jeffxu@...omium.org>
> > ---
> >  Documentation/userspace-api/mseal.rst | 290 ++++++++++++--------------
> >  1 file changed, 136 insertions(+), 154 deletions(-)
> >
> > diff --git a/Documentation/userspace-api/mseal.rst b/Documentation/userspace-api/mseal.rst
> > index 4132eec995a3..68986084e191 100644
> > --- a/Documentation/userspace-api/mseal.rst
> > +++ b/Documentation/userspace-api/mseal.rst
> > @@ -23,177 +23,159 @@ applications can additionally seal security critical data at runtime.
> >  A similar feature already exists in the XNU kernel with the
> >  VM_FLAGS_PERMANENT flag [1] and on OpenBSD with the mimmutable syscall [2].
> >
> > -User API
> > -========
> > -mseal()
> > ------------
> > -The mseal() syscall has the following signature:
> > -
> > -``int mseal(void addr, size_t len, unsigned long flags)``
> > -
> > -**addr/len**: virtual memory address range.
> > -
> > -The address range set by ``addr``/``len`` must meet:
> > -   - The start address must be in an allocated VMA.
> > -   - The start address must be page aligned.
> > -   - The end address (``addr`` + ``len``) must be in an allocated VMA.
> > -   - no gap (unallocated memory) between start and end address.
> > -
> > -The ``len`` will be paged aligned implicitly by the kernel.
> > -
> > -**flags**: reserved for future use.
> > -
> > -**return values**:
> > -
> > -- ``0``: Success.
> > -
> > -- ``-EINVAL``:
> > -    - Invalid input ``flags``.
> > -    - The start address (``addr``) is not page aligned.
> > -    - Address range (``addr`` + ``len``) overflow.
> > -
> > -- ``-ENOMEM``:
> > -    - The start address (``addr``) is not allocated.
> > -    - The end address (``addr`` + ``len``) is not allocated.
> > -    - A gap (unallocated memory) between start and end address.
> > -
> > -- ``-EPERM``:
> > -    - sealing is supported only on 64-bit CPUs, 32-bit is not supported.
> > -
> > -- For above error cases, users can expect the given memory range is
> > -  unmodified, i.e. no partial update.
> > -
> > -- There might be other internal errors/cases not listed here, e.g.
> > -  error during merging/splitting VMAs, or the process reaching the max
> > -  number of supported VMAs. In those cases, partial updates to the given
> > -  memory range could happen. However, those cases should be rare.
> > -
> > -**Blocked operations after sealing**:
> > -    Unmapping, moving to another location, and shrinking the size,
> > -    via munmap() and mremap(), can leave an empty space, therefore
> > -    can be replaced with a VMA with a new set of attributes.
> > -
> > -    Moving or expanding a different VMA into the current location,
> > -    via mremap().
> > -
> > -    Modifying a VMA via mmap(MAP_FIXED).
> > -
> > -    Size expansion, via mremap(), does not appear to pose any
> > -    specific risks to sealed VMAs. It is included anyway because
> > -    the use case is unclear. In any case, users can rely on
> > -    merging to expand a sealed VMA.
> > -
> > -    mprotect() and pkey_mprotect().
> > -
> > -    Some destructive madvice() behaviors (e.g. MADV_DONTNEED)
> > -    for anonymous memory, when users don't have write permission to the
> > -    memory. Those behaviors can alter region contents by discarding pages,
> > -    effectively a memset(0) for anonymous memory.
> > -
> > -    Kernel will return -EPERM for blocked operations.
> > -
> > -    For blocked operations, one can expect the given address is unmodified,
> > -    i.e. no partial update. Note, this is different from existing mm
> > -    system call behaviors, where partial updates are made till an error is
> > -    found and returned to userspace. To give an example:
> > -
> > -    Assume following code sequence:
> > -
> > -    - ptr = mmap(null, 8192, PROT_NONE);
> > -    - munmap(ptr + 4096, 4096);
> > -    - ret1 = mprotect(ptr, 8192, PROT_READ);
> > -    - mseal(ptr, 4096);
> > -    - ret2 = mprotect(ptr, 8192, PROT_NONE);
> > -
> > -    ret1 will be -ENOMEM, the page from ptr is updated to PROT_READ.
> > -
> > -    ret2 will be -EPERM, the page remains to be PROT_READ.
> > -
> > -**Note**:
> > -
> > -- mseal() only works on 64-bit CPUs, not 32-bit CPU.
> > -
> > -- users can call mseal() multiple times, mseal() on an already sealed memory
> > -  is a no-action (not error).
> > -
> > -- munseal() is not supported.
> > +SYSCALL
> > +=======
> > +mseal syscall signature
> > +-----------------------
> > +   **int** mseal(**void \*** addr, **size_t** len, **unsigned long** flags)
>
> ugh. totally unreadable for people who just look at .rst files.
>
Right!
The ** (bold) in rst collides with a pointer in c. it does make it
unreadable. I will  remove them.

> Does other documentation go to this extreme?
>
> > +
> > +   **addr**/**len**: virtual memory address range.
> > +      The address range set by **addr**/**len** must meet:
> > +         - The start address must be in an allocated VMA.
> > +         - The start address must be page aligned.
> > +         - The end address (**addr** + **len**) must be in an allocated VMA.
> > +         - no gap (unallocated memory) between start and end address.
> > +
> > +      The ``len`` will be paged aligned implicitly by the kernel.
> > +
> > +   **flags**: reserved for future use.
> > +
> > +   **Return values**:
> > +      - **0**: Success.
> > +      - **-EINVAL**:
> > +         * Invalid input ``flags``.
> > +         * The start address (``addr``) is not page aligned.
> > +         * Address range (``addr`` + ``len``) overflow.
> > +      - **-ENOMEM**:
> > +         * The start address (``addr``) is not allocated.
> > +         * The end address (``addr`` + ``len``) is not allocated.
> > +         * A gap (unallocated memory) between start and end address.
> > +      - **-EPERM**:
> > +         * sealing is supported only on 64-bit CPUs, 32-bit is not supported.
> > +
> > +   **Note about error return**:
> > +      - For above error cases, users can expect the given memory range is
> > +        unmodified, i.e. no partial update.
> > +      - There might be other internal errors/cases not listed here, e.g.
> > +        error during merging/splitting VMAs, or the process reaching the max
> > +        number of supported VMAs. In those cases, partial updates to the given
> > +        memory range could happen. However, those cases should be rare.
> > +
> > +   **Architecture support**:
> > +      mseal only works on 64-bit CPUs, not 32-bit CPU.
>
>                                                      CPUs.
>
Done, thanks.

> > +
> > +   **Idempotent**:
> > +      users can call mseal multiple times, mseal on an already sealed memory
>
>                                        times. mseal
>
Done, thanks.

> > +      is a no-action (not error).
> > +
> > +   **no munseal**
> > +      Once mapping is sealed, it can't be unsealed. kernel should never
> > +      have munseal, this is consistent with other sealing feature, e.g.
> > +      F_SEAL_SEAL for file.
> > +
> > +Blocked mm syscall for sealed mapping
> > +-------------------------------------
> > +   It might be imporant to note: **once the mapping is sealed, it will
>
>                   important
>
Done, thanks.

> > +   stay in the process's memory till the process terminates**.
> > +
> > +   Example::
> > +
> > +         *ptr = mmap(0, 4096, PROT_READ, MAP_ANONYMOUS | MAP_PRIVATE, 0, 0);
> > +         rc = mseal(ptr, 4096, 0);
> > +         /* munmap will fail */
> > +         rc = munmap(ptr, 4096);
> > +         assert(rc < 0);
> > +
> > +   Blocked mm syscall:
> > +      - munmap
> > +      - mmap
> > +      - mremap
> > +      - mprotect and pkey_mprotect
> > +      - some destructive madvise behaviors: MADV_DONTNEED, MADV_FREE,
> > +        MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_WIPEONFORK
> > +
> > +   The first set of syscall to block is munmap, mremap, mmap. They can
> > +   either leave an empty space in the address space, therefore allow
> > +   replacement with a new mapping with new set of attributes, or can
> > +   overwrite the existing mapping with another mapping.
> > +
> > +   mprotect and pkey_mprotect are blocked because they changes the
> > +   protection bits (rwx) of the mapping.
>
> preferably            (RWX)
>
Done.

> > +
> > +   Some destructive madvice behaviors (MADV_DONTNEED, MADV_FREE,
>
>                        madvise
>
Done.

> > +   MADV_DONTNEED_LOCKED, MADV_FREE, MADV_DONTFORK, MADV_WIPEONFORK)
> > +   for anonymous memory, when users don't have write permission to the
> > +   memory. Those behaviors can alter region contents by discarding pages,
> > +   effectively a memset(0) for anonymous memory.
> > +
> > +   Kernel will return -EPERM for blocked syscalls.
> > +
> > +   When blocked syscall return -EPERM due to sealing, the memory regions may or may not be changed, depends on the syscall being blocked:
> > +      - munmap: munmap is atomic. If one of VMAs in the given range is
> > +        sealed, none of VMAs are updated.
> > +      - mprotect, pkey_mprotect, madvise: partial update might happen, e.g.
> > +        when mprotect over multiple VMAs, mprotect might update the beginning
> > +        VMAs before reaching the sealed VMA and return -EPERM.
> > +      - mmap and mremap: undefined behavior.
> >
> >  Use cases:
> >  ==========
> >  - glibc:
> >    The dynamic linker, during loading ELF executables, can apply sealing to
> > -  non-writable memory segments.
> > +  mapping segments.
> >
> >  - Chrome browser: protect some security sensitive data-structures.
>
>                                                      data structures.
>
Done. Thanks.

> >
> > -Notes on which memory to seal:
> > -==============================
> > -
> > -It might be important to note that sealing changes the lifetime of a mapping,
> > -i.e. the sealed mapping won’t be unmapped till the process terminates or the
> > -exec system call is invoked. Applications can apply sealing to any virtual
> > -memory region from userspace, but it is crucial to thoroughly analyze the
> > -mapping's lifetime prior to apply the sealing.
> > +Don't use mseal on:
>
> Drop the ':'. Headings should not uses trailing colons.
> (throughout this file)
>
Done everywhere. Thanks.

> Maybe change the heading to "When not to use mseal".
>
> > +===================
> > +Applications can apply sealing to any virtual memory region from userspace,
> > +but it is *crucial to thoroughly analyze the mapping's lifetime* prior to
> > +apply the sealing. This is because the sealed mapping *won’t be unmapped*
> > +till the process terminates or the exec system call is invoked.
>
> s/till/until/ preferably.
>
ok.

> >
> >  For example:
> > +   - aio/shm
> > +     aio/shm can call mmap and  munmap on behalf of userspace, e.g.
> > +     ksys_shmdt() in shm.c. The lifetime of those mapping are not tied to
> > +     the lifetime of the process. If those memories are sealed from userspace,
> > +     then munmap will fail, causing leaks in VMA address space during the
> > +     lifetime of the process.
> > +
> > +   - ptr allocated by malloc (heap)
> > +     Don't use mseal on the memory ptr return from malloc().
> > +     malloc() is implemented by allocator, e.g. by glibc. Heap manager might
> > +     allocate a ptr from brk or mapping created by mmap.
> > +     If app calls mseal on ptr returned from malloc(), this can affect the heap
>
>         If an app calls mseal on a ptr
>
Done. thanks

>
> > +     manager's ability to manage the mappings, the outcome is non-deterministic.
>
>                                         mappings; the outcome
>
Done, thanks.

> > +     Example::
> > +
> > +        ptr = malloc(size);
> > +        /* don't call mseal on ptr return from malloc. */
> > +        mseal(ptr, size);
> > +        /* free will success, allocator can't shrink heap lower than ptr */
> > +        free(ptr);
> > +
> > +mseal doesn't block:
> > +====================
> > +In a nutshell, mseal blocks certain mm syscall from modifying some of VMA's
> > +attributes, such as protection bits (rwx). Sealed mappings doesn't mean the
>
> preferably                             (RWX).
>
Done, thanks.

> > +memory is immutable.
> >
> > -- aio/shm
> > -
> > -  aio/shm can call mmap()/munmap() on behalf of userspace, e.g. ksys_shmdt() in
> > -  shm.c. The lifetime of those mapping are not tied to the lifetime of the
>
>                 lifetimes
OK.
> ?
>
> > -  process. If those memories are sealed from userspace, then munmap() will fail,
> > -  causing leaks in VMA address space during the lifetime of the process.
> > -
> > -- Brk (heap)
> > -
> > -  Currently, userspace applications can seal parts of the heap by calling
> > -  malloc() and mseal().
> > -  let's assume following calls from user space:
> > -
> > -  - ptr = malloc(size);
> > -  - mprotect(ptr, size, RO);
> > -  - mseal(ptr, size);
> > -  - free(ptr);
> > -
> > -  Technically, before mseal() is added, the user can change the protection of
> > -  the heap by calling mprotect(RO). As long as the user changes the protection
> > -  back to RW before free(), the memory range can be reused.
> > -
> > -  Adding mseal() into the picture, however, the heap is then sealed partially,
> > -  the user can still free it, but the memory remains to be RO. If the address
> > -  is re-used by the heap manager for another malloc, the process might crash
> > -  soon after. Therefore, it is important not to apply sealing to any memory
> > -  that might get recycled.
> > -
> > -  Furthermore, even if the application never calls the free() for the ptr,
> > -  the heap manager may invoke the brk system call to shrink the size of the
> > -  heap. In the kernel, the brk-shrink will call munmap(). Consequently,
> > -  depending on the location of the ptr, the outcome of brk-shrink is
> > -  nondeterministic.
> > -
> > -
> > -Additional notes:
> > -=================
> >  As Jann Horn pointed out in [3], there are still a few ways to write
> > -to RO memory, which is, in a way, by design. Those cases are not covered
> > -by mseal(). If applications want to block such cases, sandbox tools (such as
> > -seccomp, LSM, etc) might be considered.
> > +to RO memory, which is, in a way, by design. And those could be blocked
> > +by different security measures.
> >
> >  Those cases are:
> > -
> > -- Write to read-only memory through /proc/self/mem interface.
> > -- Write to read-only memory through ptrace (such as PTRACE_POKETEXT).
> > -- userfaultfd.
> > +   - Write to read-only memory through /proc/self/mem interface (FOLL_FORCE).
> > +   - Write to read-only memory through ptrace (such as PTRACE_POKETEXT).
> > +   - userfaultfd.
> >
> >  The idea that inspired this patch comes from Stephen Röttger’s work in V8
> >  CFI [4]. Chrome browser in ChromeOS will be the first user of this API.
> >
> >  Reference:
> >  ==========
> > -[1] https://github.com/apple-oss-distributions/xnu/blob/1031c584a5e37aff177559b9f69dbd3c8c3fd30a/osfmk/mach/vm_statistics.h#L274
> > -
> > -[2] https://man.openbsd.org/mimmutable.2
> > -
> > -[3] https://lore.kernel.org/lkml/CAG48ez3ShUYey+ZAFsU2i1RpQn0a5eOs2hzQ426FkcgnfUGLvA@mail.gmail.com
> > -
> > -[4] https://docs.google.com/document/d/1O2jwK4dxI3nRcOJuPYkonhTkNQfbmwdvxQMyXgeaRHo/edit#heading=h.bvaojj9fu6hc
> > +- [1] https://github.com/apple-oss-distributions/xnu/blob/1031c584a5e37aff177559b9f69dbd3c8c3fd30a/osfmk/mach/vm_statistics.h#L274
> > +- [2] https://man.openbsd.org/mimmutable.2
> > +- [3] https://lore.kernel.org/lkml/CAG48ez3ShUYey+ZAFsU2i1RpQn0a5eOs2hzQ426FkcgnfUGLvA@mail.gmail.com
> > +- [4] https://docs.google.com/document/d/1O2jwK4dxI3nRcOJuPYkonhTkNQfbmwdvxQMyXgeaRHo/edit#heading=h.bvaojj9fu6hc

I will update and send v2.

Appreciate the review comments.
-Jeff

Powered by blists - more mailing lists