linux-kernel - Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <20080718161833.GA23381@us.ibm.com>
Date:	Fri, 18 Jul 2008 11:18:33 -0500
From:	"Serge E. Hallyn" <serue@...ibm.com>
To:	Oren Laadan <orenl@...columbia.edu>
Cc:	Dave Hansen <dave@...ux.vnet.ibm.com>,
	"Eric W. Biederman" <ebiederm@...ssion.com>,
	Kirill Korotaev <dev@...allels.com>,
	containers@...ts.linux-foundation.org,
	linux-kernel@...r.kernel.org, Nadia.Derbey@...l.net,
	Andrew Morton <akpm@...ux-foundation.org>,
	nick@...k-andrew.net, Alexey Dobriyan <adobriyan@...il.com>
Subject: Re: Checkpoint/restart (was Re: [PATCH 0/4] - v2 - Object creation
	with a specified id)

Quoting Oren Laadan (orenl@...columbia.edu):
>
>
> Serge E. Hallyn wrote:
>> Quoting Dave Hansen (dave@...ux.vnet.ibm.com):
>>> On Wed, 2008-07-09 at 18:58 -0700, Eric W. Biederman wrote:
>>>> In the worst case today we can restore a checkpoint by replaying all of
>>>> the user space actions that took us to get there.  That is a tedious
>>>> and slow approach.
>>> Yes, tedious and slow, *and* minimally invasive in the kernel.  Once we
>>> have a tedious and slow process, we'll have some really good points when
>>> we try to push the next set of patches to make it less slow and tedious.
>>> We'll be able to describe an _actual_ set of problems to our fellow
>>> kernel hackers.
>>>
>>> So, the checkpoint-as-a-corefile idea sounds good to me, but it
>>> definitely leaves a lot of questions about exactly how we'll need to do
>>> the restore.
>>
>> Talking with Dave over irc, I kind of liked the idea of creating a new
>> fs/binfmt_cr.c that executes a checkpoint-as-a-coredump file.
>>
>> One thing I do not like about the checkpoint-as-coredump is that it begs
>> us to dump all memory out into the file.  Our plan/hope was to save
>> ourselves from writing out most memory by:
>>
>> 	1. associating a separate swapfile with each container
>> 	2. doing a swapfile snapshot at each checkpoint
>> 	3. dumping the pte entries (/proc/self/)
>>
>> If we do checkpoint-as-a-coredump, then we need userspace to coordinate
>> a kernel-generated coredump with a user-generated (?) swapfile snapshot.
>> But I guess we figure that out later.
>
> I'm not sure how this approach integrates with (a) live migration (and
> the iterative process of sending over memory modified since previous
> iteration), and (b) incremental checkpoint (where except for the first
> snapshot, additional snapshots only save what changed since the previous
> one).

Oh, well I was seeing them as pretty orthogonal actually.  The reason is
that my checkpoint-as-a-coredump file would NOT include memory contents.
I'm still hoping that we can lock a container into its own nfs-mounted
swapfile, and take a snapshot of that at checkpoint.  Let the snapshot
solution take care of the incremental snapshot.

Attached are one kernel and one cryo patch, purely for a toy
implementation.  It doesn't even *quite* work yet.  Here are the notes
I wrote when I put it down wednesday afternoon:

===============================================================

The kernel patch impelments (only for x86_32) sys_checkpoint.  You
can create a checkpoint with the following test program:

#include <unistd.h>
#include <stdio.h>
#include <sys/syscall.h>
#include <errno.h>

int main()
{
        int ret;

        ret = syscall(327, "output.kckpt");
        printf("ret was %d\n", ret);
        if (ret == -1)
                perror("checkpoint");
        return 0;
}

Make output.kckpt executable and run it.  Works fine.

With the cryo patch, you can do:

SHELL 1:
./tests/sleep
SHELL 2:
./cr -p <sleeppid> -f o1.bin

and you'll see <sleeppid.kckpt> created.  Make that
executable and run it, and you'll see it runs as though
it were the original sleep program.

On the other hand, two problems with using cryo:

        1. the checkpointed app segfaults

        2. if you restart with "cr -r -f o1.bin" it
           fails somewhere.

===============================================================

Matt has already pointed out some coding errors so *really* it's
just to show one way we could integrate a binary format handler
for partial restart with more userspace control.  I'd be too
embarassed to send it out, but figure I should send it out before
the mini-summit.

-serge

View attachment "0001-checkpoint-add-sys_checkpoint-and-binfmt_cr.c.patch" of type "text/x-diff" (12578 bytes)

View attachment "0001-cryo--sys_checkpoint-first-attempt-at-exploiting.patch" of type "text/x-diff" (3191 bytes)