Date:	Thu, 20 Nov 2008 16:44:40 +0900
From:	"Ryota OZAKI" <ozaki.ryota@...il.com>
To:	"Izik Eidus" <ieidus@...hat.com>
Cc:	akpm@...ux-foundation.org, linux-kernel@...r.kernel.org,
	linux-mm@...ck.org, kvm@...r.kernel.org, aarcange@...hat.com,
	chrisw@...hat.com, avi@...hat.com, dlaor@...hat.com,
	kamezawa.hiroyu@...fujitsu.com, cl@...ux-foundation.org,
	corbet@....net
Subject: Re: [PATCH 0/4] ksm - dynamic page sharing driver for linux v2

Hi Izik,

I've tried your patch set, but ksm doesn't work in my machine.

I compiled linux patched with the four patches and configured with KSM
and KVM enabled. After boot with the linux, I run two VMs running linux
using QEMU with a patch in your mail and started KSM scanner with your
script, then the host linux caused panic with the following oops.


== BEGINNING of OOPS
kernel BUG at arch/x86/mm/highmem_32.c:87!
invalid opcode: 0000 [#1] SMP
last sysfs file: /sys/class/net/vnet-ssh2/address
Dumping ftrace buffer:
   (ftrace buffer empty)
Modules linked in: netconsole autofs4 nf_conntrack_ipv4 nf_defrag_ipv4
xt_state nf_conntrack xt_tcpudp ipt_REJECT iptable_filter ip_tables
x_tables loop kvm_intel kvm iTCO_wdt iTCO_vendor_support igb
netxen_nic button ext3 jbd mbcache uhci_hcd ohci_hcd ehci_hcd usbcore
[last unloaded: microcode]

Pid: 343, comm: kksmd Not tainted
(2.6.28-rc5-linus-head-20081119-sparsemem #1) X7DWA
EIP: 0060:[<c041eff9>] EFLAGS: 00010206 CPU: 6
EIP is at kmap_atomic_prot+0x7d/0xeb
EAX: c0008d94 EBX: c1ff6240 ECX: 00000163 EDX: 7e000000
ESI: 00000154 EDI: 00000055 EBP: f5cdbf10 ESP: f5cdbef8
 DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0068
Process kksmd (pid: 343, ti=f5cda000 task=f617b140 task.ti=f5cda000)
Stack:
 7fa12163 fffff000 c204efbc f50479e8 9eb7e000 c08a34d0 f5cdbf18 c041f07a
 f5cdbf28 c048339c 00000000 f5c271e0 f5cdbf30 c04833bc f5cdbfb0 c0483b0d
 f5cdbf50 c0425845 00000000 00000064 00000009 c08a34d0 f5cdbfb0 c06384c1
Call Trace:
 [<c041f07a>] ? kmap_atomic+0x13/0x15
 [<c048339c>] ? get_pte+0x50/0x63
 [<c04833bc>] ? is_present_pte+0xd/0x1f
 [<c0483b0d>] ? ksm_scan_start+0x9a/0x7ac
 [<c0425845>] ? finish_task_switch+0x29/0xa4
 [<c06384c1>] ? schedule+0x6bf/0x719
 [<c041b3fc>] ? default_spin_lock_flags+0x8/0xc
 [<c043bffa>] ? finish_wait+0x49/0x4e
 [<c04845f4>] ? kthread_ksm_scan_thread+0x0/0xdc
 [<c048462e>] ? kthread_ksm_scan_thread+0x3a/0xdc
 [<c043bf31>] ? autoremove_wake_function+0x0/0x38
 [<c043be3e>] ? kthread+0x40/0x66
 [<c043bdfe>] ? kthread+0x0/0x66
 [<c0404997>] ? kernel_thread_helper+0x7/0x10
Code: 86 00 00 00 64 a1 04 a0 82 c0 6b c0 0d 8d 3c 30 a1 78 b0 77 c0
8d 34 bd 00 00 00 00 89 45 ec a1 0c d0 84 c0 29 f0 83 38 00 74 04 <0f>
0b eb fe c1 ea 1a 8b 04 d5 80 32 8a c0 83 e0 fc 29 c3 c1 fb
EIP: [<c041eff9>] kmap_atomic_prot+0x7d/0xeb SS:ESP 0068:f5cdbef8
Kernel panic - not syncing: Fatal exception
== END of OOPS


I used a version of linux kernel from linus tree (commit e14c8bf863...
at Mon Nov 17)
with a configuration as the attached file and kvm-userspace (commit
0cf57d569f3...
at Mon Nov 10). Other configurations are as follows:

Host: Fedora 8
Arch: i686
CPU: Intel(R) Xeon(R) CPU E5410  @ 2.33GHz
Mem: 2 GB
Guest: Fedora Core 7, linux-2.6.27
Cmd: qemu-system-x86_64 -kernel vmlinuz-2.6.27 -initrd
initrd-2.6.27.img -append ro root=/dev/VolGroup00/LogVol00
kvm_paravirt=1 loglevel=4 rhgb quiet -hda -M pc -smp 1 -curses -clock
dynticks -net nic,vlan=0,macaddr=00:00:00:1:11:0,model=virtio -net
tap,vlan=0,ifname=vnet1,script=no -m 384


Note that if I disable HIGHMEM of the host kernel, ksm works well.

Any idea?

Thanks in advance,
  ozaki-r


2008/11/17 Izik Eidus <ieidus@...hat.com>:
> (From v1 to v2 the main change is much more documentation)
>
> KSM is a linux driver that allows dynamicly sharing identical memory
> pages between one or more processes.
>
> Unlike tradtional page sharing that is made at the allocation of the
> memory, ksm do it dynamicly after the memory was created.
> Memory is periodically scanned; identical pages are identified and
> merged.
> The sharing is unnoticeable by the process that use this memory.
> (the shared pages are marked as readonly, and in case of write
> do_wp_page() take care to create new copy of the page)
>
> This driver is very useful for KVM as in cases of runing multiple guests
> operation system of the same type.
> (For desktop work loads we have achived more than x2 memory overcommit
> (more like x3))
>
> This driver have found users other than KVM, for example CERN,
> Fons Rademakers:
> "on many-core machines we run one large detector simulation program per core.
> These simulation programs are identical but run each in their own process and
> need about 2 - 2.5 GB RAM.
> We typically buy machines with 2GB RAM per core and so have a problem to run
> one of these programs per core.
> Of the 2 - 2.5 GB about 700MB is identical data in the form of magnetic field
> maps, detector geometry, etc.
> Currently people have been trying to start one program, initialize the geometry
> and field maps and then fork it N times, to have the data shared.
> With KSM this would be done automatically by the system so it sounded extremely
> attractive when Andrea presented it."
>
> (We have are already started to test KSM on their systems...)
>
> KSM can run as kernel thread or as userspace application or both
>
> example for how to control the kernel thread:
>
> #include <stdio.h>
> #include <stdlib.h>
> #include <string.h>
> #include <sys/types.h>
> #include <sys/stat.h>
> #include <sys/ioctl.h>
> #include <fcntl.h>
> #include <sys/mman.h>
> #include <unistd.h>
> #include "ksm.h"
>
> int main(int argc, char *argv[])
> {
>        int fd;
>        int used = 0;
>        int fd_start;
>        struct ksm_kthread_info info;
>
>
>        if (argc < 2) {
>                fprintf(stderr,
>                        "usage: %s {start npages sleep | stop | info}\n",
>                        argv[0]);
>                exit(1);
>        }
>
>        fd = open("/dev/ksm", O_RDWR | O_TRUNC, (mode_t)0600);
>        if (fd == -1) {
>                fprintf(stderr, "could not open /dev/ksm\n");
>                exit(1);
>        }
>
>        if (!strncmp(argv[1], "start", strlen(argv[1]))) {
>                used = 1;
>                if (argc < 4) {
>                        fprintf(stderr,
>                    "usage: %s start npages_to_scan max_pages_to_merge sleep\n",
>                    argv[0]);
>                        exit(1);
>                }
>                info.pages_to_scan = atoi(argv[2]);
>                info.max_pages_to_merge = atoi(argv[3]);
>                info.sleep = atoi(argv[4]);
>                info.flags = ksm_control_flags_run;
>
>                fd_start = ioctl(fd, KSM_START_STOP_KTHREAD, &info);
>                if (fd_start == -1) {
>                        fprintf(stderr, "KSM_START_KTHREAD failed\n");
>                        exit(1);
>                }
>                printf("created scanner\n");
>        }
>
>        if (!strncmp(argv[1], "stop", strlen(argv[1]))) {
>                used = 1;
>                info.flags = 0;
>                fd_start = ioctl(fd, KSM_START_STOP_KTHREAD, &info);
>                printf("stopped scanner\n");
>        }
>
>        if (!strncmp(argv[1], "info", strlen(argv[1]))) {
>                used = 1;
>                ioctl(fd, KSM_GET_INFO_KTHREAD, &info);
>         printf("flags %d, pages_to_scan %d npages_merge %d, sleep_time %d\n",
>         info.flags, info.pages_to_scan, info.max_pages_to_merge, info.sleep);
>        }
>
>        if (!used)
>                fprintf(stderr, "unknown command %s\n", argv[1]);
>
>        return 0;
> }
>
> example of how to register qemu to ksm (or any userspace application)
>
> diff --git a/qemu/vl.c b/qemu/vl.c
> index 4721fdd..7785bf9 100644
> --- a/qemu/vl.c
> +++ b/qemu/vl.c
> @@ -21,6 +21,7 @@
>  * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
>  * DEALINGS IN
>  * THE SOFTWARE.
>  */
> +#include "ksm.h"
>  #include "hw/hw.h"
>  #include "hw/boards.h"
>  #include "hw/usb.h"
> @@ -5799,6 +5800,37 @@ static void termsig_setup(void)
>
>  #endif
>
> +int ksm_register_memory(void)
> +{
> +    int fd;
> +    int ksm_fd;
> +    int r = 1;
> +    struct ksm_memory_region ksm_region;
> +
> +    fd = open("/dev/ksm", O_RDWR | O_TRUNC, (mode_t)0600);
> +    if (fd == -1)
> +        goto out;
> +
> +    ksm_fd = ioctl(fd, KSM_CREATE_SHARED_MEMORY_AREA);
> +    if (ksm_fd == -1)
> +        goto out_free;
> +
> +    ksm_region.npages = phys_ram_size / TARGET_PAGE_SIZE;
> +    ksm_region.addr = phys_ram_base;
> +    r = ioctl(ksm_fd, KSM_REGISTER_MEMORY_REGION, &ksm_region);
> +    if (r)
> +        goto out_free1;
> +
> +    return r;
> +
> +out_free1:
> +    close(ksm_fd);
> +out_free:
> +    close(fd);
> +out:
> +    return r;
> +}
> +
>  int main(int argc, char **argv)
>  {
>  #ifdef CONFIG_GDBSTUB
> @@ -6735,6 +6767,8 @@ int main(int argc, char **argv)
>     /* init the dynamic translator */
>     cpu_exec_init_all(tb_size * 1024 * 1024);
>
> +    ksm_register_memory();
> +
>     bdrv_init();
>
>     /* we always create the cdrom drive, even if no disk is there */
>
> --
> To unsubscribe from this list: send the line "unsubscribe kvm" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Download attachment "config-2.6.28-rc5.i686" of type "application/octet-stream" (50736 bytes)

Powered by blists - more mailing lists