Date:	Fri, 14 Jan 2011 16:30:51 -0700
From:	xing lin <linxingnku@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: BUG: unable to handle kernel paging request at ffff8800cf669000

Hi,

I wrote a kernel module to calculate the hash for each page frame, in
order to see how many pages are duplicated. It works well in 32-bit
Linux with 2GB physical memory. But when I tried to run this module in
64-bit Linux(2.6.32-24-generic with NUMA enabled) with 12GB physical
memory, it can calculates hashes for the first 63553 in-use pages but
then reports a bug when the page frame number reaches 849513. This
page frame should be in use since the count for the page instance is
1. I do not know why sha_transform results in a page_fault(the data to
be digested is the page which should be in memory). Would anyone give
me some hints? Thanks.

 oops.txt from dmesg:
-----------------------------------
[  590.424038] BUG: unable to handle kernel paging request at ffff8800cf669000
[  590.438705] IP: [<ffffffff8152edc8>] sha_transform+0x18/0x1e0
[  590.452164] PGD 1002063 PUD a067 PMD b067 PTE 0
[  590.464888] Oops: 0000 [#1] SMP
[  590.476077] last sysfs file:
/sys/devices/system/cpu/cpu7/cache/index2/shared_cpu_map
[  590.499137] CPU 7
[  590.508929] Modules linked in: sha1_generic pageinfo_NUMA nfs lockd
nfs_acl auth_rpcgss sunrpc joydev fbcon tileblit dell_wmi dcdbas font
bitblit softcursor power_meter vga16fb vgastate psmouse serio_raw bnx2
lp parport usbhid hid mptsas mptscsih mptbase scsi_transport_sas
[  590.570965] Pid: 1887, comm: cat Not tainted 2.6.32-24-generic
#38+emulab1 PowerEdge R710
[  590.597398] RIP: 0010:[<ffffffff8152edc8>]  [<ffffffff8152edc8>]
sha_transform+0x18/0x1e0
[  590.625098] RSP: 0018:ffff88031eeaba58  EFLAGS: 00010292
[  590.640580] RAX: 0000000000000000 RBX: 0000000000000040 RCX: 0000000000001000
[  590.658272] RDX: ffff88031eeabaa8 RSI: ffff8800cf669000 RDI: ffff88031f45e418
[  590.676439] RBP: ffff88031eeaba88 R08: 0000000000001000 R09: ffff88031eeabd98
[  590.694972] R10: 0000000000000080 R11: 0000000000000000 R12: 0000000000001000
[  590.713897] R13: ffff88031eeabaa8 R14: ffff88031f45e410 R15: ffff8800cf669000
[  590.733149] FS:  00007f2407450700(0000) GS:ffff8800330e0000(0000)
knlGS:0000000000000000
[  590.765186] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  590.783268] CR2: ffff8800cf669000 CR3: 000000031b790000 CR4: 00000000000006e0
[  590.802954] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  590.823136] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  590.843410] Process cat (pid: 1887, threadinfo ffff88031eeaa000,
task ffff88031b5bdbc0)
[  590.878043] Stack:
[  590.893066]  0d4c51d000000018 0000000000000040 0000000000001000
ffff88031eeabaa8
[  590.914054] <0> ffff88031f45e410 ffff8800cf669000 ffff88031eeabc18
ffffffffa00ca19a
[  590.948862] <0> ffff88031eeabc28 ffff88031f45e418 0000000000000000
ffff88031f45e418
[  590.997951] Call Trace:
[  591.013756]  [<ffffffffa00ca19a>] sha1_update+0x7a/0xe0 [sha1_generic]
[  591.033840]  [<ffffffff812ba88c>] ? put_dec+0x10c/0x110
[  591.052640]  [<ffffffff812bab7e>] ? number+0x2ee/0x320
[  591.070975]  [<ffffffff8113783f>] ? __kmalloc+0x1bf/0x1d0
[  591.089383]  [<ffffffff812887fc>] ? crypto_create_tfm+0x3c/0xe0
[  591.107931]  [<ffffffff8128f71f>] ? crypto_init_shash_ops_compat+0x5f/0xe0
[  591.127289]  [<ffffffff8128f188>] crypto_shash_update+0x18/0x30
[  591.145595]  [<ffffffff8128f1de>] shash_compat_update+0x3e/0x60
[  591.163827]  [<ffffffff8128f420>] shash_compat_digest+0x100/0x130
[  591.181974]  [<ffffffffa00a624d>] de_seq_show+0x14d/0x340 [pageinfo_NUMA]
[  591.200520]  [<ffffffff81013b0e>] ? common_interrupt+0xe/0x13
[  591.217877]  [<ffffffffa00a604f>] ? de_seq_next+0x3f/0xd0 [pageinfo_NUMA]
[  591.236138]  [<ffffffff81162eb7>] seq_read+0x267/0x3f0
[  591.252486]  [<ffffffff81162c50>] ? seq_read+0x0/0x3f0
[  591.268687]  [<ffffffff8119b751>] proc_reg_read+0x81/0xc0
[  591.285048]  [<ffffffff81144385>] vfs_read+0xb5/0x1a0
[  591.300730]  [<ffffffff81144541>] sys_read+0x51/0x80
[  591.316125]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b
[  591.332247] Code: 32 10 c7 47 10 f0 e1 d2 c3 c9 c3 0f 1f 84 00 00
00 00 00 55 31 c0 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 08 0f
1f 44 00 00 <8b> 0c 06 0f c9 89 0c 02 48 83 c0 04 48 83 f8 40 75 ee 31
c9 0f
[  591.392058] RIP  [<ffffffff8152edc8>] sha_transform+0x18/0x1e0
[  591.409156]  RSP <ffff88031eeaba58>
[  591.423662] CR2: ffff8800cf669000
[  591.437952] ---[ end trace c858b9629815433b ]---
[  591.453399] note: cat[1887] exited with preempt_count 1
[  591.469249] BUG: scheduling while atomic: cat/1887/0x10000001
[  591.485508] Modules linked in: sha1_generic pageinfo_NUMA nfs lockd
nfs_acl auth_rpcgss sunrpc joydev fbcon tileblit dell_wmi dcdbas font
bitblit softcursor power_meter vga16fb vgastate psmouse serio_raw bnx2
lp parport usbhid hid mptsas mptscsih mptbase scsi_transport_sas
[  591.556226] Pid: 1887, comm: cat Tainted: G      D
2.6.32-24-generic #38+emulab1
[  591.585964] Call Trace:
[  591.599686]  [<ffffffff8105860d>] __schedule_bug+0x7d/0x90
[  591.616501]  [<ffffffff81541f5f>] thread_return+0x1f7/0x418
[  591.633331]  [<ffffffff8106326a>] __cond_resched+0x2a/0x40
[  591.650081]  [<ffffffff81542280>] _cond_resched+0x30/0x40
[  591.666767]  [<ffffffff815434b6>] down_read+0x16/0x30
[  591.683048]  [<ffffffff8106b0fb>] exit_mm+0x3b/0x150
[  591.699185]  [<ffffffff81544325>] ? _spin_lock_irq+0x15/0x20
[  591.715919]  [<ffffffff8106b445>] do_exit+0x125/0x380
[  591.731998]  [<ffffffff81545530>] oops_end+0xb0/0xf0
[  591.748125]  [<ffffffff810408e3>] no_context+0xf3/0x190
[  591.764474]  [<ffffffff81040aa5>] __bad_area_nosemaphore+0x125/0x1e0
[  591.781962]  [<ffffffff81040b73>] bad_area_nosemaphore+0x13/0x20
[  591.799111]  [<ffffffff81547114>] do_page_fault+0x2e4/0x3b0
[  591.815948]  [<ffffffff815448a5>] page_fault+0x25/0x30
[  591.832094]  [<ffffffff8152edc8>] ? sha_transform+0x18/0x1e0
[  591.848585]  [<ffffffffa00ca19a>] sha1_update+0x7a/0xe0 [sha1_generic]
[  591.864986]  [<ffffffff812ba88c>] ? put_dec+0x10c/0x110
[  591.880099]  [<ffffffff812bab7e>] ? number+0x2ee/0x320
[  591.894975]  [<ffffffff8113783f>] ? __kmalloc+0x1bf/0x1d0
[  591.909966]  [<ffffffff812887fc>] ? crypto_create_tfm+0x3c/0xe0
[  591.925493]  [<ffffffff8128f71f>] ? crypto_init_shash_ops_compat+0x5f/0xe0
[  591.942021]  [<ffffffff8128f188>] crypto_shash_update+0x18/0x30
[  591.957478]  [<ffffffff8128f1de>] shash_compat_update+0x3e/0x60
[  591.972784]  [<ffffffff8128f420>] shash_compat_digest+0x100/0x130
[  591.988182]  [<ffffffffa00a624d>] de_seq_show+0x14d/0x340 [pageinfo_NUMA]
[  592.004313]  [<ffffffff81013b0e>] ? common_interrupt+0xe/0x13
[  592.019110]  [<ffffffffa00a604f>] ? de_seq_next+0x3f/0xd0 [pageinfo_NUMA]
[  592.035007]  [<ffffffff81162eb7>] seq_read+0x267/0x3f0
[  592.048918]  [<ffffffff81162c50>] ? seq_read+0x0/0x3f0
[  592.062783]  [<ffffffff8119b751>] proc_reg_read+0x81/0xc0
[  592.076957]  [<ffffffff81144385>] vfs_read+0xb5/0x1a0
[  592.090769]  [<ffffffff81144541>] sys_read+0x51/0x80
[  592.104463]  [<ffffffff810131b2>] system_call_fastpath+0x16/0x1b

my c file:
-----------------------------------------
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/proc_fs.h>
#include <linux/seq_file.h>

#include <linux/mm.h>
#include <linux/mmzone.h>
#include <linux/highmem.h>

#include <linux/crypto.h>
#include <linux/scatterlist.h>
#include <linux/err.h>

#define DRIVER_AUTHOR "utos"
#define DRIVER_DESC   "A driver for memory de-duplication"
#define proc_fn       "pageinfo-NUMA"

//extern struct page *mem_map;
extern unsigned long num_physpages;
//extern unsigned long max_pfn;
static unsigned long pfn = 0;
char debug = 0;

/* This function is called at the beginning of a sequence.
 * ie, when:
 *	- the /proc file is read (first time)
 *	- after the function stop (end of sequence)
 *  pos is the position in /proc/filename.
 */
static void *
de_seq_start(struct seq_file *s, loff_t * pos)
{
	if (debug == 1) {
		printk(KERN_INFO "start: seq_start, pos: %8lu\n",
	    	   *(unsigned long *) pos);
	}

	while( !pfn_valid(*pos) && *pos < num_physpages){
		*pos ++;
	}
	pfn = *(unsigned long *) pos;

	if (pfn < num_physpages) {
		/* begin the sequence. */
		return pfn_to_page(pfn);
	} else {
		printk(KERN_INFO "seq_start: index > num_physpages\n");
		*pos = 0;
		return NULL;
	}
};


/*
 * This function is called after the beginning of a sequence.
 * It's called until the return is NULL (this ends the sequence).
 */
static void *
de_seq_next(struct seq_file *s, void *v, loff_t * pos)
{
	*pos += 1;
	while( !pfn_valid(*pos) && *pos < num_physpages){
		*pos ++;
	}
	pfn = *(unsigned long *) pos;

	if (debug == 1)
		printk(KERN_INFO "seq_next, pfn:%8lu\n", pfn);

	if (pfn >= num_physpages) {
		printk(KERN_INFO "seq_next: pfn >= num_physpages!\n");
		return NULL;
	}
	return pfn_to_page(pfn);
}

/*
 * This function is called for each "step" of a sequence
 */
static int
de_seq_show(struct seq_file *s, void *v)
{
	struct page *page = (struct page *) v;
	void *virt = kmap(page);
	struct scatterlist sg;
	struct crypto_hash *tfm;
	struct hash_desc desc;
	unsigned char result[20];
	if (debug == 1)
		printk(KERN_INFO "seq_show: %8lu\n", pfn);

#ifdef SKIP_UNUSED_PAGES
	if (page_count(page) == 0) {
		if (debug == 1)
			printk(KERN_INFO "skipping page: %8lu\n", pfn);
		return 1;
	}
#endif
	if (virt == NULL) {
		printk(KERN_ALERT "This should not happen in 64-bit machine\n");
		return 1;
	}

	/* get hash of this page */
	tfm = crypto_alloc_hash("sha1", 0, CRYPTO_ALG_ASYNC);
	if (IS_ERR(tfm)) {
		printk(KERN_ALERT "Fail to allocate transformer object!");
		return -EFAULT;
	}
	sg_init_table(&sg, 1);
	sg_set_page(&sg, page, PAGE_SIZE, 0);
	desc.tfm = tfm;
	desc.flags = 0;
	if (crypto_hash_digestsize(tfm) > sizeof (result)) {
		printk("digest size(%u) > outputbuffer(%zu)\n",
		       crypto_hash_digestsize(tfm), sizeof (result));
		return -EFAULT;
	}

	if (crypto_hash_digest(&desc, &sg, PAGE_SIZE, (u8 *) result)) {
		printk(KERN_ALERT "Fail to call digest function!");
		return -EFAULT;
	}
	crypto_free_hash(tfm);
	seq_printf(s, "%8lu: 0x%lx, "
		   "%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x"
		   "%02x%02x%02x%02x%02x%02x%02x%02x%02x%02x,"
		   " %d, %d, %d, %d\n",
		   pfn, (unsigned long) virt,
		   result[0], result[1], result[2], result[3], result[4],
		   result[5], result[6], result[7], result[8], result[9],
		   result[10], result[11], result[12], result[13], result[14],
		   result[15], result[16], result[17], result[18], result[19],
		   page_count(page), PageActive(page),
		   PageReserved(page), PageReclaim(page));

	kunmap(page);

	return 0;
}

/*
 * This function is called at the end of a sequence
 */
static void
de_seq_stop(struct seq_file *s, void *v)
{
	//printk(KERN_INFO "seq_read finished \n");
}

static struct seq_operations de_seq_ops = {
	.start = de_seq_start,
	.next = de_seq_next,
	.stop = de_seq_stop,
	.show = de_seq_show
};

static int
proc_open(struct inode *inode, struct file *file)
{
	return seq_open(file, &de_seq_ops);
};

static struct file_operations file_ops = {
	.owner = THIS_MODULE,
	.open = proc_open,
	.read = seq_read,
	.llseek = seq_lseek,
	.release = seq_release
};

static int __init
lkp_init(void)
{
	struct proc_dir_entry *proc_file = NULL;
	printk(KERN_INFO "Hello from memory de-duplication module for NUMA\n");
	printk("num_physpages: %lu\n", num_physpages);
	proc_file = create_proc_entry(proc_fn, 0644, NULL);
	if (proc_file == NULL) {
		printk(KERN_ALERT "Error: Could not initialize /proc/%s\n",
		       proc_fn);
		return -ENOMEM;
	}
	proc_file->proc_fops = &file_ops;
	printk(KERN_INFO "/proc/%s created\n", proc_fn);
	
	return 0;
}
static void __exit
lkp_cleanup(void)
{
	remove_proc_entry(proc_fn, NULL);
	printk(KERN_INFO "Exit from memory de-duplication module\n");
}

MODULE_LICENSE("GPL");
MODULE_AUTHOR(DRIVER_AUTHOR);
MODULE_DESCRIPTION(DRIVER_DESC);

module_init(lkp_init);
module_exit(lkp_cleanup);

Makefile:
--------------------------------
#
# Modules and the files they are made from
#
PAGEINFO = pageinfo-NUMA
MODULES = $(PAGEINFO)
CFILES = $(MODULES:=.c)
OBJFILES = $(CFILES:.c=.o)
KOFILES = $(OBJFILES:.o=.ko)

obj-m += $(OBJFILES)

# If set, don't report pages that have no users.
EXTRA_CFLAGS=-DSKIP_UNUSED_PAGES

# Make sure to set up dependencies between source and object files
%.o: %.c
%.ko: %.o

KVERSION = $(shell uname -r)
all: $(CFILES)
	make -C /lib/modules/$(KVERSION)/build M=$(PWD) modules
clean:
	make -C /lib/modules/$(KVERSION)/build M=$(PWD) clean

#
# Convience targets
#

# insmod can't be used to install multiple modules at once. So create
# this to install all modules at once.
install: pageinfo_install

pageinfo_install: $(PAGEINFO).ko
	insmod $(PAGEINFO).ko

#singlepage_install: $(SINGLEPAGE).ko
#	insmod $(SINGLEPAGE).ko

deinstall:
	rmmod $(KOFILES)

reinstall: deinstall install

.PHONY: install deinstall reinstall clean



-- 
Regards,
Xing
School of Computing, University of Utah
http://www.cs.utah.edu/~xinglin/
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists