| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <20110504083559.GB25724@elte.hu>
Date: Wed, 4 May 2011 10:35:59 +0200
From: Ingo Molnar <mingo@...e.hu>
To: Linus Torvalds <torvalds@...ux-foundation.org>,
Jens Axboe <axboe@...nel.dk>,
Andrew Morton <akpm@...ux-foundation.org>
Cc: werner <w.landgraf@...ru>, "H. Peter Anvin" <hpa@...or.com>,
Thomas Gleixner <tglx@...utronix.de>,
Linux Kernel Mailing List <linux-kernel@...r.kernel.org>
Subject: [block IO crash] Re: 2.6.39-rc5-git2 boot crashs
* Linus Torvalds <torvalds@...ux-foundation.org> wrote:
> On Tue, May 3, 2011 at 12:08 PM, Ingo Molnar <mingo@...e.hu> wrote:
> >
> > I have no smart ideas straight away - just an observation: i keep testing
> > CONFIG_X86_ELAN=y on real hardware, and it's enabled in about 4% of my
> > configs:
>
> So how often do you do more than just boot?
Not very often - but 'to boot' means a certain amount of filesystem work - and
so does the 'prepare to boot the next kernel' step.
So i took Werner's .config.zipproblem and modified it to make it bootable:
removed CONFIG_ROOT_NFS=y and disabled CONFIG_IDE - both of which keep my box
from booting. I've attached an updated .config.zipproblem2 file: Werner, can
you confirm that this still fails for you?
So i booted v2.6.39-rc5-254-g5933f2a on an AMD box (which is SMP in fact, so
should trigger races even faster):
Kernel 2.6.39-rc5-i486-1sys+ on an i686
and started a couple of such IO-intense loops:
FILE=bigfile.$RANDOM;
while sync; do rm -f $FILE; dd if=/dev/urandom of=$FILE bs=1000 count=10000; done &
this creates patterns of high IO combined with periods waiting for IO to flush:
procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
8 0 0 621140 8968 346500 0 0 29 874 1044 135 0 27 71 2 0
8 0 0 614692 8984 352904 0 0 0 0 2011 658 0 100 0 0 0
8 0 0 608492 8992 359104 0 0 0 0 2096 709 1 99 0 0 0
8 0 0 602016 8992 365332 0 0 0 0 2006 651 0 100 0 0 0
8 0 0 595692 8992 371760 0 0 0 0 2005 647 0 100 0 0 0
4 5 0 588128 9000 378080 0 0 0 31508 2031 610 0 100 0 0 0
6 2 0 583168 9068 383664 0 0 4 7532 2256 668 0 91 0 9 0
8 0 0 586888 9116 380212 0 0 0 456 2073 804 1 100 0 0 0
5 3 0 620484 9408 346904 0 0 0 14932 2768 1901 0 94 1 6 0
8 0 0 633656 9576 333392 0 0 0 6180 2369 1232 0 98 0 2 0
8 0 0 627208 9576 339720 0 0 0 0 2004 650 0 100 0 0 0
8 0 0 621008 9580 346032 0 0 0 0 2048 680 1 100 0 0 0
8 0 0 614436 9588 352296 0 0 0 0 2004 657 0 100 0 0 0
8 0 0 618652 9708 348296 0 0 0 25984 2283 816 0 92 0 7 0
8 0 0 612424 9708 354576 0 0 0 0 2005 651 0 100 0 0 0
8 0 0 605976 9716 360892 0 0 0 0 2004 652 0 100 0 0 0
8 0 0 599652 9716 367208 0 0 0 0 2006 654 0 100 0 0 0
8 0 0 593204 9720 373528 0 0 0 0 2004 652 0 100 0 0 0
7 2 0 585764 9728 379852 0 0 0 31612 2038 649 0 100 0 0 0
8 0 0 590104 9832 376180 0 0 4 7400 2274 920 0 98 0 2 0
2 6 0 597288 10008 368892 0 0 0 8636 2432 1373 0 95 0 5 0
8 0 0 627180 10208 339096 0 0 0 4324 2426 1429 0 97 0 2 0
8 0 0 630404 10284 335480 0 0 0 6408 2202 970 1 97 0 2 0
And indeed, after a couple of minutes testing i triggered this beauty:
BUG: unable to handle kernel NULL pointer dereference at 00000008
IP: [<c14e85c0>] generic_make_request+0x86/0x3f4
*pde = 00000000
Oops: 0000 [#1] SMP
last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/net/eth0/address
Modules linked in:
Pid: 2969, comm: flush-8:0 Not tainted 2.6.39-rc5-i486-1sys+ #122580 System manufacturer System Product Name/A8N-E
EIP: 0060:[<c14e85c0>] EFLAGS: 00010202 CPU: 1
EIP is at generic_make_request+0x86/0x3f4
EAX: 00000000 EBX: f569e280 ECX: f6a00000 EDX: f5528000
ESI: 00000008 EDI: 00000001 EBP: f5fd7c8c ESP: f5fd7c14
DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
Process flush-8:0 (pid: 2969, ti=f5fd6000 task=f64d0ab0 task.ti=f5fd6000)
Stack:
f569e280 c10988af f5fd7c84 c1d74030 f6402400 f569e300 00081ddb f569e280
c10988af f5fd7c84 f64d0ab0 0000007b 0000007b 000000d8 00000008 ffffffc1
00000000 00000000 00000246 f6a06c80 055c08fe 00000000 c10988af 00000080
Call Trace:
[<c10988af>] ? mempool_alloc_slab+0x13/0x15
[<c1d74030>] ? common_interrupt+0x30/0x40
[<c10988af>] ? mempool_alloc_slab+0x13/0x15
[<c10988af>] ? mempool_alloc_slab+0x13/0x15
[<c14e89f9>] submit_bio+0xcb/0xe4
[<c10a8048>] ? inc_zone_page_state+0xe/0x88
[<c10ed8b5>] ? bio_init+0x9/0x2e
[<c10ee05b>] ? bio_alloc_bioset+0x3c/0x9c
[<c10ea27a>] submit_bh+0xc6/0xe0
[<c10eb8f3>] __block_write_full_page+0x20a/0x2df
[<c10ed9d9>] ? bio_put+0x8/0x2c
[<c10ea9e5>] ? end_buffer_async_read+0xd5/0xd5
[<c10eba88>] block_write_full_page_endio+0xc0/0xc8
[<c10ea9e5>] ? end_buffer_async_read+0xd5/0xd5
[<c10ebaa7>] block_write_full_page+0x17/0x19
[<c10ea9e5>] ? end_buffer_async_read+0xd5/0xd5
[<c1156fd0>] ext3_ordered_writepage+0xc8/0x19c
[<c11566ee>] ? bput_one+0x10/0x10
[<c109c9ce>] __writepage+0x10/0x28
[<c109cd96>] write_cache_pages+0x1c9/0x283
[<c109c9be>] ? bdi_set_max_ratio+0x52/0x52
[<c109ce86>] generic_writepages+0x36/0x49
[<c109d994>] do_writepages+0x28/0x2b
[<c10e5642>] writeback_single_inode+0xa6/0x18d
[<c10e58ef>] writeback_sb_inodes+0xa6/0x10b
[<c10e6267>] writeback_inodes_wb+0xd9/0xee
[<c10e642b>] wb_writeback+0x1af/0x26d
[<c1045c12>] ? try_to_del_timer_sync+0x81/0x89
[<c103faca>] ? local_bh_disable+0x8/0x18
[<c10e655a>] wb_do_writeback+0x71/0x181
[<c1045dc0>] ? add_timer_on+0x95/0x95
[<c1045cb1>] ? del_timer+0xc/0x86
[<c10e66d8>] bdi_writeback_thread+0x6e/0x186
[<c10e666a>] ? wb_do_writeback+0x181/0x181
[<c1051f73>] kthread+0x67/0x6c
[<c1051f0c>] ? kthread_worker_fn+0x114/0x114
[<c1d74046>] kernel_thread_helper+0x6/0x10
Code: 00 c7 45 c8 00 00 00 00 8d 55 c8 89 90 f4 02 00 00 89 45 b0 8b 53 20 c1 ea 09 89 55 c0 e8 e3 8c 88 00 83 7d c0 00 74 69 8b 43 0c <8b> 40 08 8b 90 84 00 00 00 f6 c2 01 74 04 f3 90 eb f1 8b 70 7c
EIP: [<c14e85c0>] generic_make_request+0x86/0x3f4 SS:ESP 0068:f5fd7c14
CR2: 0000000000000008
---[ end trace c45e837de578cd2f ]---
------------[ cut here ]------------
( the full crashlog is attached as well - hardware details can be found there
although i doubt it matters. )
Seems to be a generic IO/BDI badness at first sight. It gives me the appearance
of a race or boundary condition bug, not that of memory corruption.
Here's the crashing generic_make_request() function:
c14e853a <generic_make_request>:
c14e853a: 55 push %ebp
c14e853b: 89 e5 mov %esp,%ebp
c14e853d: 57 push %edi
c14e853e: 56 push %esi
c14e853f: 53 push %ebx
c14e8540: 83 ec 6c sub $0x6c,%esp
c14e8543: e8 08 bb 88 00 call c1d74050 <mcount>
c14e8548: 89 c3 mov %eax,%ebx
c14e854a: 65 a1 14 00 00 00 mov %gs:0x14,%eax
c14e8550: 89 45 f0 mov %eax,-0x10(%ebp)
c14e8553: 31 c0 xor %eax,%eax
c14e8555: 64 a1 c4 c4 38 c2 mov %fs:0xc238c4c4,%eax
c14e855b: 83 b8 f4 02 00 00 00 cmpl $0x0,0x2f4(%eax)
c14e8562: 74 23 je c14e8587 <generic_make_request+0x4d>
c14e8564: 8b 80 f4 02 00 00 mov 0x2f4(%eax),%eax
c14e856a: c7 43 08 00 00 00 00 movl $0x0,0x8(%ebx)
c14e8571: 8b 50 04 mov 0x4(%eax),%edx
c14e8574: 85 d2 test %edx,%edx
c14e8576: 74 05 je c14e857d <generic_make_request+0x43>
c14e8578: 89 5a 08 mov %ebx,0x8(%edx)
c14e857b: eb 02 jmp c14e857f <generic_make_request+0x45>
c14e857d: 89 18 mov %ebx,(%eax)
c14e857f: 89 58 04 mov %ebx,0x4(%eax)
c14e8582: e9 8e 03 00 00 jmp c14e8915 <generic_make_request+0x3db>
c14e8587: 83 7b 08 00 cmpl $0x0,0x8(%ebx)
c14e858b: 74 02 je c14e858f <generic_make_request+0x55>
c14e858d: 0f 0b ud2
c14e858f: c7 45 cc 00 00 00 00 movl $0x0,-0x34(%ebp)
c14e8596: c7 45 c8 00 00 00 00 movl $0x0,-0x38(%ebp)
c14e859d: 8d 55 c8 lea -0x38(%ebp),%edx
c14e85a0: 89 90 f4 02 00 00 mov %edx,0x2f4(%eax)
c14e85a6: 89 45 b0 mov %eax,-0x50(%ebp)
c14e85a9: 8b 53 20 mov 0x20(%ebx),%edx
c14e85ac: c1 ea 09 shr $0x9,%edx
c14e85af: 89 55 c0 mov %edx,-0x40(%ebp)
c14e85b2: e8 e3 8c 88 00 call c1d7129a <_cond_resched>
c14e85b7: 83 7d c0 00 cmpl $0x0,-0x40(%ebp)
c14e85bb: 74 69 je c14e8626 <generic_make_request+0xec>
c14e85bd: 8b 43 0c mov 0xc(%ebx),%eax
c14e85c0: 8b 40 08 mov 0x8(%eax),%eax
c14e85c3: 8b 90 84 00 00 00 mov 0x84(%eax),%edx
c14e85c9: f6 c2 01 test $0x1,%dl
c14e85cc: 74 04 je c14e85d2 <generic_make_request+0x98>
c14e85ce: f3 90 pause
c14e85d0: eb f1 jmp c14e85c3 <generic_make_request+0x89>
c14e85d2: 8b 70 7c mov 0x7c(%eax),%esi
c14e85d5: 8b b8 80 00 00 00 mov 0x80(%eax),%edi
c14e85db: 39 90 84 00 00 00 cmp %edx,0x84(%eax)
c14e85e1: 75 e0 jne c14e85c3 <generic_make_request+0x89>
c14e85e3: 89 f0 mov %esi,%eax
c14e85e5: 89 fa mov %edi,%edx
c14e85e7: 0f ac d0 09 shrd $0x9,%edx,%eax
c14e85eb: c1 fa 09 sar $0x9,%edx
c14e85ee: 89 d1 mov %edx,%ecx
c14e85f0: 09 c1 or %eax,%ecx
c14e85f2: 74 32 je c14e8626 <generic_make_request+0xec>
c14e85f4: 8b 0b mov (%ebx),%ecx
c14e85f6: 89 4d c4 mov %ecx,-0x3c(%ebp)
c14e85f9: 8b 4b 04 mov 0x4(%ebx),%ecx
c14e85fc: 8b 75 c0 mov -0x40(%ebp),%esi
c14e85ff: 31 ff xor %edi,%edi
c14e8601: 83 fa 00 cmp $0x0,%edx
c14e8604: 77 09 ja c14e860f <generic_make_request+0xd5>
c14e8606: 3b 45 c0 cmp -0x40(%ebp),%eax
c14e8609: 0f 82 21 02 00 00 jb c14e8830 <generic_make_request+0x2f6>
c14e860f: 29 f0 sub %esi,%eax
c14e8611: 19 fa sbb %edi,%edx
c14e8613: 39 ca cmp %ecx,%edx
c14e8615: 77 0f ja c14e8626 <generic_make_request+0xec>
c14e8617: 0f 82 13 02 00 00 jb c14e8830 <generic_make_request+0x2f6>
c14e861d: 3b 45 c4 cmp -0x3c(%ebp),%eax
c14e8620: 0f 82 0a 02 00 00 jb c14e8830 <generic_make_request+0x2f6>
c14e8626: c7 45 b4 00 00 00 00 movl $0x0,-0x4c(%ebp)
c14e862d: c7 45 b8 ff ff ff ff movl $0xffffffff,-0x48(%ebp)
c14e8634: c7 45 bc ff ff ff ff movl $0xffffffff,-0x44(%ebp)
c14e863b: 8b 45 c0 mov -0x40(%ebp),%eax
c14e863e: 89 45 9c mov %eax,-0x64(%ebp)
c14e8641: c7 45 a0 00 00 00 00 movl $0x0,-0x60(%ebp)
c14e8648: 8b 43 0c mov 0xc(%ebx),%eax
c14e864b: 89 45 98 mov %eax,-0x68(%ebp)
c14e864e: 89 c2 mov %eax,%edx
c14e8650: 8b 40 58 mov 0x58(%eax),%eax
c14e8653: 8b 80 c8 01 00 00 mov 0x1c8(%eax),%eax
c14e8659: 89 45 c4 mov %eax,-0x3c(%ebp)
c14e865c: 85 c0 test %eax,%eax
c14e865e: 75 33 jne c14e8693 <generic_make_request+0x159>
c14e8660: 89 d1 mov %edx,%ecx
c14e8662: 8b 33 mov (%ebx),%esi
c14e8664: 8b 7b 04 mov 0x4(%ebx),%edi
c14e8667: 8d 55 d0 lea -0x30(%ebp),%edx
c14e866a: 89 c8 mov %ecx,%eax
c14e866c: e8 27 8c c2 ff call c1111298 <bdevname>
c14e8671: 89 74 24 08 mov %esi,0x8(%esp)
c14e8675: 89 7c 24 0c mov %edi,0xc(%esp)
c14e8679: 89 44 24 04 mov %eax,0x4(%esp)
c14e867d: c7 04 24 52 b5 01 c2 movl $0xc201b552,(%esp)
c14e8684: e8 79 17 87 00 call c1d59e02 <printk>
c14e8689: ba fb ff ff ff mov $0xfffffffb,%edx
c14e868e: e9 43 02 00 00 jmp c14e88d6 <generic_make_request+0x39c>
c14e8693: f6 43 14 40 testb $0x40,0x14(%ebx)
c14e8697: 75 36 jne c14e86cf <generic_make_request+0x195>
c14e8699: 8b 4d c4 mov -0x3c(%ebp),%ecx
c14e869c: 8b b9 3c 02 00 00 mov 0x23c(%ecx),%edi
c14e86a2: 39 7d c0 cmp %edi,-0x40(%ebp)
c14e86a5: 76 28 jbe c14e86cf <generic_make_request+0x195>
c14e86a7: 8b 4d 98 mov -0x68(%ebp),%ecx
c14e86aa: 8b 73 20 mov 0x20(%ebx),%esi
c14e86ad: c1 ee 09 shr $0x9,%esi
c14e86b0: 8d 55 d0 lea -0x30(%ebp),%edx
c14e86b3: 89 c8 mov %ecx,%eax
c14e86b5: e8 de 8b c2 ff call c1111298 <bdevname>
c14e86ba: 89 7c 24 0c mov %edi,0xc(%esp)
c14e86be: 89 74 24 08 mov %esi,0x8(%esp)
c14e86c2: 89 44 24 04 mov %eax,0x4(%esp)
c14e86c6: c7 04 24 9f b5 01 c2 movl $0xc201b59f,(%esp)
c14e86cd: eb b5 jmp c14e8684 <generic_make_request+0x14a>
c14e86cf: 8b 55 c4 mov -0x3c(%ebp),%edx
c14e86d2: 8b 82 a4 01 00 00 mov 0x1a4(%edx),%eax
c14e86d8: a8 20 test $0x20,%al
c14e86da: 75 ad jne c14e8689 <generic_make_request+0x14f>
c14e86dc: 8b 43 20 mov 0x20(%ebx),%eax
c14e86df: c1 e8 09 shr $0x9,%eax
c14e86e2: 0f 84 80 00 00 00 je c14e8768 <generic_make_request+0x22e>
c14e86e8: 8b 45 98 mov -0x68(%ebp),%eax
c14e86eb: 3b 40 44 cmp 0x44(%eax),%eax
c14e86ee: 74 78 je c14e8768 <generic_make_request+0x22e>
c14e86f0: 8b 50 4c mov 0x4c(%eax),%edx
c14e86f3: 89 55 a8 mov %edx,-0x58(%ebp)
c14e86f6: 8b 32 mov (%edx),%esi
c14e86f8: 8b 7a 04 mov 0x4(%edx),%edi
c14e86fb: 03 33 add (%ebx),%esi
c14e86fd: 13 7b 04 adc 0x4(%ebx),%edi
c14e8700: 89 33 mov %esi,(%ebx)
c14e8702: 89 7b 04 mov %edi,0x4(%ebx)
c14e8705: 8b 40 44 mov 0x44(%eax),%eax
c14e8708: 89 45 a4 mov %eax,-0x5c(%ebp)
c14e870b: 89 43 0c mov %eax,0xc(%ebx)
c14e870e: 89 f2 mov %esi,%edx
c14e8710: 89 f9 mov %edi,%ecx
c14e8712: 8b 45 a8 mov -0x58(%ebp),%eax
c14e8715: 2b 10 sub (%eax),%edx
c14e8717: 1b 48 04 sbb 0x4(%eax),%ecx
c14e871a: 89 55 a8 mov %edx,-0x58(%ebp)
c14e871d: 89 4d ac mov %ecx,-0x54(%ebp)
c14e8720: 8b 45 98 mov -0x68(%ebp),%eax
c14e8723: 8b 38 mov (%eax),%edi
c14e8725: 8b 55 a4 mov -0x5c(%ebp),%edx
c14e8728: 8b 42 58 mov 0x58(%edx),%eax
c14e872b: 8b 80 c8 01 00 00 mov 0x1c8(%eax),%eax
c14e8731: 89 45 a4 mov %eax,-0x5c(%ebp)
c14e8734: e9 00 00 00 00 jmp c14e8739 <generic_make_request+0x1ff>
c14e8739: eb 2d jmp c14e8768 <generic_make_request+0x22e>
c14e873b: 8b 35 84 4e 2a c2 mov 0xc22a4e84,%esi
c14e8741: 85 f6 test %esi,%esi
c14e8743: 74 23 je c14e8768 <generic_make_request+0x22e>
c14e8745: 8b 46 04 mov 0x4(%esi),%eax
c14e8748: 8b 55 a8 mov -0x58(%ebp),%edx
c14e874b: 8b 4d ac mov -0x54(%ebp),%ecx
c14e874e: 89 54 24 04 mov %edx,0x4(%esp)
c14e8752: 89 4c 24 08 mov %ecx,0x8(%esp)
c14e8756: 89 3c 24 mov %edi,(%esp)
c14e8759: 89 d9 mov %ebx,%ecx
c14e875b: 8b 55 a4 mov -0x5c(%ebp),%edx
c14e875e: ff 16 call *(%esi)
c14e8760: 83 c6 08 add $0x8,%esi
c14e8763: 83 3e 00 cmpl $0x0,(%esi)
c14e8766: eb db jmp c14e8743 <generic_make_request+0x209>
c14e8768: 89 d8 mov %ebx,%eax
c14e876a: e8 a1 a2 c0 ff call c10f2a10 <bio_integrity_enabled>
c14e876f: 85 c0 test %eax,%eax
c14e8771: 74 0f je c14e8782 <generic_make_request+0x248>
c14e8773: 89 d8 mov %ebx,%eax
c14e8775: e8 06 a4 c0 ff call c10f2b80 <bio_integrity_prep>
c14e877a: 85 c0 test %eax,%eax
c14e877c: 0f 85 07 ff ff ff jne c14e8689 <generic_make_request+0x14f>
c14e8782: 83 7d bc ff cmpl $0xffffffff,-0x44(%ebp)
c14e8786: 75 06 jne c14e878e <generic_make_request+0x254>
c14e8788: 83 7d b8 ff cmpl $0xffffffff,-0x48(%ebp)
c14e878c: 74 37 je c14e87c5 <generic_make_request+0x28b>
c14e878e: e9 00 00 00 00 jmp c14e8793 <generic_make_request+0x259>
c14e8793: eb 30 jmp c14e87c5 <generic_make_request+0x28b>
c14e8795: 8b 35 84 4e 2a c2 mov 0xc22a4e84,%esi
c14e879b: 85 f6 test %esi,%esi
c14e879d: 74 26 je c14e87c5 <generic_make_request+0x28b>
c14e879f: 8b 46 04 mov 0x4(%esi),%eax
c14e87a2: 8b 55 b8 mov -0x48(%ebp),%edx
c14e87a5: 8b 4d bc mov -0x44(%ebp),%ecx
c14e87a8: 89 54 24 04 mov %edx,0x4(%esp)
c14e87ac: 89 4c 24 08 mov %ecx,0x8(%esp)
c14e87b0: 8b 4d b4 mov -0x4c(%ebp),%ecx
c14e87b3: 89 0c 24 mov %ecx,(%esp)
c14e87b6: 89 d9 mov %ebx,%ecx
c14e87b8: 8b 55 c4 mov -0x3c(%ebp),%edx
c14e87bb: ff 16 call *(%esi)
c14e87bd: 83 c6 08 add $0x8,%esi
c14e87c0: 83 3e 00 cmpl $0x0,(%esi)
c14e87c3: eb d8 jmp c14e879d <generic_make_request+0x263>
c14e87c5: 8b 03 mov (%ebx),%eax
c14e87c7: 8b 53 04 mov 0x4(%ebx),%edx
c14e87ca: 89 45 b8 mov %eax,-0x48(%ebp)
c14e87cd: 89 55 bc mov %edx,-0x44(%ebp)
c14e87d0: 8b 43 0c mov 0xc(%ebx),%eax
c14e87d3: 8b 10 mov (%eax),%edx
c14e87d5: 89 55 b4 mov %edx,-0x4c(%ebp)
c14e87d8: 83 7d c0 00 cmpl $0x0,-0x40(%ebp)
c14e87dc: 74 5e je c14e883c <generic_make_request+0x302>
c14e87de: 8b 50 08 mov 0x8(%eax),%edx
c14e87e1: 8b 82 84 00 00 00 mov 0x84(%edx),%eax
c14e87e7: a8 01 test $0x1,%al
c14e87e9: 74 04 je c14e87ef <generic_make_request+0x2b5>
c14e87eb: f3 90 pause
c14e87ed: eb f2 jmp c14e87e1 <generic_make_request+0x2a7>
c14e87ef: 8b 72 7c mov 0x7c(%edx),%esi
c14e87f2: 8b ba 80 00 00 00 mov 0x80(%edx),%edi
c14e87f8: 39 82 84 00 00 00 cmp %eax,0x84(%edx)
c14e87fe: 75 e1 jne c14e87e1 <generic_make_request+0x2a7>
c14e8800: 89 f0 mov %esi,%eax
c14e8802: 89 fa mov %edi,%edx
c14e8804: 0f ac d0 09 shrd $0x9,%edx,%eax
c14e8808: c1 fa 09 sar $0x9,%edx
c14e880b: 89 d1 mov %edx,%ecx
c14e880d: 09 c1 or %eax,%ecx
c14e880f: 74 2b je c14e883c <generic_make_request+0x302>
c14e8811: 8b 33 mov (%ebx),%esi
c14e8813: 8b 4b 04 mov 0x4(%ebx),%ecx
c14e8816: 83 fa 00 cmp $0x0,%edx
c14e8819: 77 05 ja c14e8820 <generic_make_request+0x2e6>
c14e881b: 3b 45 c0 cmp -0x40(%ebp),%eax
c14e881e: 72 10 jb c14e8830 <generic_make_request+0x2f6>
c14e8820: 2b 45 9c sub -0x64(%ebp),%eax
c14e8823: 1b 55 a0 sbb -0x60(%ebp),%edx
c14e8826: 39 ca cmp %ecx,%edx
c14e8828: 77 12 ja c14e883c <generic_make_request+0x302>
c14e882a: 72 04 jb c14e8830 <generic_make_request+0x2f6>
c14e882c: 39 f0 cmp %esi,%eax
c14e882e: 73 0c jae c14e883c <generic_make_request+0x302>
c14e8830: 89 d8 mov %ebx,%eax
c14e8832: e8 87 e7 ff ff call c14e6fbe <handle_bad_sector>
c14e8837: e9 4d fe ff ff jmp c14e8689 <generic_make_request+0x14f>
c14e883c: 8b 43 14 mov 0x14(%ebx),%eax
c14e883f: a9 00 10 80 00 test $0x801000,%eax
c14e8844: 74 1a je c14e8860 <generic_make_request+0x326>
c14e8846: 8b 55 c4 mov -0x3c(%ebp),%edx
c14e8849: 83 ba 7c 02 00 00 00 cmpl $0x0,0x27c(%edx)
c14e8850: 75 0e jne c14e8860 <generic_make_request+0x326>
c14e8852: 25 ff ef 7f ff and $0xff7fefff,%eax
c14e8857: 89 43 14 mov %eax,0x14(%ebx)
c14e885a: 83 7d c0 00 cmpl $0x0,-0x40(%ebp)
c14e885e: 74 6d je c14e88cd <generic_make_request+0x393>
c14e8860: 8b 43 14 mov 0x14(%ebx),%eax
c14e8863: a8 40 test $0x40,%al
c14e8865: 74 2d je c14e8894 <generic_make_request+0x35a>
c14e8867: 8b 4d c4 mov -0x3c(%ebp),%ecx
c14e886a: 8b 91 a4 01 00 00 mov 0x1a4(%ecx),%edx
c14e8870: 80 e6 40 and $0x40,%dh
c14e8873: 74 5c je c14e88d1 <generic_make_request+0x397>
c14e8875: a9 00 00 00 08 test $0x8000000,%eax
c14e887a: 74 18 je c14e8894 <generic_make_request+0x35a>
c14e887c: 8b 81 a4 01 00 00 mov 0x1a4(%ecx),%eax
c14e8882: f6 c4 40 test $0x40,%ah
c14e8885: 74 4a je c14e88d1 <generic_make_request+0x397>
c14e8887: 8b 81 a4 01 00 00 mov 0x1a4(%ecx),%eax
c14e888d: a9 00 00 02 00 test $0x20000,%eax
c14e8892: 74 3d je c14e88d1 <generic_make_request+0x397>
c14e8894: 85 db test %ebx,%ebx
c14e8896: 74 45 je c14e88dd <generic_make_request+0x3a3>
c14e8898: e9 00 00 00 00 jmp c14e889d <generic_make_request+0x363>
c14e889d: eb 1c jmp c14e88bb <generic_make_request+0x381>
c14e889f: 8b 35 fc 4e 2a c2 mov 0xc22a4efc,%esi
c14e88a5: 85 f6 test %esi,%esi
c14e88a7: 74 12 je c14e88bb <generic_make_request+0x381>
c14e88a9: 8b 46 04 mov 0x4(%esi),%eax
c14e88ac: 89 d9 mov %ebx,%ecx
c14e88ae: 8b 55 c4 mov -0x3c(%ebp),%edx
c14e88b1: ff 16 call *(%esi)
c14e88b3: 83 c6 08 add $0x8,%esi
c14e88b6: 83 3e 00 cmpl $0x0,(%esi)
c14e88b9: eb ec jmp c14e88a7 <generic_make_request+0x36d>
c14e88bb: 89 da mov %ebx,%edx
c14e88bd: 8b 45 c4 mov -0x3c(%ebp),%eax
c14e88c0: ff 50 44 call *0x44(%eax)
c14e88c3: 85 c0 test %eax,%eax
c14e88c5: 0f 85 7d fd ff ff jne c14e8648 <generic_make_request+0x10e>
c14e88cb: eb 10 jmp c14e88dd <generic_make_request+0x3a3>
c14e88cd: 31 d2 xor %edx,%edx
c14e88cf: eb 05 jmp c14e88d6 <generic_make_request+0x39c>
c14e88d1: ba a1 ff ff ff mov $0xffffffa1,%edx
c14e88d6: 89 d8 mov %ebx,%eax
c14e88d8: e8 b8 49 c0 ff call c10ed295 <bio_endio>
c14e88dd: 8b 55 b0 mov -0x50(%ebp),%edx
c14e88e0: 8b 82 f4 02 00 00 mov 0x2f4(%edx),%eax
c14e88e6: 8b 18 mov (%eax),%ebx
c14e88e8: 85 db test %ebx,%ebx
c14e88ea: 74 1c je c14e8908 <generic_make_request+0x3ce>
c14e88ec: 8b 53 08 mov 0x8(%ebx),%edx
c14e88ef: 89 10 mov %edx,(%eax)
c14e88f1: 85 d2 test %edx,%edx
c14e88f3: 75 07 jne c14e88fc <generic_make_request+0x3c2>
c14e88f5: c7 40 04 00 00 00 00 movl $0x0,0x4(%eax)
c14e88fc: c7 43 08 00 00 00 00 movl $0x0,0x8(%ebx)
c14e8903: e9 a1 fc ff ff jmp c14e85a9 <generic_make_request+0x6f>
c14e8908: 8b 4d b0 mov -0x50(%ebp),%ecx
c14e890b: c7 81 f4 02 00 00 00 movl $0x0,0x2f4(%ecx)
c14e8912: 00 00 00
c14e8915: 8b 45 f0 mov -0x10(%ebp),%eax
c14e8918: 65 33 05 14 00 00 00 xor %gs:0x14,%eax
c14e891f: 74 05 je c14e8926 <generic_make_request+0x3ec>
c14e8921: e8 56 27 b5 ff call c103b07c <__stack_chk_fail>
c14e8926: 83 c4 6c add $0x6c,%esp
c14e8929: 5b pop %ebx
c14e892a: 5e pop %esi
c14e892b: 5f pop %edi
c14e892c: 5d pop %ebp
c14e892d: c3 ret
The crash is at:
c14e85ac: c1 ea 09 shr $0x9,%edx
c14e85af: 89 55 c0 mov %edx,-0x40(%ebp)
c14e85b2: e8 e3 8c 88 00 call c1d7129a <_cond_resched>
c14e85b7: 83 7d c0 00 cmpl $0x0,-0x40(%ebp)
c14e85bb: 74 69 je c14e8626 <generic_make_request+0xec>
c14e85bd: 8b 43 0c mov 0xc(%ebx),%eax
* c14e85c0: 8b 40 08 mov 0x8(%eax),%eax <==== [ **CRASH** ]
c14e85c3: 8b 90 84 00 00 00 mov 0x84(%eax),%edx
c14e85c9: f6 c2 01 test $0x1,%dl
which corresponds to:
1414
1415 if (!nr_sectors)
1416 return 0;
1417
1418 /* Test device or partition size, when known. */
1419 maxsector = i_size_read(bio->bi_bdev->bd_inode) >> 9; <==== [ **CRASH** ]
1420 if (maxsector) {
1421 sector_t sector = bio->bi_sector;
1422
1423 if (maxsector < nr_sectors || maxsector - nr_sectors < sector) {
bio->bi_bdev has become NULL?
I do not think the _cond_resched() was called, judging from stack contents. But
we just had an IRQ:
[<c1d74030>] ? common_interrupt+0x30/0x40
So we might have raced with block IO IRQ queue-completion/submission activites.
But maybe it was a reschedule after all, just the stack does not carry any
traces of it anymore. IRQs do not clear ->bi_bdev, right? Unless the bio
refcounts are wrong and an IRQ's completion actually frees the bio, right?
I've built a CONFIG_DEBUG_PAGEALLOC=y and CONFIG_SLUB_DEBUG=y kernel, maybe the
crash triggers in a more revealing way.
Thanks,
Ingo
View attachment "config.zipproblem2" of type "text/plain" (130545 bytes)
View attachment "crash.log" of type "text/plain" (205089 bytes)
Powered by blists - more mailing lists