| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <5072A8AD.8040708@cn.fujitsu.com> Date: Mon, 08 Oct 2012 18:19:25 +0800 From: Wen Congyang <wency@...fujitsu.com> To: Vasilis Liaskovitis <vasilis.liaskovitis@...fitbricks.com> CC: x86@...nel.org, linux-mm@...ck.org, lkml <linux-kernel@...r.kernel.org>, linux-acpi@...r.kernel.org, minchan.kim@...il.com, akpm@...ux-foundation.org, isimatu.yasuaki@...fujitsu.com Subject: Re: [RFC v9 PATCH 00/21] memory-hotplug: hot-remove physical memory At 09/27/2012 12:46 AM, Vasilis Liaskovitis Wrote: > Hi, > > I am testing 3.6.0-rc7 with this v9 patchset plus more recent fixes [1],[2],[3] > Running in a guest (qemu+seabios from [4]). > CONFIG_SLAB=y > CONFIG_DEBUG_SLAB=y > > After succesfull hot-add and online, I am doing a hot-remove with "echo 1 > /sys/bus/acpi/devices/PNP/eject" > When I do the OSPM-eject, I often get slab corruption in "acpi-state" cache, or in other caches I can reproduce this problem without my patchset. Thanks Wen Congyang > > [ 170.566995] Slab corruption (Not tainted): Acpi-State start=ffff88009fc1e548, len=80 > [ 170.567265] Redzone: 0x0/0x0. > [ 170.567399] Last user: [< (null)>](0x0) > [ 170.567667] 000: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [ 170.568078] 010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [ 170.568487] 020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [ 170.568894] 030: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [ 170.569302] 040: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ > [ 170.569712] Prev obj: start=000000009fc1e4d0, len=80 > [ 170.569869] BUG: unable to handle kernel paging request at 000000009fc1e520 > [ 170.570171] IP: [<ffffffff8112803c>] print_objinfo+0x9c/0x110 > [ 170.570397] PGD 7cf37067 PUD 0 > [ 170.570619] Oops: 0000 [#1] SMP > [ 170.570843] Modules linked in: netconsole acpiphp pci_hotplug acpi_memhotplug loop kvm_amd kvm tpm_tis microcode tpm tpm_bios psmouse parport_pc serio_raw evdev parport i2c_piix4 processor thermal_sys i2c_core button ext3 jbd mbcache sg sr_mod cdrom ata_generic virtio_net virtio_blk ata_piix libata scsi_mod virtio_pci virtio_ring virtio > [ 170.573474] CPU 0 > [ 170.573568] Pid: 29, comm: kworker/0:1 Not tainted 3.6.0-rc7-guest #12 Bochs Bochs > [ 170.573830] RIP: 0010:[<ffffffff8112803c>] [<ffffffff8112803c>] print_objinfo+0x9c/0x110 > [ 170.574106] RSP: 0018:ffff88003eaf3a70 EFLAGS: 00010202 > [ 170.574268] RAX: 000000009fc1e4c8 RBX: 0000000000000002 RCX: 00000000000024b8 > [ 170.574468] RDX: 000000009fc1e4c8 RSI: 000000009fc1e4c8 RDI: ffff88003e9bb980 > [ 170.574668] RBP: ffff88003e9bb980 R08: ffff880037964078 R09: 0000000000000000 > [ 170.574870] R10: 000000000000021e R11: 0000000000000002 R12: 000000009fc1e4c8 > [ 170.575070] R13: 000000009fc1e520 R14: 000000000000004f R15: 00000000ffffffa5 > [ 170.575274] FS: 00007fc6b7530700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000 > [ 170.575494] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 170.575665] CR2: 000000009fc1e520 CR3: 000000007c9c1000 CR4: 00000000000006f0 > [ 170.575870] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 170.576075] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 170.576276] Process kworker/0:1 (pid: 29, threadinfo ffff88003eaf2000, task ffff88003ea941c0) > [ 170.576507] Stack: > [ 170.576599] 0000000000000010 0000000001893fbe ffff88009fc1e000 0000000000000050 > [ 170.576938] 000000009fc1e4c8 000000000000004f 00000000ffffffa5 ffffffff8112899f > [ 170.576938] ffff88003eb309d8 ffffffff81712d6d ffff88003e9bb980 ffff88009fc1e540 > [ 170.576938] Call Trace: > [ 170.576938] [<ffffffff8112899f>] ? check_poison_obj+0x1df/0x1f0 > [ 170.576938] [<ffffffff813047d0>] ? acpi_ut_create_generic_state+0x2f/0x4c > [ 170.576938] [<ffffffff813047d0>] ? acpi_ut_create_generic_state+0x2f/0x4c > [ 170.576938] [<ffffffff81128a9d>] ? cache_alloc_debugcheck_after.isra.52+0xed/0x220 > [ 170.576938] [<ffffffff813047d0>] ? acpi_ut_create_generic_state+0x2f/0x4c > [ 170.576938] [<ffffffff8112beb5>] ? kmem_cache_alloc+0xb5/0x1e0 > [ 170.576938] [<ffffffff813047d0>] ? acpi_ut_create_generic_state+0x2f/0x4c > [ 170.576938] [<ffffffff812edf2d>] ? acpi_ds_result_push+0x5d/0x12e > [ 170.576938] [<ffffffff812ed127>] ? acpi_ds_exec_end_op+0x28e/0x3d3 > [ 170.576938] [<ffffffff812fd86a>] ? acpi_ps_parse_loop+0x79f/0x931 > [ 170.576938] [<ffffffff812fdd6c>] ? acpi_ps_parse_aml+0x89/0x261 > [ 170.576938] [<ffffffff812fe50c>] ? acpi_ps_execute_method+0x1be/0x266 > [ 170.576938] [<ffffffff812f91f7>] ? acpi_ns_evaluate+0xd3/0x19a > [ 170.576938] [<ffffffff812fb93e>] ? acpi_evaluate_object+0xf3/0x1f4 > [ 170.576938] [<ffffffff812e1104>] ? acpi_os_wait_events_complete+0x1b/0x1b > [ 170.576938] [<ffffffff812e4782>] ? acpi_bus_hot_remove_device+0xeb/0x123 > [ 170.576938] [<ffffffff812e1121>] ? acpi_os_execute_deferred+0x1d/0x29 > [ 170.576938] [<ffffffff81058ec5>] ? process_one_work+0x125/0x560 > [ 170.576938] [<ffffffff81059e7a>] ? worker_thread+0x16a/0x4e0 > [ 170.576938] [<ffffffff81059d10>] ? manage_workers+0x310/0x310 > [ 170.576938] [<ffffffff8105e6c5>] ? kthread+0x85/0x90 > [ 170.576938] [<ffffffff814eb2c4>] ? kernel_thread_helper+0x4/0x10 > [ 170.576938] [<ffffffff8105e640>] ? flush_kthread_worker+0xa0/0xa0 > [ 170.576938] [<ffffffff814eb2c0>] ? gs_change+0x13/0x13 > [ 170.576938] Code: cb 75 dc 48 83 c4 08 5b 5d 41 5c 41 5d 41 5e 41 5f c3 8b 7f 0c 4c 89 e2 e8 02 fd ff ff 4c 89 e6 49 89 c5 48 89 ef e8 d4 fc ff ff <49> 8b 55 00 48 8b 30 48 c7 c7 8c 39 6f 81 31 c0 e8 3e 34 3b 00 > > Other times, the problem happens on a slab object free: > > [ 52.313366] Offlined Pages 32768 > [ 52.800232] slab error in verify_redzone_free(): cache `Acpi-ParseExt': memory outside object was overwritten > [ 52.801298] Pid: 29, comm: kworker/0:1 Not tainted 3.6.0-rc7-guest #12 > [ 52.802039] Call Trace: > [ 52.802443] [<ffffffff811280cb>] ? __slab_error.isra.46+0x1b/0x30 > [ 52.803199] [<ffffffff811287b6>] ? cache_free_debugcheck+0x256/0x260 > [ 52.803940] [<ffffffff812e1b0e>] ? acpi_os_release_object+0x7/0xc > [ 52.804645] [<ffffffff81128fe3>] ? kmem_cache_free+0x63/0x260 > [ 52.805321] [<ffffffff812e1b0e>] ? acpi_os_release_object+0x7/0xc > [ 52.806023] [<ffffffff812fe298>] ? acpi_ps_delete_parse_tree+0x34/0x58 > [ 52.806762] [<ffffffff812fe517>] ? acpi_ps_execute_method+0x1c9/0x266 > [ 52.807499] [<ffffffff812f91f7>] ? acpi_ns_evaluate+0xd3/0x19a > [ 52.808183] [<ffffffff812fb93e>] ? acpi_evaluate_object+0xf3/0x1f4 > [ 52.808897] [<ffffffff812e1104>] ? acpi_os_wait_events_complete+0x1b/0x1b > [ 52.809659] [<ffffffff812e4782>] ? acpi_bus_hot_remove_device+0xeb/0x123 > [ 52.810032] [<ffffffff812e1121>] ? acpi_os_execute_deferred+0x1d/0x29 > [ 52.810032] [<ffffffff81058ec5>] ? process_one_work+0x125/0x560 > [ 52.810032] [<ffffffff81059e7a>] ? worker_thread+0x16a/0x4e0 > [ 52.810032] [<ffffffff81059d10>] ? manage_workers+0x310/0x310 > [ 52.810032] [<ffffffff8105e6c5>] ? kthread+0x85/0x90 > [ 52.810032] [<ffffffff814eb2c4>] ? kernel_thread_helper+0x4/0x10 > [ 52.810032] [<ffffffff8105e640>] ? flush_kthread_worker+0xa0/0xa0 > [ 52.810032] [<ffffffff814eb2c0>] ? gs_change+0x13/0x13 > [ 52.810032] ffff88008f809670: redzone 1:0x0, redzone 2:0x0. > [ 52.810032] ------------[ cut here ]------------ > [ 52.810032] kernel BUG at mm/slab.c:3125! > [ 52.810032] invalid opcode: 0000 [#1] SMP > [ 52.810032] Modules linked in: netconsole acpiphp pci_hotplug acpi_memhotplug loop kvm_amd kvm tpm_tis tpm tpm_bios microcode parport_pc parport evdev processor thermal_sys psmouse i2c_piix4 serio_raw i2c_core button ext3 jbd mbcache sg sr_mod cdrom virtio_net ata_generic virtio_blk virtio_pci virtio_ring virtio ata_piix libata scsi_mod > [ 52.810032] CPU 0 > [ 52.810032] Pid: 29, comm: kworker/0:1 Not tainted 3.6.0-rc7-guest #12 Bochs Bochs > [ 52.810032] RIP: 0010:[<ffffffff81128733>] [<ffffffff81128733>] cache_free_debugcheck+0x1d3/0x260 > [ 52.810032] RSP: 0018:ffff88003eaf3bc0 EFLAGS: 00010093 > [ 52.810032] RAX: 00000000017eac3c RBX: ffff88003e9bb700 RCX: 0000000002aaaaab > [ 52.810032] RDX: 0000000000000000 RSI: 0000000000010000 RDI: 0000000000000060 > [ 52.810032] RBP: ffff88008f809670 R08: 09f911029d74e35b R09: 0000000000000000 > [ 52.810032] R10: 00000000000001d3 R11: 0000000000000002 R12: ffff88008f809000 > [ 52.810032] R13: ffffffff812e1b0e R14: 0000000000000000 R15: 0000000000010c00 > [ 52.810032] FS: 00007f63fc263700(0000) GS:ffff88003fc00000(0000) knlGS:0000000000000000 > [ 52.810032] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > [ 52.810032] CR2: 00007fff8d895b78 CR3: 000000007c866000 CR4: 00000000000006f0 > [ 52.810032] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 52.810032] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 52.810032] Process kworker/0:1 (pid: 29, threadinfo ffff88003eaf2000, task ffff88003ea941c0) > [ 52.810032] Stack: > [ 52.810032] ffff88003e9bb980 ffff88008f809670 ffff880037ba8e18 ffff88008f809678 > [ 52.810032] ffff88003e9bb700 0000000000000282 ffff88003e9bf270 ffffffff812e1b0e > [ 52.810032] 0000000000000001 ffffffff81128fe3 ffff88003e80f5e8 ffff88003eb31748 > [ 52.810032] Call Trace: > [ 52.810032] [<ffffffff812e1b0e>] ? acpi_os_release_object+0x7/0xc > [ 52.810032] [<ffffffff81128fe3>] ? kmem_cache_free+0x63/0x260 > [ 52.810032] [<ffffffff812e1b0e>] ? acpi_os_release_object+0x7/0xc > [ 52.810032] [<ffffffff812fe298>] ? acpi_ps_delete_parse_tree+0x34/0x58 > [ 52.810032] [<ffffffff812fe517>] ? acpi_ps_execute_method+0x1c9/0x266 > [ 52.810032] [<ffffffff812f91f7>] ? acpi_ns_evaluate+0xd3/0x19a > [ 52.810032] [<ffffffff812fb93e>] ? acpi_evaluate_object+0xf3/0x1f4 > [ 52.810032] [<ffffffff812e1104>] ? acpi_os_wait_events_complete+0x1b/0x1b > [ 52.810032] [<ffffffff812e4782>] ? acpi_bus_hot_remove_device+0xeb/0x123 > [ 52.810032] [<ffffffff812e1121>] ? acpi_os_execute_deferred+0x1d/0x29 > [ 52.810032] [<ffffffff81058ec5>] ? process_one_work+0x125/0x560 > [ 52.810032] [<ffffffff81059e7a>] ? worker_thread+0x16a/0x4e0 > [ 52.810032] [<ffffffff81059d10>] ? manage_workers+0x310/0x310 > [ 52.810032] [<ffffffff8105e6c5>] ? kthread+0x85/0x90 > [ 52.810032] [<ffffffff814eb2c4>] ? kernel_thread_helper+0x4/0x10 > [ 52.810032] [<ffffffff8105e640>] ? flush_kthread_worker+0xa0/0xa0 > [ 52.810032] [<ffffffff814eb2c0>] ? gs_change+0x13/0x13 > [ 52.810032] Code: 89 ea 49 89 38 8b 73 14 8b 7b 0c e8 18 f6 ff ff 49 b8 5b e3 74 9d 02 11 f9 09 4c 89 00 44 8b 7b 14 44 89 f8 e9 fa fe ff ff 0f 0b <0f> 0b 48 8b 40 30 e9 d9 fe ff ff e8 c6 43 3b 00 0f 0b 48 8b 40 > [ 52.810032] RIP [<ffffffff81128733>] cache_free_debugcheck+0x1d3/0x260 > [ 52.810032] RSP <ffff88003eaf3bc0> > [ 52.810032] ---[ end trace c699c8cecd5870a3 ]--- > > > And other times, I see a filesystem related slab corruption when doing the eject. > Here I have also seen a bad rss-counter state message. > > [ 232.114232] BUG: Bad rss-counter state mm:ffff88007d9c1f80 idx:0 val:1 > [ 232.115214] BUG: unable to handle kernel NULL pointer dereference at (null) > [ 232.115807] IP: [<ffffffffa00cfce3>] do_get_write_access+0x43/0x480 [jbd] > [ 232.116186] PGD 7cdec067 PUD 7c946067 PMD 0 > [ 232.116627] Oops: 0000 [#3] SMP > [ 232.116990] Modules linked in: netconsole acpiphp pci_hotplug acpi_memhotplug loop kvm_amd kvm microcode tpm_tis tpm tpm_bios evdev psmouse serio_raw i2c_piix4 i2c_core parport_pc parport processor thermal_sys button ext3 jbd mbcache virtio_net sg sr_mod cdrom virtio_blk ata_generic virtio_pci virtio_ring virtio ata_piix libata scsi_mod > [ 232.120013] CPU 2 > [ 232.120013] Pid: 880, comm: dhclient Tainted: G D 3.6.0-rc7-guest #1 Bochs Bochs > [ 232.120013] RIP: 0010:[<ffffffffa00cfce3>] [<ffffffffa00cfce3>] do_get_write_access+0x43/0x480 [jbd] > [ 232.120013] RSP: 0018:ffff88007cd6fab8 EFLAGS: 00010246 > [ 232.120013] RAX: ffff88003e513f50 RBX: ffff88003e513f50 RCX: 0000000000000000 > [ 232.120013] RDX: 0000000000000000 RSI: ffff88003e513f50 RDI: ffff880097800000 > [ 232.120013] RBP: ffff88003e513f50 R08: 000000003e513f01 R09: 0000000180240024 > [ 232.120013] R10: ffff88003e513f50 R11: 00000000000198e0 R12: 0000000000000000 > [ 232.120013] R13: ffffffffa00eb437 R14: ffff880097800000 R15: 000000000000027a > [ 232.120013] FS: 00007f082638d700(0000) GS:ffff88003ec80000(0000) knlGS:0000000000000000 > [ 232.120013] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 232.120013] CR2: 0000000000000000 CR3: 000000007c94c000 CR4: 00000000000006e0 > [ 232.120013] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 > [ 232.120013] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 > [ 232.120013] Process dhclient (pid: 880, threadinfo ffff88007cd6e000, task ffff88007ca46800) > [ 232.120013] Stack: > [ 232.120013] ffff88007dc205b0 ffffffff8116645f ffff88007dc205b0 ffff88007dc205b0 > [ 232.120013] ffff88007cf58800 0000000000000000 000000000000000e 0000000000000000 > [ 232.120013] 000000000000027a ffffffff8112b934 0000005000000003 0000000000000fff > [ 232.120013] Call Trace: > [ 232.120013] [<ffffffff8116645f>] ? __find_get_block+0x7f/0x200 > [ 232.120013] [<ffffffff8112b934>] ? kmem_cache_alloc+0xe4/0x140 > [ 232.120013] [<ffffffffa00eb437>] ? ext3_dirty_inode+0x57/0xb0 [ext3] > [ 232.120013] [<ffffffffa00d0279>] ? journal_get_write_access+0x29/0x50 [jbd] > [ 232.120013] [<ffffffffa00eaeef>] ? __ext3_get_inode_loc+0xcf/0x360 [ext3] > [ 232.120013] [<ffffffffa0101317>] ? __ext3_journal_get_write_access+0x27/0x60 [ext3] > [ 232.120013] [<ffffffffa00eb213>] ? ext3_reserve_inode_write+0x73/0xa0 [ext3] > [ 232.120013] [<ffffffffa00eb27b>] ? ext3_mark_inode_dirty+0x3b/0xa0 [ext3] > [ 232.120013] [<ffffffffa00eb437>] ? ext3_dirty_inode+0x57/0xb0 [ext3] > [ 232.120013] [<ffffffff8115ded6>] ? __mark_inode_dirty+0x36/0x230 > [ 232.120013] [<ffffffff811504a1>] ? update_time+0x71/0xb0 > [ 232.120013] [<ffffffff811536c9>] ? mnt_clone_write+0x9/0x20 > [ 232.120013] [<ffffffff81150581>] ? file_update_time+0xa1/0xf0 > [ 232.120013] [<ffffffff8103385c>] ? ptep_set_access_flags+0x6c/0x70 > [ 232.120013] [<ffffffff810e3f30>] ? __generic_file_aio_write+0x1a0/0x3c0 > [ 232.120013] [<ffffffff811396fb>] ? __sb_start_write+0x6b/0x130 > [ 232.120013] [<ffffffff810e41ce>] ? generic_file_aio_write+0x7e/0x100 > [ 232.120013] [<ffffffff81137484>] ? do_sync_write+0x94/0xd0 > [ 232.120013] [<ffffffff81137caa>] ? vfs_write+0xaa/0x160 > [ 232.120013] [<ffffffff81137f87>] ? sys_write+0x47/0x90 > [ 232.120013] [<ffffffff814e5765>] ? async_page_fault+0x25/0x30 > [ 232.120013] [<ffffffff814eca79>] ? system_call_fastpath+0x16/0x1b > [ 232.120013] Code: 54 24 2c f6 47 14 04 74 1f 41 bc e2 ff ff ff 48 81 c4 98 00 00 00 44 89 e0 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 4c 8b 27 <4d> 8b 3c 24 41 f6 07 02 75 d4 65 48 8b 04 25 80 b9 00 00 48 89 > > Is this a known issue? If yes, can you point me to any relevant patches? > > When I do an SCI(hardware) eject, I have not seen corruptions. The acpi driver is > evaluating fewer objects in this path I think, but I don't see why corruption can't > happen here as well. > > I have seen similar problems with CONFIG_SLUB (I did not have debug_options > there but I can also provide those if helpful) > > [1] https://lkml.org/lkml/2012/9/6/635 > [2] https://lkml.org/lkml/2012/9/11/542 > [3] https://lkml.org/lkml/2012/9/20/37 > [4] http://permalink.gmane.org/gmane.comp.emulators.kvm.devel/98691 > > thanks, > > - Vasilis > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@...r.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists