| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4AF0DC81.10305@intel.com> Date: Tue, 03 Nov 2009 17:44:33 -0800 From: Alexander Duyck <alexander.h.duyck@...el.com> To: "netdev@...r.kernel.org" <netdev@...r.kernel.org> CC: "Tantilov, Emil S" <emil.s.tantilov@...el.com>, "davem@...emloft.net" <davem@...emloft.net>, dwmw2@...radead.org Subject: kernel WARNING/BUG with IOMMU + skb_dma_map/unmap + pktgen It looks like skb_dma_map/unmap plus trying to transmit multiple copies of the same skb via pktgen doesn't work when a hardware IOMMU is enabled. We first saw this on igb doing a loopback test partially due to the fact that I had copied the approach used by pktgen of just incrementing the user count and sending the same skb into the transmit path multiple times. However we did some testing to verify that this is reproducible on other NICs using skb_dma_map/unmap and pktgen and generated the traces below on a tg3 w/ pktgen configured to clone_skb several times. The issue seems to be due to the fact that skb_dma_map assumes that dma_map_single/page should return the same value every time it is called, however in the case of having a hardware iommu enabled this is not so and as a result we end up leaking dma mappings and then attempt multiple unmaps on the last dma mapping we received. I'm wondering if this is a valid issue since I am not sure how many different situations will result in a skb being cloned and sent either to the same device multiple times or to multiple devices. Also I am looking for suggestions on how best to address this without negatively impacting TX performance. Thanks, Alex DRHD: handling fault status reg 2 DMAR:[DMA Read] Request device [05:00.0] fault addr ffb9c000 DMAR:[fault reason 06] PTE Read access is not set ------------[ cut here ]------------ WARNING: at lib/dma-debug.c:798 check_unmap+0x158/0x5bf() Hardware name: S5520HC tg3 0000:05:00.0: DMA-API: device driver tries to free DMA memory it has not allocated [device address=0x00000000ffb9c822] [size=1200 bytes] Modules linked in: pktgen tg3 ipv6 autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table video output sbs sbshc joydev shpchp iTCO_wdt i2c_i801 i2c_core pcspkr iTCO_vendor_support usb_storage [last unloaded: igb] Pid: 8701, comm: kpktgend_0 Not tainted 2.6.32-rc3-net-next-igb-tag110309 #12 Call Trace: <IRQ> [<ffffffff811c2500>] ? check_unmap+0x158/0x5bf [<ffffffff81046006>] warn_slowpath_common+0x77/0x8f [<ffffffff810460ce>] warn_slowpath_fmt+0x9f/0xa1 [<ffffffff8137fca5>] ? _spin_lock_irqsave+0x44/0x50 [<ffffffff811c182f>] ? get_hash_bucket+0x28/0x33 [<ffffffff8106c648>] ? register_lock_class+0x20/0x29f [<ffffffff811c2500>] check_unmap+0x158/0x5bf [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffff8100e020>] ? do_IRQ+0x97/0xae [<ffffffff811c2b11>] debug_dma_unmap_page+0x80/0x92 [<ffffffff811dcea8>] ? intel_unmap_page+0xf6/0x103 [<ffffffff812e1dd9>] dma_unmap_single_attrs+0x5c/0x68 [<ffffffff812e2043>] skb_dma_unmap+0x3f/0x78 [<ffffffffa01a8890>] tg3_poll+0x172/0x77d [tg3] [<ffffffff812e2f84>] ? rcu_read_lock+0x0/0x3a [<ffffffff812e7ff1>] net_rx_action+0xbc/0x19f [<ffffffff8104bcf1>] __do_softirq+0xd5/0x19d [<ffffffff8100cc9c>] call_softirq+0x1c/0x34 <EOI> [<ffffffff8100e2d8>] do_softirq+0x33/0x6b [<ffffffff8104b541>] _local_bh_enable_ip+0x78/0xa7 [<ffffffff8104b579>] local_bh_enable_ip+0x9/0xb [<ffffffff8137f9a2>] _spin_unlock_bh+0x39/0x3d [<ffffffffa004a2c3>] pktgen_xmit+0xf5f/0xfe3 [pktgen] [<ffffffffa01a65f7>] ? tg3_start_xmit_dma_bug+0x0/0x7cd [tg3] [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffffa004a3e5>] ? pktgen_thread_worker+0x9e/0x660 [pktgen] [<ffffffffa004a4ed>] pktgen_thread_worker+0x1a6/0x660 [pktgen] [<ffffffff8103236b>] ? need_resched+0x1e/0x28 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffffa004a347>] ? pktgen_thread_worker+0x0/0x660 [pktgen] [<ffffffff8105e459>] kthread+0x69/0x71 [<ffffffff8100cb9a>] child_rip+0xa/0x20 [<ffffffff8105e3f0>] ? kthread+0x0/0x71 [<ffffffff8100cb90>] ? child_rip+0x0/0x20 ---[ end trace 39dadbc5c44cf3fd ]--- ------------[ cut here ]------------ WARNING: at drivers/pci/intel-iommu.c:2719 intel_unmap_page+0x8e/0x103() Hardware name: S5520HC Driver unmaps unmatched page at PFN ffb9c822 Modules linked in: pktgen tg3 ipv6 autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table video output sbs sbshc joydev shpchp iTCO_wdt i2c_i801 i2c_core pcspkr iTCO_vendor_support usb_storage [last unloaded: igb] Pid: 8701, comm: kpktgend_0 Tainted: G W 2.6.32-rc3-net-next-igb-tag110309 #12 Call Trace: <IRQ> [<ffffffff811dce40>] ? intel_unmap_page+0x8e/0x103 [<ffffffff81046006>] warn_slowpath_common+0x77/0x8f [<ffffffff810460ce>] warn_slowpath_fmt+0x9f/0xa1 [<ffffffff811dac28>] ? find_iova+0x18/0x66 [<ffffffff8137fa55>] ? _spin_unlock_irqrestore+0x57/0x5c [<ffffffff811dac6b>] ? find_iova+0x5b/0x66 [<ffffffff811dce40>] intel_unmap_page+0x8e/0x103 [<ffffffff812e1dc3>] dma_unmap_single_attrs+0x46/0x68 [<ffffffff812e2043>] skb_dma_unmap+0x3f/0x78 [<ffffffffa01a8890>] tg3_poll+0x172/0x77d [tg3] [<ffffffff812e2f84>] ? rcu_read_lock+0x0/0x3a [<ffffffff812e7ff1>] net_rx_action+0xbc/0x19f [<ffffffff8104bcf1>] __do_softirq+0xd5/0x19d [<ffffffff8100cc9c>] call_softirq+0x1c/0x34 <EOI> [<ffffffff8100e2d8>] do_softirq+0x33/0x6b [<ffffffff8104b541>] _local_bh_enable_ip+0x78/0xa7 [<ffffffff8104b579>] local_bh_enable_ip+0x9/0xb [<ffffffff8137f9a2>] _spin_unlock_bh+0x39/0x3d [<ffffffffa004a2c3>] pktgen_xmit+0xf5f/0xfe3 [pktgen] [<ffffffffa01a65f7>] ? tg3_start_xmit_dma_bug+0x0/0x7cd [tg3] [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffffa004a3e5>] ? pktgen_thread_worker+0x9e/0x660 [pktgen] [<ffffffffa004a4ed>] pktgen_thread_worker+0x1a6/0x660 [pktgen] [<ffffffff8103236b>] ? need_resched+0x1e/0x28 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffffa004a347>] ? pktgen_thread_worker+0x0/0x660 [pktgen] [<ffffffff8105e459>] kthread+0x69/0x71 [<ffffffff8100cb9a>] child_rip+0xa/0x20 [<ffffffff8105e3f0>] ? kthread+0x0/0x71 [<ffffffff8100cb90>] ? child_rip+0x0/0x20 ---[ end trace 39dadbc5c44cf3fe ]--- ------------[ cut here ]------------ kernel BUG at drivers/pci/iova.c:155! invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC last sysfs file: /sys/devices/pci0000:00/0000:00:1e.0/0000:0a:00.0/local_cpus CPU 0 Modules linked in: pktgen tg3 ipv6 autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table video output sbs sbshc joydev shpchp iTCO_wdt i2c_i801 i2c_core pcspkr iTCO_vendor_support usb_storage [last unloaded: igb] Pid: 8701, comm: kpktgend_0 Tainted: G W 2.6.32-rc3-net-next-igb-tag110309 #12 S5520HC RIP: 0010:[<ffffffff811db065>] [<ffffffff811db065>] alloc_iova+0x174/0x1d7 RSP: 0018:ffff88036570fac0 EFLAGS: 00010046 RAX: 00000000000ffb9b RBX: ffff8801e78d4180 RCX: 0000000000000000 RDX: ffff8801e73ba710 RSI: ffff8801e73ba700 RDI: 0000000000000001 RBP: ffff88036570fb20 R08: 0000000000000000 R09: ffff880100000000 R10: ffffffff811daf48 R11: 0000fffffffff000 R12: 00000000000ffb9b R13: 0000000000000001 R14: ffff8801e5cad520 R15: ffff8801e73ba700 FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 00007fef515530a0 CR3: 0000000001001000 CR4: 00000000000006f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process kpktgend_0 (pid: 8701, threadinfo ffff88036570e000, task ffff8803661d3fc0) Stack: ffffffff81e5a328 ffffffff81e5a310 00000000000fffff 0000000000000286 <0> 01ff8801e88de890 ffffffff81e5a310 ffff8801e78d4180 ffff8801e88de890 <0> 0000fffffffff000 ffff8801e5cad500 0000000000000001 ffff8801e88de890 Call Trace: [<ffffffff811dd992>] intel_alloc_iova+0x68/0xb4 [<ffffffff811ddc3a>] __intel_map_single+0x7d/0x152 [<ffffffff811ddd0f>] ? intel_map_page+0x0/0x3e [<ffffffff811ddd4b>] intel_map_page+0x3c/0x3e [<ffffffff812e1e77>] skb_dma_map+0x92/0x21f [<ffffffff81039012>] ? finish_task_switch+0x7c/0x9e [<ffffffffa01a6976>] tg3_start_xmit_dma_bug+0x37f/0x7cd [tg3] [<ffffffffa004a1ef>] ? pktgen_xmit+0xe8b/0xfe3 [pktgen] [<ffffffffa004a230>] pktgen_xmit+0xecc/0xfe3 [pktgen] [<ffffffffa01a65f7>] ? tg3_start_xmit_dma_bug+0x0/0x7cd [tg3] [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffffa004a3e5>] ? pktgen_thread_worker+0x9e/0x660 [pktgen] [<ffffffffa004a4ed>] pktgen_thread_worker+0x1a6/0x660 [pktgen] [<ffffffff8103236b>] ? need_resched+0x1e/0x28 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffffa004a347>] ? pktgen_thread_worker+0x0/0x660 [pktgen] [<ffffffff8105e459>] kthread+0x69/0x71 [<ffffffff8100cb9a>] child_rip+0xa/0x20 [<ffffffff8105e3f0>] ? kthread+0x0/0x71 [<ffffffff8100cb90>] ? child_rip+0x0/0x20 Code: 05 ff 49 89 47 18 48 83 7d d0 00 75 04 49 8d 56 30 31 c0 eb 1d 48 8b 46 20 48 8d 56 10 49 39 47 20 72 0c 76 06 48 8d 56 08 eb 04 <0f> 0b eb fe 48 89 f0 48 8b 32 48 85 f6 75 db 48 89 c6 4c 89 ff RIP [<ffffffff811db065>] alloc_iova+0x174/0x1d7 RSP <ffff88036570fac0> ---[ end trace 39dadbc5c44cf3ff ]--- Kernel panic - not syncing: Fatal exception in interrupt Pid: 8701, comm: kpktgend_0 Tainted: G D W 2.6.32-rc3-net-next-igb-tag110309 #12 Call Trace: [<ffffffff810461c4>] panic+0xe5/0x195 [<ffffffff8137fa4c>] ? _spin_unlock_irqrestore+0x4e/0x5c [<ffffffff81046860>] ? release_console_sem+0x19e/0x1ab [<ffffffff81046a8b>] ? console_unblank+0x65/0x69 [<ffffffff81045e36>] ? print_oops_end_marker+0x1e/0x20 [<ffffffff81380d8f>] oops_end+0xaa/0xba [<ffffffff8100f41d>] die+0x55/0x5e [<ffffffff813806b2>] do_trap+0x110/0x11f [<ffffffff8100d4bb>] do_invalid_op+0x91/0x9a [<ffffffff811db065>] ? alloc_iova+0x174/0x1d7 [<ffffffff8137faa4>] ? _spin_unlock+0x4a/0x57 [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffff8100c935>] invalid_op+0x15/0x20 [<ffffffff811daf48>] ? alloc_iova+0x57/0x1d7 [<ffffffff811db065>] ? alloc_iova+0x174/0x1d7 [<ffffffff811dd992>] intel_alloc_iova+0x68/0xb4 [<ffffffff811ddc3a>] __intel_map_single+0x7d/0x152 [<ffffffff811ddd0f>] ? intel_map_page+0x0/0x3e [<ffffffff811ddd4b>] intel_map_page+0x3c/0x3e [<ffffffff812e1e77>] skb_dma_map+0x92/0x21f [<ffffffff81039012>] ? finish_task_switch+0x7c/0x9e [<ffffffffa01a6976>] tg3_start_xmit_dma_bug+0x37f/0x7cd [tg3] [<ffffffffa004a1ef>] ? pktgen_xmit+0xe8b/0xfe3 [pktgen] [<ffffffffa004a230>] pktgen_xmit+0xecc/0xfe3 [pktgen] [<ffffffffa01a65f7>] ? tg3_start_xmit_dma_bug+0x0/0x7cd [tg3] [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffffa004a3e5>] ? pktgen_thread_worker+0x9e/0x660 [pktgen] [<ffffffffa004a4ed>] pktgen_thread_worker+0x1a6/0x660 [pktgen] [<ffffffff8103236b>] ? need_resched+0x1e/0x28 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffffa004a347>] ? pktgen_thread_worker+0x0/0x660 [pktgen] [<ffffffff8105e459>] kthread+0x69/0x71 [<ffffffff8100cb9a>] child_rip+0xa/0x20 [<ffffffff8105e3f0>] ? kthread+0x0/0x71 [<ffffffff8100cb90>] ? child_rip+0x0/0x20 ------------[ cut here ]------------ WARNING: at arch/x86/kernel/smp.c:117 native_smp_send_reschedule+0x25/0x43() Hardware name: S5520HC Modules linked in: pktgen tg3 ipv6 autofs4 sunrpc cpufreq_ondemand acpi_cpufreq freq_table video output sbs sbshc joydev shpchp iTCO_wdt i2c_i801 i2c_core pcspkr iTCO_vendor_support usb_storage [last unloaded: igb] Pid: 8701, comm: kpktgend_0 Tainted: G D W 2.6.32-rc3-net-next-igb-tag110309 #12 Call Trace: <IRQ> [<ffffffff81034a05>] ? __enqueue_entity+0x74/0x76 [<ffffffff8101eefc>] ? native_smp_send_reschedule+0x25/0x43 [<ffffffff81046006>] warn_slowpath_common+0x77/0x8f [<ffffffff8104602d>] warn_slowpath_null+0xf/0x11 [<ffffffff8101eefc>] native_smp_send_reschedule+0x25/0x43 [<ffffffff8102ff5d>] smp_send_reschedule+0xa/0xc [<ffffffff810324ea>] resched_task+0x67/0x6c [<ffffffff81032585>] check_preempt_curr_idle+0x10/0x12 [<ffffffff81030056>] check_preempt_curr+0x12/0x14 [<ffffffff8103dfbf>] try_to_wake_up+0x249/0x2b2 [<ffffffff8103e035>] default_wake_function+0xd/0xf [<ffffffff8105e6a4>] autoremove_wake_function+0x11/0x38 [<ffffffff81030bad>] __wake_up_common+0x41/0x74 [<ffffffff810341d8>] __wake_up+0x38/0x4f [<ffffffff810468c0>] printk_tick+0x53/0x58 [<ffffffff81051f13>] update_process_times+0x3e/0x54 [<ffffffff8106b278>] tick_sched_timer+0x70/0x99 [<ffffffff8106b208>] ? tick_sched_timer+0x0/0x99 [<ffffffff81061101>] __run_hrtimer+0xba/0x121 [<ffffffff8106125e>] hrtimer_interrupt+0xf6/0x152 [<ffffffff81096d71>] ? rcu_irq_enter+0x15/0x63 [<ffffffff81020124>] smp_apic_timer_interrupt+0x73/0x87 [<ffffffff8100c673>] apic_timer_interrupt+0x13/0x20 <EOI> [<ffffffff81382434>] ? __atomic_notifier_call_chain+0x0/0xb2 [<ffffffff81382434>] ? __atomic_notifier_call_chain+0x0/0xb2 [<ffffffff81046252>] ? panic+0x173/0x195 [<ffffffff810461e6>] ? panic+0x107/0x195 [<ffffffff8137fa4c>] ? _spin_unlock_irqrestore+0x4e/0x5c [<ffffffff81046860>] ? release_console_sem+0x19e/0x1ab [<ffffffff81046a8b>] ? console_unblank+0x65/0x69 [<ffffffff81045e36>] ? print_oops_end_marker+0x1e/0x20 [<ffffffff81380d8f>] ? oops_end+0xaa/0xba [<ffffffff8100f41d>] ? die+0x55/0x5e [<ffffffff813806b2>] ? do_trap+0x110/0x11f [<ffffffff8100d4bb>] ? do_invalid_op+0x91/0x9a [<ffffffff811db065>] ? alloc_iova+0x174/0x1d7 [<ffffffff8137faa4>] ? _spin_unlock+0x4a/0x57 [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffff8100c935>] ? invalid_op+0x15/0x20 [<ffffffff811daf48>] ? alloc_iova+0x57/0x1d7 [<ffffffff811db065>] ? alloc_iova+0x174/0x1d7 [<ffffffff811dd992>] ? intel_alloc_iova+0x68/0xb4 [<ffffffff811ddc3a>] ? __intel_map_single+0x7d/0x152 [<ffffffff811ddd0f>] ? intel_map_page+0x0/0x3e [<ffffffff811ddd4b>] ? intel_map_page+0x3c/0x3e [<ffffffff812e1e77>] ? skb_dma_map+0x92/0x21f [<ffffffff81039012>] ? finish_task_switch+0x7c/0x9e [<ffffffffa01a6976>] ? tg3_start_xmit_dma_bug+0x37f/0x7cd [tg3] [<ffffffffa004a1ef>] ? pktgen_xmit+0xe8b/0xfe3 [pktgen] [<ffffffffa004a230>] ? pktgen_xmit+0xecc/0xfe3 [pktgen] [<ffffffffa01a65f7>] ? tg3_start_xmit_dma_bug+0x0/0x7cd [tg3] [<ffffffff8106ccb9>] ? __lock_acquire+0x20d/0x3ce [<ffffffffa004a3e5>] ? pktgen_thread_worker+0x9e/0x660 [pktgen] [<ffffffffa004a4ed>] ? pktgen_thread_worker+0x1a6/0x660 [pktgen] [<ffffffff8103236b>] ? need_resched+0x1e/0x28 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffff8105e693>] ? autoremove_wake_function+0x0/0x38 [<ffffffffa004a347>] ? pktgen_thread_worker+0x0/0x660 [pktgen] [<ffffffff8105e459>] ? kthread+0x69/0x71 [<ffffffff8100cb9a>] ? child_rip+0xa/0x20 [<ffffffff8105e3f0>] ? kthread+0x0/0x71 [<ffffffff8100cb90>] ? child_rip+0x0/0x20 ---[ end trace 39dadbc5c44cf400 ]--- -- To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Powered by blists - more mailing lists