linux-kernel - Re: [3.11.4] Thunderbolt/PCI unplug oops in pci_pme_list

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CAMxnaaUdN7k7wjUKW53zELb9wRwBRU_arNXNwrJTSSMRH=rQZQ@mail.gmail.com>
Date:	Sun, 27 Oct 2013 02:39:30 +0200
From:	Andreas Noever <andreas.noever@...il.com>
To:	Bjorn Helgaas <bhelgaas@...gle.com>
Cc:	Yinghai Lu <yinghai@...nel.org>,
	Matthew Garrett <mjg59@...f.ucam.org>,
	"linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
	"Rafael J. Wysocki" <rjw@...k.pl>,
	"linux-pci@...r.kernel.org" <linux-pci@...r.kernel.org>,
	Mika Westerberg <mika.westerberg@...ux.intel.com>,
	"Kirill A. Shutemov" <kirill.shutemov@...ux.intel.com>
Subject: Re: [3.11.4] Thunderbolt/PCI unplug oops in pci_pme_list_scan

> Sorry, I didn't understand this.  Is this supposed to be an
> explanation of how 928bea fixes the oops that Andreas saw?  If so, can
> you be a little more explicit about when the pci_dev got freed and
> when pci_pme_list_scan() walked the list and accessed the freed area?

I did some more debugging and it seems that 928bea is innocent after
all. I added some debugging statements to pci_pme_active. The
additional delay seems to make the oops easier to trigger and I can
now replicate it up to
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=5137a2ee2007d9cbbbeebd14abe08357a079b607
which makes much more sense.

Here is what's going on (in 3.11). First of all pci_pme_activate is
only ever called with false as the second paramter during boot. Now
when I unplug the adapter, the first call is:
 [<ffffffff814b1cc7>] dump_stack+0x54/0x8d
 [<ffffffff812ae970>] pci_pme_active+0x30/0x210
 [<ffffffff813bf2bc>] ? pci_read+0x2c/0x30 (this should be pci_stop_dev imho)
 [<ffffffff812ac8ae>] pci_stop_bus_device+0x4e/0xa0
 [<ffffffff812ac89b>] pci_stop_bus_device+0x3b/0xa0
 [<ffffffff812ac89b>] pci_stop_bus_device+0x3b/0xa0
 [<ffffffff812aca02>] pci_stop_and_remove_bus_device+0x12/0x20
 [<ffffffff812c4698>] pciehp_unconfigure_device+0xa8/0x1b0
 [<ffffffff812c3ff8>] pciehp_disable_slot+0x68/0x200
 [<ffffffff812c4213>] pciehp_power_thread+0x83/0xf0
 [<ffffffff8107b5b8>] process_one_work+0x178/0x470
 [<ffffffff8107bf81>] worker_thread+0x121/0x3a0
 [<ffffffff8107be60>] ? manage_workers.isra.21+0x2b0/0x2b0
 [<ffffffff81082d80>] kthread+0xc0/0xd0
 [<ffffffff81060000>] ? SyS_unshare+0x220/0x280
 [<ffffffff81082cc0>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff814c07ec>] ret_from_fork+0x7c/0xb0
 [<ffffffff81082cc0>] ? kthread_create_on_node+0x120/0x120
tg3 0000:0a:00.0: PME# disabled

This is still fine. But then it gets interesting. The next call is:
 [<ffffffff814b1cc7>] dump_stack+0x54/0x8d
 [<ffffffff812ae970>] pci_pme_active+0x30/0x210
 [<ffffffff812aebb5>] __pci_enable_wake+0x65/0x160
 [<ffffffff812aecd5>] pci_wake_from_d3+0x25/0x40
 [<ffffffffa0511c29>] tg3_power_down+0x29/0x40 [tg3]
 [<ffffffffa0511d4c>] tg3_close+0x10c/0x1d0 [tg3]
 [<ffffffff813d67b5>] __dev_close_many+0x85/0xd0
 [<ffffffff813d68cb>] dev_close_many+0x8b/0x100
 [<ffffffff813d8dd8>] rollback_registered_many+0xd8/0x250
 [<ffffffff813d8f7d>] rollback_registered+0x2d/0x40
 [<ffffffff813da828>] unregister_netdevice_queue+0x58/0xb0
 [<ffffffff813da89c>] unregister_netdev+0x1c/0x30
 [<ffffffffa050104b>] tg3_remove_one+0x6b/0x120 [tg3]
 [<ffffffff812b1b0b>] pci_device_remove+0x3b/0xb0
 [<ffffffff81371c1f>] __device_release_driver+0x7f/0xf0
 [<ffffffff81371cb3>] device_release_driver+0x23/0x30
 [<ffffffff81371484>] bus_remove_device+0xf4/0x170
 [<ffffffff8136df45>] device_del+0x135/0x1d0
 [<ffffffff812ac8f4>] pci_stop_bus_device+0x94/0xa0
 [<ffffffff812ac89b>] pci_stop_bus_device+0x3b/0xa0
 [<ffffffff812ac89b>] pci_stop_bus_device+0x3b/0xa0
 [<ffffffff812aca02>] pci_stop_and_remove_bus_device+0x12/0x20
 [<ffffffff812c4698>] pciehp_unconfigure_device+0xa8/0x1b0
 [<ffffffff812c3ff8>] pciehp_disable_slot+0x68/0x200
 [<ffffffff812c4213>] pciehp_power_thread+0x83/0xf0
 [<ffffffff8107b5b8>] process_one_work+0x178/0x470
 [<ffffffff8107bf81>] worker_thread+0x121/0x3a0
 [<ffffffff8107be60>] ? manage_workers.isra.21+0x2b0/0x2b0
 [<ffffffff81082d80>] kthread+0xc0/0xd0
 [<ffffffff81060000>] ? SyS_unshare+0x220/0x280
 [<ffffffff81082cc0>] ? kthread_create_on_node+0x120/0x120
 [<ffffffff814c07ec>] ret_from_fork+0x7c/0xb0
 [<ffffffff81082cc0>] ? kthread_create_on_node+0x120/0x120
tg3 0000:0a:00.0: PME# enabled

On removal tg3 calls pci_wake_from_d3 to enable/disable wake-on-lan.
This then calls pci_pme_activate(dev, true) for a device which is
about to be deleted. The linked commit does no longer call
pci_wake_from_d3, which "fixes" the problem.

Andreas
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/