linux-kernel - [5.15-rc1][PPC/PCI] IPR adapter reset threw EEH and fail to recover while booting mainline on powerpc

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <65f0085f-c6a9-e3ea-4d60-fcf09b7c7360@linux.vnet.ibm.com>
Date:   Mon, 20 Sep 2021 11:25:49 +0530
From:   Abdul Haleem <abdhalee@...ux.vnet.ibm.com>
To:     linux-kernel <linux-kernel@...r.kernel.org>
Cc:     linuxppc-dev <linuxppc-dev@...ts.ozlabs.org>,
        linux-pci@...r.kernel.org, linux-scsi <linux-scsi@...r.kernel.org>,
        dougmill@...ibm.com, sachinp <sachinp@...ux.vnet.ibm.com>,
        Brian King <brking@...ux.vnet.ibm.com>
Subject: [5.15-rc1][PPC/PCI] IPR adapter reset threw EEH and fail to recover
 while booting mainline on powerpc

Greeting's

IPR adapter reset triggers EEH errors and recovered 6 times and than 
task hung on today's mainline kernel on powerpc box with loads of hung 
task call traces

The problem is not seen with 5.14-rc7 (good)  and is last recreated on 
5.15-rc1 (bad) kernel

Console logs
EEH: Recovering PHB#134-PE#800000
EEH: PE location: N/A, PHB location: N/A
EEH: Frozen PHB#134-PE#800000 detected
EEH: Call Trace:
EEH: [c000000000054f90] __eeh_send_failure_event+0x70/0x150
EEH: [c00000000004e148] eeh_dev_check_failure+0x2e8/0x680
EEH: [c00000000004e5a8] eeh_check_failure+0xc8/0x100
EEH: [c008000000501c08] eeh_readl+0x70/0xa0 [ipr]
EEH: [c00800000050e76c] ipr_reset_enable_ioa+0x474/0x5a0 [ipr]
EEH: [c0080000005049a0] ipr_reset_ioa_job+0x78/0x150 [ipr]
EEH: [c008000000508f04] ipr_pci_slot_reset+0xac/0x140 [ipr]
EEH: [c000000000052ce8] eeh_report_reset+0xb8/0xe0
EEH: [c000000000054834] eeh_pe_report+0x254/0x4b4
EEH: [c000000000053efc] eeh_handle_normal_event+0x9dc/0xa80
EEH: [c000000000054de8] eeh_event_handler+0x108/0x190
EEH: [c00000000018151c] kthread+0x18c/0x1a0
EEH: [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
EEH: PHB#134-PE#800000 has failed 6 times in the last hour and has been 
permanently disabled.
EEH: Unable to recover from failure from PHB#134-PE#800000.
Please try reseating or replacing it
EEH: of node=0134:80:00.0
EEH: PCI device/vendor: 034a1014
EEH: PCI cmd/status register: 00100546
EEH: PCI-E capabilities and status follow:
EEH: PCI-E 00: 0002a410 00008fc2 0000585e 0003f503
EEH: PCI-E 10: 10830000 00000000 00000000 00000000
EEH: PCI-E 20: 00000000
EEH: PCI-E AER capability register set follows:
EEH: PCI-E AER 00: 18820001 00000000 00400000 00462030
EEH: PCI-E AER 10: 00000000 0000e000 000001e0 00000000
EEH: PCI-E AER 20: 00000000 00000000 00000000 00000000
EEH: PCI-E AER 30: 00000000 00000000
EEH: Beginning: 'error_detected(permanent failure)'
PCI 0134:80:00.0#800000: EEH: not actionable (1,1,1)
[FAILED] Failed to start udev Wait for Complete Device Initialization.
See 'systemctl status systemd-udev-settle.service' for details.
          Starting Device-Mapper Multipath Device Controller...
[  OK  ] Started Device-Mapper Multipath Device Controller.
[  OK  ] Reached target Local File Systems (Pre).
[  OK  ] Reached target Local File Systems.
          Starting Create Volatile Files and Directories...
[  OK  ] Started Create Volatile Files and Directories.
[  OK  ] Reached target System Initialization.
[  OK  ] Reached target Basic System.
EEH: Finished:'error_detected(permanent failure)'
device-mapper: multipath service-time: version 0.3.0 loaded
INFO: task eehd:137 blocked for more than 122 seconds.
       Not tainted 5.15.0-rc1-autotest #3
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:eehd            state:D stack:    0 pid:  137 ppid:     2 
flags:0x00000800
Call Trace:
[c00000000d167650] [c0000000001c329c] __wake_up_common+0xcc/0x290 
(unreliable)
[c00000000d167840] [c00000000001e6c8] __switch_to+0x278/0x490
[c00000000d1678a0] [c000000000c8b9bc] __schedule+0x31c/0xa20
[c00000000d167960] [c000000000c8c128] schedule+0x68/0x120
[c00000000d167990] [c00800000050a3c4] __ipr_remove+0x10c/0x4b0 [ipr]
[c00000000d167a60] [c00800000050a828] ipr_remove+0xc0/0x110 [ipr]
[c00000000d167ae0] [c000000000732bd4] pci_device_remove+0x64/0x120
[c00000000d167b20] [c000000000811318] 
device_release_driver_internal+0x168/0x2c0
[c00000000d167b60] [c000000000724618] pci_stop_bus_device+0xa8/0x100
[c00000000d167ba0] [c000000000724810] 
pci_stop_and_remove_bus_device+0x20/0x40
[c00000000d167bd0] [c000000000075e60] pci_hp_remove_devices+0x90/0x130
[c00000000d167c60] [c000000000053cd8] eeh_handle_normal_event+0x7b8/0xa80
[c00000000d167d40] [c000000000054de8] eeh_event_handler+0x108/0x190
[c00000000d167da0] [c00000000018151c] kthread+0x18c/0x1a0
[c00000000d167e10] [c00000000000cfd4] ret_from_kernel_thread+0x5c/0x64
INFO: task systemd-udevd:473 blocked for more than 122 seconds.
       Not tainted 5.15.0-rc1-autotest #3

-- 
Regard's

Abdul Haleem
IBM Linux Technology Center


View attachment "iprdebuglogs.txt" of type "text/plain" (79530 bytes)