Date:   Fri, 11 Feb 2022 15:19:56 +0530
From:   Shyam Sundar S K <Shyam-sundar.S-k@....com>
To:     Thomas Kupper <thomas@...per.org>,
        Tom Lendacky <thomas.lendacky@....com>
Cc:     netdev@...r.kernel.org
Subject: Re: AMD XGBE "phy irq request failed" kernel v5.17-rc2 on V1500B
 based board



On 2/11/2022 3:03 PM, Thomas Kupper wrote:
> 
> Am 08.02.22 um 17:24 schrieb Tom Lendacky:
>> On 2/7/22 11:59, Thomas Kupper wrote:
>>>
>>> Am 07.02.22 um 16:19 schrieb Shyam Sundar S K:
>>>>
>>>> On 2/7/2022 8:02 PM, Tom Lendacky wrote:
>>>>> On 2/5/22 12:14, Thomas Kupper wrote:
>>>>>> Am 05.02.22 um 16:51 schrieb Tom Lendacky:
>>>>>>> On 2/5/22 04:06, Thomas Kupper wrote:
>>>>>>> Reloading the module and specify the dyndbg option to get some
>>>>>>> additional debug output.
>>>>>>>
>>>>>>> I'm adding Shyam to the thread, too, as I'm not familiar with the
>>>>>>> configuration for this chip.
>>>>>>>
>>>>>> Right after boot:
>>>>>>
>>>>>> [    5.352977] amd-xgbe 0000:06:00.1 eth0: net device enabled
>>>>>> [    5.354198] amd-xgbe 0000:06:00.2 eth1: net device enabled
>>>>>> ...
>>>>>> [    5.382185] amd-xgbe 0000:06:00.1 enp6s0f1: renamed from eth0
>>>>>> [    5.426931] amd-xgbe 0000:06:00.2 enp6s0f2: renamed from eth1
>>>>>> ...
>>>>>> [    9.701637] amd-xgbe 0000:06:00.2 enp6s0f2: phy powered off
>>>>>> [    9.701679] amd-xgbe 0000:06:00.2 enp6s0f2: CL73 AN disabled
>>>>>> [    9.701715] amd-xgbe 0000:06:00.2 enp6s0f2: CL37 AN disabled
>>>>>> [    9.738191] amd-xgbe 0000:06:00.2 enp6s0f2: starting PHY
>>>>>> [    9.738219] amd-xgbe 0000:06:00.2 enp6s0f2: starting I2C
>>>>>> ...
>>>>>> [   10.742622] amd-xgbe 0000:06:00.2 enp6s0f2: firmware mailbox
>>>>>> command did not complete
>>>>>> [   10.742710] amd-xgbe 0000:06:00.2 enp6s0f2: firmware mailbox reset
>>>>>> performed
>>>>>> [   10.750813] amd-xgbe 0000:06:00.2 enp6s0f2: 10GbE SFI mode set
>>>>>> [   10.768366] amd-xgbe 0000:06:00.2 enp6s0f2: 10GbE SFI mode set
>>>>>> [   10.768371] amd-xgbe 0000:06:00.2 enp6s0f2: fixed PHY
>>>>>> configuration
>>>>>>
>>>>>> Then after 'ifconfig enp6s0f2 up':
>>>>>>
>>>>>> [  189.184928] amd-xgbe 0000:06:00.2 enp6s0f2: phy powered off
>>>>>> [  189.191828] amd-xgbe 0000:06:00.2 enp6s0f2: 10GbE SFI mode set
>>>>>> [  189.191863] amd-xgbe 0000:06:00.2 enp6s0f2: CL73 AN disabled
>>>>>> [  189.191894] amd-xgbe 0000:06:00.2 enp6s0f2: CL37 AN disabled
>>>>>> [  189.196338] amd-xgbe 0000:06:00.2 enp6s0f2: starting PHY
>>>>>> [  189.198792] amd-xgbe 0000:06:00.2 enp6s0f2: 10GbE SFI mode set
>>>>>> [  189.212036] genirq: Flags mismatch irq 69. 00000000 (enp6s0f2-pcs)
>>>>>> vs. 00000000 (enp6s0f2-pcs)
>>>>>> [  189.221700] amd-xgbe 0000:06:00.2 enp6s0f2: phy irq request failed
>>>>>> [  189.231051] amd-xgbe 0000:06:00.2 enp6s0f2: phy powered off
>>>>>> [  189.231054] amd-xgbe 0000:06:00.2 enp6s0f2: stopping I2C
>>>>>>
>>>>> Please ensure that the ethtool msglvl is on for drv and probe. I was
>>>>> expecting to see some additional debug messages that I don't see here.
>>>>>
>>>>> Also, if you can provide the lspci output for the device (using -nn
>>>>> and
>>>>> -vv) that might be helpful as well.
>>>>>
>>>>> Shyam will be the best one to understand what is going on here.
>>>> On some other platforms, we have seen similar kind of problems getting
>>>> reported. There is a fix sent for validation.
>>>>
>>>> The root cause is that removal of xgbe driver is causing interrupt
>>>> storm
>>>> on the MP2 device (Sensor Fusion Hub).
>>>>
>>>> Shall submit a fix soon to upstream once the validation is done, you
>>>> may
>>>> give it a try with that and see if that helps.
>>>>
>>>> Thanks,
>>>> Shyam
>>>>
>>>>> Thanks,
>>>>> Tom
>>>
>>> Shyam, I will check the git logs for the relevant commit then from
>>> time to time.
>>> Looking at the code diff from OPNsense and the latest Linux kernel I
>>> assumed that there would much more to do then fix a irq strom (but I
>>> have no idea about the inner working of the kernel).
>>>
>>> Nevermind: Setting the 'msglvl 0x3' with ethtool the following info
>>> can be found in dmesg:
>>>
>>> Running : $ ifconfig enp6s0f2 up
>>> SIOCSIFFLAGS: Invalid argument
>>>
>>> ... and 'dmesg':
>>>
>>> [   55.177447] amd-xgbe 0000:06:00.2 enp6s0f2: channel-0: cpu=0, node=0
>>> [   55.177456] amd-xgbe 0000:06:00.2 enp6s0f2: channel-0:
>>> dma_regs=00000000d11bf3f1, dma_irq=74, tx=00000000dd57b5c4,
>>> rx=00000000d73e70f8
>>> [   55.177464] amd-xgbe 0000:06:00.2 enp6s0f2: channel-1: cpu=1, node=0
>>> [   55.177467] amd-xgbe 0000:06:00.2 enp6s0f2: channel-1:
>>> dma_regs=000000000d972dd7, dma_irq=75, tx=00000000573bcff8,
>>> rx=000000003d9a6f65
>>> [   55.177473] amd-xgbe 0000:06:00.2 enp6s0f2: channel-2: cpu=2, node=0
>>> [   55.177476] amd-xgbe 0000:06:00.2 enp6s0f2: channel-2:
>>> dma_regs=0000000046f71179, dma_irq=76, tx=00000000897116c9,
>>> rx=0000000004ba17e7
>>> [   55.177480] amd-xgbe 0000:06:00.2 enp6s0f2: channel-0 - Tx ring:
>>> [   55.177502] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=00000000794657ba, rdesc_dma=0x000000010fad8000,
>>> rdata=0000000008ace7d8, node=0
>>> [   55.177507] amd-xgbe 0000:06:00.2 enp6s0f2: channel-0 - Rx ring:
>>> [   55.177523] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=000000009313d9b3, rdesc_dma=0x0000000114538000,
>>> rdata=00000000510e3b77, node=0
>>> [   55.177527] amd-xgbe 0000:06:00.2 enp6s0f2: channel-1 - Tx ring:
>>> [   55.177543] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=00000000d26d9194, rdesc_dma=0x000000010a774000,
>>> rdata=00000000b9419829, node=0
>>> [   55.177547] amd-xgbe 0000:06:00.2 enp6s0f2: channel-1 - Rx ring:
>>> [   55.177564] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=0000000007bf60dd, rdesc_dma=0x000000010fb84000,
>>> rdata=00000000aa48e8c0, node=0
>>> [   55.177568] amd-xgbe 0000:06:00.2 enp6s0f2: channel-2 - Tx ring:
>>> [   55.177584] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=00000000e7e6c52e, rdesc_dma=0x000000010fa2a000,
>>> rdata=0000000017b5d85c, node=0
>>> [   55.177587] amd-xgbe 0000:06:00.2 enp6s0f2: channel-2 - Rx ring:
>>> [   55.177603] amd-xgbe 0000:06:00.2 enp6s0f2:
>>> rdesc=000000000898fbf4, rdesc_dma=0x0000000101f08000,
>>> rdata=00000000aded7d4c, node=0
>>> [   55.182366] amd-xgbe 0000:06:00.2 enp6s0f2: TXq0 mapped to TC0
>>> [   55.182381] amd-xgbe 0000:06:00.2 enp6s0f2: TXq1 mapped to TC1
>>> [   55.182388] amd-xgbe 0000:06:00.2 enp6s0f2: TXq2 mapped to TC2
>>> [   55.182395] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO0 mapped to RXq0
>>> [   55.182400] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO1 mapped to RXq0
>>> [   55.182405] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO2 mapped to RXq0
>>> [   55.182410] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO3 mapped to RXq1
>>> [   55.182414] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO4 mapped to RXq1
>>> [   55.182418] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO5 mapped to RXq1
>>> [   55.182423] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO6 mapped to RXq2
>>> [   55.182427] amd-xgbe 0000:06:00.2 enp6s0f2: PRIO7 mapped to RXq2
>>> [   55.182473] amd-xgbe 0000:06:00.2 enp6s0f2: 3 Tx hardware queues,
>>> 21760 byte fifo per queue
>>> [   55.182501] amd-xgbe 0000:06:00.2 enp6s0f2: 3 Rx hardware queues,
>>> 21760 byte fifo per queue
>>> [   55.182544] amd-xgbe 0000:06:00.2 enp6s0f2: flow control enabled
>>> for RXq0
>>> [   55.182550] amd-xgbe 0000:06:00.2 enp6s0f2: flow control enabled
>>> for RXq1
>>> [   55.182556] amd-xgbe 0000:06:00.2 enp6s0f2: flow control enabled
>>> for RXq2
>>> [   56.178946] amd-xgbe 0000:06:00.2 enp6s0f2: SFP detected:
>>> [   56.178954] amd-xgbe 0000:06:00.2 enp6s0f2:   vendor: MikroTik
>>> [   56.178958] amd-xgbe 0000:06:00.2 enp6s0f2:   part number: S+AO0005
>>> [   56.178961] amd-xgbe 0000:06:00.2 enp6s0f2:   revision level: 1.0
>>> [   56.178963] amd-xgbe 0000:06:00.2 enp6s0f2:   serial number:
>>> STST050B1900001
>>>
>>
>> Ah, it's been a while since I've had to use the debug support. Could
>> you also set the module debug parameter to 0x37 (debug=0x37) when
>> loading the module. That will capture some of the debug messages that
>> are issued on driver load. Sorry about that...
>>
>> Thanks,
>> Tom
> 
> Thanks Tom, I now got time to update to 5.17-rc3 and add the 'debug'
> module parameter. I assume that parameter works with the non-debug
> kernel? I don't really see any new messages related to the amd-xgbe driver:
> 
> dmesg right after boot:
> 
> [    0.000000] Linux version 5.17.0-rc3-tk (jane@...0q-ubu21) (gcc
> (Ubuntu 11.2.0-7ubuntu2) 11.2.0, GNU ld (GNU Binutils for Ubuntu) 2.37)
> #12 SMP PREEMPT Tue Feb 8 19:52:19 CET 2022
> [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.17.0-rc3-tk
> root=UUID=8e462830-8ba0-4061-8f23-6f29ce751792 ro console=tty0
> console=ttyS0,115200n8 amd_xgbe.dyndbg=+p amd_xgbe.debug=0x37
> ...
> [    5.275730] amd-xgbe 0000:06:00.1 eth0: net device enabled
> [    5.277766] amd-xgbe 0000:06:00.2 eth1: net device enabled
> [    5.665315] amd-xgbe 0000:06:00.2 enp6s0f2: renamed from eth1
> [    5.696665] amd-xgbe 0000:06:00.1 enp6s0f1: renamed from eth0
> 
> dmesg right after 'ifconfig enp6s0f2 up'
> 
> [   88.843454] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-0: cpu=0, node=0
> [   88.843464] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-0: dma_regs=000000001078e433, dma_irq=55,
> tx=00000000e8736669, rx=00000000fadd04ec
> [   88.843474] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-1: cpu=1, node=0
> [   88.843478] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-1: dma_regs=000000003c3cbea8, dma_irq=56,
> tx=000000000836d88c, rx=00000000920d02c4
> [   88.843485] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-2: cpu=2, node=0
> [   88.843488] amd_xgbe:xgbe_alloc_channels: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-2: dma_regs=000000008d034191, dma_irq=57,
> tx=00000000a0664378, rx=00000000d72ce726
> [   88.843493] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-0 - Tx ring:
> [   88.843514] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=00000000c6703013, rdesc_dma=0x0000000101c44000,
> rdata=0000000029951e4c, node=0
> [   88.843519] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-0 - Rx ring:
> [   88.843537] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=000000003262c446, rdesc_dma=0x0000000103c74000,
> rdata=000000001b7a4275, node=0
> [   88.843542] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-1 - Tx ring:
> [   88.843560] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=000000007ce3cc7e, rdesc_dma=0x00000001023c0000,
> rdata=00000000c0fc51d9, node=0
> [   88.843565] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-1 - Rx ring:
> [   88.843583] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=00000000448612df, rdesc_dma=0x00000001185b6000,
> rdata=00000000a23b7f86, node=0
> [   88.843587] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-2 - Tx ring:
> [   88.843606] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=00000000e509050e, rdesc_dma=0x0000000104db2000,
> rdata=000000000d605e1a, node=0
> [   88.843610] amd_xgbe:xgbe_alloc_ring_resources: amd-xgbe 0000:06:00.2
> enp6s0f2: channel-2 - Rx ring:
> [   88.843629] amd_xgbe:xgbe_init_ring: amd-xgbe 0000:06:00.2 enp6s0f2:
> rdesc=00000000436c5cc6, rdesc_dma=0x0000000114aaa000,
> rdata=00000000246ed062, node=0
> [   88.848416] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: TXq0 mapped to TC0
> [   88.848432] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: TXq1 mapped to TC1
> [   88.848440] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: TXq2 mapped to TC2
> [   88.848449] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO0 mapped to RXq0
> [   88.848455] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO1 mapped to RXq0
> [   88.848461] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO2 mapped to RXq0
> [   88.848467] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO3 mapped to RXq1
> [   88.848472] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO4 mapped to RXq1
> [   88.848478] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO5 mapped to RXq1
> [   88.848483] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO6 mapped to RXq2
> [   88.848489] amd_xgbe:xgbe_config_queue_mapping: amd-xgbe 0000:06:00.2
> enp6s0f2: PRIO7 mapped to RXq2
> [   88.848536] amd-xgbe 0000:06:00.2 enp6s0f2: 3 Tx hardware queues,
> 21760 byte fifo per queue
> [   88.848565] amd-xgbe 0000:06:00.2 enp6s0f2: 3 Rx hardware queues,
> 21760 byte fifo per queue
> [   88.848609] amd_xgbe:xgbe_enable_tx_flow_control: amd-xgbe
> 0000:06:00.2 enp6s0f2: flow control enabled for RXq0
> [   88.848619] amd_xgbe:xgbe_enable_tx_flow_control: amd-xgbe
> 0000:06:00.2 enp6s0f2: flow control enabled for RXq1
> [   88.848627] amd_xgbe:xgbe_enable_tx_flow_control: amd-xgbe
> 0000:06:00.2 enp6s0f2: flow control enabled for RXq2
> [   89.862558] amd_xgbe:xgbe_phy_sfp_eeprom_info: amd-xgbe 0000:06:00.2
> enp6s0f2: SFP detected:
> [   89.862567] amd_xgbe:xgbe_phy_sfp_eeprom_info: amd-xgbe 0000:06:00.2
> enp6s0f2:   vendor:         MikroTik
> [   89.862572] amd_xgbe:xgbe_phy_sfp_eeprom_info: amd-xgbe 0000:06:00.2
> enp6s0f2:   part number:    S+AO0005
> [   89.862576] amd_xgbe:xgbe_phy_sfp_eeprom_info: amd-xgbe 0000:06:00.2
> enp6s0f2:   revision level: 1.0
> [   89.862580] amd_xgbe:xgbe_phy_sfp_eeprom_info: amd-xgbe 0000:06:00.2
> enp6s0f2:   serial number:  STST050B1900001
> 
> again, dmesg diff after 'rmmod':

Can you add this change and see if it solves the problem?

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net.git/commit/?id=68c2d6af1f1e

Thanks,
Shyam

> 
> [  127.068380] ------------[ cut here ]------------
> [  127.068386] remove_proc_entry: removing non-empty directory 'irq/53',
> leaking at least 'enp6s0f2-i2c'
> [  127.068398] WARNING: CPU: 4 PID: 803 at fs/proc/generic.c:715
> remove_proc_entry+0x196/0x1b0
> [  127.068411] Modules linked in: nls_iso8859_1 intel_rapl_msr
> intel_rapl_common snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi
> edac_mce_amd snd_hda_codec snd_hda_core snd_hwdep snd_pcm kvm snd_timer
> snd_rn_pci_acp3x snd rapl efi_pstore k10temp soundcore snd_pci_acp3x ccp
> mac_hid sch_fq_codel msr drm ip_tables x_tables autofs4 btrfs
> blake2b_generic zstd_compress raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1
> raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
> aesni_intel crypto_simd igb nvme cryptd dca xhci_pci amd_xgbe(-)
> i2c_piix4 i2c_amd_mp2_pci xhci_pci_renesas nvme_core i2c_algo_bit video
> spi_amd
> [  127.068485] CPU: 4 PID: 803 Comm: rmmod Not tainted 5.17.0-rc3-tk #12
> [  127.068490] Hardware name: Deciso B.V. DEC2700 - OPNsense
> Appliance/Netboard-A10 Gen.3, BIOS 05.32.50.0012-A10.20 11/15/2021
> [  127.068493] RIP: 0010:remove_proc_entry+0x196/0x1b0
> [  127.068499] Code: 60 50 5e 84 48 85 c0 48 8d 90 78 ff ff ff 48 0f 45
> c2 49 8b 54 24 78 4c 8b 80 a0 00 00 00 48 8b 92 a0 00 00 00 e8 38 56 81
> 00 <0f> 0b e9 44 ff ff ff e8 9e c0 87 00 66 66 2e 0f 1f 84 00 00 00 00
> [  127.068502] RSP: 0018:ffffaf2940fffb60 EFLAGS: 00010286
> [  127.068506] RAX: 0000000000000000 RBX: ffff91fa4022ed80 RCX:
> 0000000000000000
> [  127.068509] RDX: 0000000000000001 RSI: ffffffff845bf281 RDI:
> 00000000ffffffff
> [  127.068511] RBP: ffffaf2940fffb90 R08: 0000000000000000 R09:
> ffffaf2940fff950
> [  127.068513] R10: ffffaf2940fff948 R11: ffffffff84f55f48 R12:
> ffff91fa44e8c540
> [  127.068515] R13: ffff91fa44e8c5c0 R14: 0000000000000036 R15:
> 0000000000000036
> [  127.068517] FS:  00007f3a68f9c400(0000) GS:ffff91fa6af00000(0000)
> knlGS:0000000000000000
> [  127.068520] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  127.068522] CR2: 00007fd6c2e7fd04 CR3: 0000000104ee0000 CR4:
> 00000000003506e0
> [  127.068525] Call Trace:
> [  127.068528]  <TASK>
> [  127.068534]  unregister_irq_proc+0xe4/0x110
> [  127.068541]  free_desc+0x2e/0x70
> [  127.068546]  irq_free_descs+0x50/0x80
> [  127.068550]  irq_domain_free_irqs+0x16b/0x1c0
> [  127.068554]  __msi_domain_free_irqs+0xf1/0x160
> [  127.068560]  msi_domain_free_irqs_descs_locked+0x20/0x50
> [  127.068565]  pci_msi_teardown_msi_irqs+0x49/0x50
> [  127.068571]  pci_disable_msix.part.0+0xff/0x160
> [  127.068575]  pci_free_irq_vectors+0x45/0x60
> [  127.068578]  xgbe_pci_remove+0x24/0x40 [amd_xgbe]
> [  127.068596]  pci_device_remove+0x39/0xa0
> [  127.068602]  __device_release_driver+0x181/0x250
> [  127.068608]  driver_detach+0xd3/0x120
> [  127.068612]  bus_remove_driver+0x59/0xd0
> [  127.068615]  driver_unregister+0x31/0x50
> [  127.068619]  pci_unregister_driver+0x40/0x90
> [  127.068623]  xgbe_pci_exit+0x15/0x20 [amd_xgbe]
> [  127.068639]  xgbe_mod_exit+0x9/0x880 [amd_xgbe]
> [  127.068654]  __do_sys_delete_module.constprop.0+0x183/0x290
> [  127.068660]  ? exit_to_user_mode_prepare+0x49/0x1e0
> [  127.068666]  __x64_sys_delete_module+0x12/0x20
> [  127.068670]  do_syscall_64+0x5c/0xc0
> [  127.068676]  ? irqentry_exit_to_user_mode+0x9/0x20
> [  127.068681]  ? irqentry_exit+0x33/0x40
> [  127.068685]  ? exc_page_fault+0x89/0x180
> [  127.068689]  ? asm_exc_page_fault+0x8/0x30
> [  127.068694]  entry_SYSCALL_64_after_hwframe+0x44/0xae
> [  127.068697] RIP: 0033:0x7f3a690cb8eb
> [  127.068702] Code: 73 01 c3 48 8b 0d 45 e5 0e 00 f7 d8 64 89 01 48 83
> c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 15 e5 0e 00 f7 d8 64 89 01 48
> [  127.068704] RSP: 002b:00007ffed553a818 EFLAGS: 00000206 ORIG_RAX:
> 00000000000000b0
> [  127.068708] RAX: ffffffffffffffda RBX: 00007f3a6a02e7b0 RCX:
> 00007f3a690cb8eb
> [  127.068710] RDX: 000000000000000a RSI: 0000000000000800 RDI:
> 00007f3a6a02e818
> [  127.068712] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 0000000000000000
> [  127.068714] R10: 00007f3a69163ac0 R11: 0000000000000206 R12:
> 00007ffed553aa70
> [  127.068716] R13: 00007ffed553b84a R14: 00007f3a6a02e2a0 R15:
> 00007f3a6a02e7b0
> [  127.068722]  </TASK>
> [  127.068723] ---[ end trace 0000000000000000 ]---
> [  127.068744] ------------[ cut here ]------------
> [  127.068746] remove_proc_entry: removing non-empty directory 'irq/54',
> leaking at least 'enp6s0f2-pcs'
> [  127.068755] WARNING: CPU: 4 PID: 803 at fs/proc/generic.c:715
> remove_proc_entry+0x196/0x1b0
> [  127.068761] Modules linked in: nls_iso8859_1 intel_rapl_msr
> intel_rapl_common snd_hda_intel snd_intel_dspcfg snd_intel_sdw_acpi
> edac_mce_amd snd_hda_codec snd_hda_core snd_hwdep snd_pcm kvm snd_timer
> snd_rn_pci_acp3x snd rapl efi_pstore k10temp soundcore snd_pci_acp3x ccp
> mac_hid sch_fq_codel msr drm ip_tables x_tables autofs4 btrfs
> blake2b_generic zstd_compress raid10 raid456 async_raid6_recov
> async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1
> raid0 multipath linear crct10dif_pclmul crc32_pclmul ghash_clmulni_intel
> aesni_intel crypto_simd igb nvme cryptd dca xhci_pci amd_xgbe(-)
> i2c_piix4 i2c_amd_mp2_pci xhci_pci_renesas nvme_core i2c_algo_bit video
> spi_amd
> [  127.068810] CPU: 4 PID: 803 Comm: rmmod Tainted: G W        
> 5.17.0-rc3-tk #12
> [  127.068814] Hardware name: Deciso B.V. DEC2700 - OPNsense
> Appliance/Netboard-A10 Gen.3, BIOS 05.32.50.0012-A10.20 11/15/2021
> [  127.068815] RIP: 0010:remove_proc_entry+0x196/0x1b0
> [  127.068820] Code: 60 50 5e 84 48 85 c0 48 8d 90 78 ff ff ff 48 0f 45
> c2 49 8b 54 24 78 4c 8b 80 a0 00 00 00 48 8b 92 a0 00 00 00 e8 38 56 81
> 00 <0f> 0b e9 44 ff ff ff e8 9e c0 87 00 66 66 2e 0f 1f 84 00 00 00 00
> [  127.068822] RSP: 0018:ffffaf2940fffb60 EFLAGS: 00010286
> [  127.068825] RAX: 0000000000000000 RBX: ffff91fa4022ed80 RCX:
> 0000000000000000
> [  127.068827] RDX: 0000000000000001 RSI: ffffffff845bf281 RDI:
> 00000000ffffffff
> [  127.068829] RBP: ffffaf2940fffb90 R08: 0000000000000000 R09:
> ffffaf2940fff950
> [  127.068830] R10: ffffaf2940fff948 R11: ffffffff84f55f48 R12:
> ffff91fa4eca7000
> [  127.068832] R13: ffff91fa4eca7080 R14: 0000000000000037 R15:
> 0000000000000037
> [  127.068834] FS:  00007f3a68f9c400(0000) GS:ffff91fa6af00000(0000)
> knlGS:0000000000000000
> [  127.068837] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  127.068839] CR2: 00007fd6c2e7fd04 CR3: 0000000104ee0000 CR4:
> 00000000003506e0
> [  127.068841] Call Trace:
> [  127.068842]  <TASK>
> [  127.068844]  unregister_irq_proc+0xe4/0x110
> [  127.068849]  free_desc+0x2e/0x70
> [  127.068852]  irq_free_descs+0x50/0x80
> [  127.068856]  irq_domain_free_irqs+0x16b/0x1c0
> [  127.068860]  __msi_domain_free_irqs+0xf1/0x160
> [  127.068865]  msi_domain_free_irqs_descs_locked+0x20/0x50
> [  127.068870]  pci_msi_teardown_msi_irqs+0x49/0x50
> [  127.068873]  pci_disable_msix.part.0+0xff/0x160
> [  127.068877]  pci_free_irq_vectors+0x45/0x60
> [  127.068881]  xgbe_pci_remove+0x24/0x40 [amd_xgbe]
> [  127.068896]  pci_device_remove+0x39/0xa0
> [  127.068900]  __device_release_driver+0x181/0x250
> [  127.068904]  driver_detach+0xd3/0x120
> [  127.068908]  bus_remove_driver+0x59/0xd0
> [  127.068911]  driver_unregister+0x31/0x50
> [  127.068914]  pci_unregister_driver+0x40/0x90
> [  127.068919]  xgbe_pci_exit+0x15/0x20 [amd_xgbe]
> [  127.068933]  xgbe_mod_exit+0x9/0x880 [amd_xgbe]
> [  127.068948]  __do_sys_delete_module.constprop.0+0x183/0x290
> [  127.068952]  ? exit_to_user_mode_prepare+0x49/0x1e0
> [  127.068957]  __x64_sys_delete_module+0x12/0x20
> [  127.068961]  do_syscall_64+0x5c/0xc0
> [  127.068964]  ? irqentry_exit_to_user_mode+0x9/0x20
> [  127.068969]  ? irqentry_exit+0x33/0x40
> [  127.068973]  ? exc_page_fault+0x89/0x180
> [  127.068977]  ? asm_exc_page_fault+0x8/0x30
> [  127.068980]  entry_SYSCALL_64_after_hwframe+0x44/0xae
> [  127.068983] RIP: 0033:0x7f3a690cb8eb
> [  127.068985] Code: 73 01 c3 48 8b 0d 45 e5 0e 00 f7 d8 64 89 01 48 83
> c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa b8 b0 00 00 00 0f
> 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 15 e5 0e 00 f7 d8 64 89 01 48
> [  127.068987] RSP: 002b:00007ffed553a818 EFLAGS: 00000206 ORIG_RAX:
> 00000000000000b0
> [  127.068990] RAX: ffffffffffffffda RBX: 00007f3a6a02e7b0 RCX:
> 00007f3a690cb8eb
> [  127.068991] RDX: 000000000000000a RSI: 0000000000000800 RDI:
> 00007f3a6a02e818
> [  127.068993] RBP: 0000000000000000 R08: 0000000000000000 R09:
> 0000000000000000
> [  127.068994] R10: 00007f3a69163ac0 R11: 0000000000000206 R12:
> 00007ffed553aa70
> [  127.068996] R13: 00007ffed553b84a R14: 00007f3a6a02e2a0 R15:
> 00007f3a6a02e7b0
> [  127.068999]  </TASK>
> [  127.069000] ---[ end trace 0000000000000000 ]---
> [  127.667264] irq 31: nobody cared (try booting with the "irqpoll" option)
> [  127.674758] CPU: 4 PID: 0 Comm: swapper/4 Tainted: G W        
> 5.17.0-rc3-tk #12
> [  127.674764] Hardware name: Deciso B.V. DEC2700 - OPNsense
> Appliance/Netboard-A10 Gen.3, BIOS 05.32.50.0012-A10.20 11/15/2021
> [  127.674766] Call Trace:
> [  127.674769]  <IRQ>
> [  127.674773]  dump_stack_lvl+0x4c/0x63
> [  127.674781]  dump_stack+0x10/0x12
> [  127.674784]  __report_bad_irq+0x3a/0xaf
> [  127.674789]  note_interrupt.cold+0xb/0x60
> [  127.674793]  ? __this_cpu_preempt_check+0x13/0x20
> [  127.674799]  handle_irq_event+0x71/0x80
> [  127.674805]  handle_fasteoi_irq+0x95/0x1e0
> [  127.674810]  __common_interrupt+0x6e/0x110
> [  127.674815]  common_interrupt+0xbd/0xe0
> [  127.674819]  </IRQ>
> [  127.674820]  <TASK>
> [  127.674822]  asm_common_interrupt+0x1e/0x40
> [  127.674826] RIP: 0010:cpuidle_enter_state+0xdf/0x380
> [  127.674834] Code: ff e8 25 76 73 ff 80 7d d7 00 74 17 9c 58 0f 1f 44
> 00 00 f6 c4 02 0f 85 82 02 00 00 31 ff e8 18 8c 7a ff fb 66 0f 1f 44 00
> 00 <45> 85 ff 0f 88 1a 01 00 00 49 63 d7 4c 89 f1 48 2b 4d c8 48 8d 04
> [  127.674837] RSP: 0018:ffffaf29400e3e68 EFLAGS: 00000246
> [  127.674841] RAX: ffff91fa6af00000 RBX: 0000000000000002 RCX:
> 000000000000001f
> [  127.674843] RDX: 0000000000000000 RSI: ffffffff845bf281 RDI:
> ffffffff845cddcf
> [  127.674845] RBP: ffffaf29400e3ea0 R08: 0000001db98fd21c R09:
> 0000001d7b8fd3fc
> [  127.674847] R10: 0000000000000001 R11: ffff91fa6af2fd84 R12:
> ffff91fa41de6c00
> [  127.674849] R13: ffffffff8506e4c0 R14: 0000001db98fd21c R15:
> 0000000000000002
> [  127.674854]  ? cpuidle_enter_state+0xbb/0x380
> [  127.674860]  cpuidle_enter+0x2e/0x40
> [  127.674864]  do_idle+0x203/0x290
> [  127.674869]  cpu_startup_entry+0x20/0x30
> [  127.674872]  start_secondary+0x118/0x150
> [  127.674877]  secondary_startup_64_no_verify+0xd5/0xdb
> [  127.674885]  </TASK>
> [  127.674886] handlers:
> [  127.677425] [<00000000b61e344c>] amd_mp2_irq_isr [i2c_amd_mp2_pci]
> [  127.684335] Disabling IRQ #31
> 
> and command line output after 'modprobe -vvv amd_xgbe':
> 
> jane@...740-ubu21:~$ sudo modprobe -vvv amd_xgbe
> modprobe: INFO: ../libkmod/libkmod.c:365 kmod_set_log_fn() custom
> logging function 0x7f74d79de780 registered
> modprobe: DEBUG: ../libkmod/libkmod-index.c:757 index_mm_open()
> file=/lib/modules/5.17.0-rc3-tk/modules.dep.bin
> modprobe: DEBUG: ../libkmod/libkmod-index.c:757 index_mm_open()
> file=/lib/modules/5.17.0-rc3-tk/modules.alias.bin
> modprobe: DEBUG: ../libkmod/libkmod-index.c:757 index_mm_open()
> file=/lib/modules/5.17.0-rc3-tk/modules.symbols.bin
> modprobe: DEBUG: ../libkmod/libkmod-index.c:757 index_mm_open()
> file=/lib/modules/5.17.0-rc3-tk/modules.builtin.alias.bin
> modprobe: DEBUG: ../libkmod/libkmod-index.c:757 index_mm_open()
> file=/lib/modules/5.17.0-rc3-tk/modules.builtin.bin
> modprobe: DEBUG: ../libkmod/libkmod-module.c:556
> kmod_module_new_from_lookup() input alias=amd_xgbe, normalized=amd_xgbe
> modprobe: DEBUG: ../libkmod/libkmod-module.c:562
> kmod_module_new_from_lookup() lookup modules.dep amd_xgbe
> modprobe: DEBUG: ../libkmod/libkmod.c:595 kmod_search_moddep() use
> mmaped index 'modules.dep' modname=amd_xgbe
> modprobe: DEBUG: ../libkmod/libkmod.c:403 kmod_pool_get_module() get
> module name='amd_xgbe' found=(nil)
> modprobe: DEBUG: ../libkmod/libkmod.c:411 kmod_pool_add_module() add
> 0x7f74d83862a0 key='amd_xgbe'
> modprobe: DEBUG: ../libkmod/libkmod-module.c:202
> kmod_module_parse_depline() 0 dependencies for amd_xgbe
> modprobe: DEBUG: ../libkmod/libkmod-module.c:589
> kmod_module_new_from_lookup() lookup amd_xgbe=0, list=0x7f74d8385c40
> modprobe: DEBUG: ../libkmod/libkmod.c:500 lookup_builtin_file() use
> mmaped index 'modules.builtin' modname=amd_xgbe
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1760
> kmod_module_get_initstate() could not open
> '/sys/module/amd_xgbe/initstate': No such file or directory
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1770
> kmod_module_get_initstate() could not open '/sys/module/amd_xgbe': No
> such file or directory
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=snd_pcsp mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=snd_usb_audio mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=cx88_alsa mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=snd_atiixp_modem mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=snd_intel8x0m mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=snd_via82xx_modem mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1409
> kmod_module_get_options() passed = modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=md_mod mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=bonding mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=dummy mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1409
> kmod_module_get_options() passed = modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1404
> kmod_module_get_options() modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1409
> kmod_module_get_options() passed = modname=amd_xgbe mod->name=amd_xgbe
> mod->alias=(null)
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1760
> kmod_module_get_initstate() could not open
> '/sys/module/amd_xgbe/initstate': No such file or directory
> modprobe: DEBUG: ../libkmod/libkmod-module.c:1770
> kmod_module_get_initstate() could not open '/sys/module/amd_xgbe': No
> such file or directory
> modprobe: DEBUG: ../libkmod/libkmod-module.c:750 kmod_module_get_path()
> name='amd_xgbe'
> path='/lib/modules/5.17.0-rc3-tk/kernel/drivers/net/ethernet/amd/xgbe/amd-xgbe.ko'
> 
> modprobe: DEBUG: ../libkmod/libkmod-module.c:750 kmod_module_get_path()
> name='amd_xgbe'
> path='/lib/modules/5.17.0-rc3-tk/kernel/drivers/net/ethernet/amd/xgbe/amd-xgbe.ko'
> 
> insmod
> /lib/modules/5.17.0-rc3-tk/kernel/drivers/net/ethernet/amd/xgbe/amd-xgbe.ko
> dyndbg="+pfm" debug=0x37 dyndbg=+p debug=0x37
> modprobe: DEBUG: ../libkmod/libkmod-module.c:750 kmod_module_get_path()
> name='amd_xgbe'
> path='/lib/modules/5.17.0-rc3-tk/kernel/drivers/net/ethernet/amd/xgbe/amd-xgbe.ko'
> 
> modprobe: DEBUG: ../libkmod/libkmod-module.c:468 kmod_module_unref()
> kmod_module 0x7f74d83862a0 released
> modprobe: DEBUG: ../libkmod/libkmod.c:419 kmod_pool_del_module() del
> 0x7f74d83862a0 key='amd_xgbe'
> modprobe: INFO: ../libkmod/libkmod.c:332 kmod_unref() context
> 0x7f74d83854c0 released
> 
> and the corresponding dmesg diff:
> 
> [  151.599892] amd-xgbe 0000:06:00.1 eth0: net device enabled
> [  151.601333] amd-xgbe 0000:06:00.2 eth1: net device enabled
> [  151.606044] amd-xgbe 0000:06:00.1 enp6s0f1: renamed from eth0
> [  151.646262] amd-xgbe 0000:06:00.2 enp6s0f2: renamed from eth1
> 
> 
> Cheers
> Thomas
> 

Powered by blists - more mailing lists