netdev - 6.9.3 Hung tasks

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [thread-next>] [day] [month] [year] [list]

Message-ID: <9d189ec329cfe68ed68699f314e191a10d4b5eda.camel@sapience.com>
Date: Thu, 30 May 2024 08:53:17 -0400
From: Genes Lists <lists@...ience.com>
To: linux-kernel@...r.kernel.org
Cc: netdev@...r.kernel.org, andrew@...n.ch, hkallweit1@...il.com, 
 linux@...linux.org.uk, davem@...emloft.net, edumazet@...gle.com,
 kuba@...nel.org,  pabeni@...hat.com
Subject: 6.9.3 Hung tasks

I have one server which has failed to boot twice in the same way :
    once on 6.9 and again on 6.9.3
I have not seen this problem in any earlier kernels.

In both cases the log shows same problems. Several hung tasks including

  drivers/net/phy/phy_led_triggers.c
  drivers/net/phy/phy_device.c

Machine has
    - 32 GB mem
    - 2 nvme and 6 spinners in raid-6 with lvmcache from nvme
partition.
    - RTL8125 ethernet  
    - Raptor Lake-S PCH CNVi WiFi unused (not configured at all)    

The first 2 hung tasks in log are kworker:
  Workqueue: events linkwatch_event
  __schedule+0x3c7/0x1510

and lower down
  reg_check_chans_work+0x31/0x5a0 [cfg80211
4357c1c638dc9d54269e11808818932f9a7b7e79

the next is systemd-network which shows:
  led_trigger_register+0x118/0x190
  phy_led_triggers_register+0xee/0x260
  phy_attach_direct+0x363/0x370
...



I apologize in advance, since this only happens on one production
server, and very infrequently,
I will not be able to take this offline to git bisect.  Nonetheless,
hopefully the logs will be useful.

More details attached including journal from bad and good boot.

-- 
Gene


View attachment "lspci.out" of type "text/plain" (2163 bytes)

View attachment "hung-tasks" of type "text/plain" (1070 bytes)

View attachment "block-devices" of type "text/plain" (327 bytes)

View attachment "6.9.3-hang.good" of type "text/plain" (108737 bytes)

View attachment "6.9.3-hang.bad" of type "text/plain" (128095 bytes)

Download attachment "signature.asc" of type "application/pgp-signature" (229 bytes)