Message-ID: <20240407093931.GA28090@reaktio.net>
Date: Sun, 7 Apr 2024 12:39:31 +0300
From: Pasi Kärkkäinen <pasik@....fi>
To: Oleksandr Natalenko <oleksandr@...alenko.name>
Cc: linux-kernel@...r.kernel.org, Norbert Preining <norbert@...ining.info>,
	stable@...r.kernel.org, regressions@...mhuis.info
Subject: Re: Regression in kernel 6.8.2 fails in various ways (USB, BT, ...)

On Sun, Apr 07, 2024 at 12:04:46AM +0200, Oleksandr Natalenko wrote:
> Hello.
> 
> On sobota 6. dubna 2024 20:29:43, CEST Pasi Kärkkäinen wrote:
> > On Wed, Apr 03, 2024 at 09:02:21PM +0200, Oleksandr Natalenko wrote:
> > > Hello.
> > > 
> > > On úterý 2. dubna 2024 12:55:29, CEST Norbert Preining wrote:
> > > > Hi all
> > > > 
> > > > (please cc)
> > > > 
> > > > I am running Arch Linux on a Lenovo X1 Carbon Gen 10.
> > > > Tests are made with uptodate system, and besides the kernel no
> > > > difference.
> > > > The kernels are as distributed by Arch, but if necessary I can compile
> > > > locally.
> > > > 
> > > > 
> > > > Arch Linux kernel 6.8.1 works without any problems.
> > > > Upgrading to 6.8.2 breaks a lot of things:
> > > > 
> > > > * Plugging in my Yubikey C does not trigger any reaction
> > > >   (as a consequence scdaemon hangs)
> > > > * sending of bluetooth firmware data fails with Oops (see below)
> > > > * shutdown hangs and does not turn off the computer
> > > > 
> > > > 
> > > > I can repeat this behaviour on every reboot into 6.8.2.
> > > > 
> > > > I have dmesg/journalctl -b for both kernels collected, if required, I
> > > > can send them. I checked the diffs but nothing really did stand out.
> > > > 
> > > > Yubikey plugin:
> > > > On 6.8.1 when I plug in the yubikey it lights up, and in the logs I see:
> > > > [  370.767739] usb 3-1: new full-speed USB device number 5 using xhci_hcd
> > > > [  370.910392] usb 3-1: New USB device found, idVendor=1050, idProduct=0407, bcdDevice= 5.43
> > > > [  370.910403] usb 3-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
> > > > [  370.910407] usb 3-1: Product: YubiKey OTP+FIDO+CCID
> > > > [  370.910409] usb 3-1: Manufacturer: Yubico
> > > > [  371.497496] input: Yubico YubiKey OTP+FIDO+CCID as /devices/pci0000:00/0000:00:14.0/usb3/3-1/3-1:1.0/0003:1050:0407.0005/input/input21
> > > > [  371.555342] hid-generic 0003:1050:0407.0005: input,hidraw4: USB HID v1.10 Keyboard [Yubico YubiKey OTP+FIDO+CCID] on usb-0000:00:14.0-1/input0
> > > > [  371.557021] hid-generic 0003:1050:0407.0006: hiddev96,hidraw5: USB HID v1.10 Device [Yubico YubiKey OTP+FIDO+CCID] on usb-0000:00:14.0-1/input1
> > > > [  371.557145] usbcore: registered new interface driver usbhid
> > > > [  371.557151] usbhid: USB HID core driver
> > > > 
> > > > On 6.8.2 not short blinking, and nothing in the logs.
> > > > 
> > > > 
> > > > 
> > > > Bluetooth Ooops:
> > > >    14.271713] usb usb3-port10: disabled by hub (EMI?), re-enabling...
> > > > [   14.271725] usb 3-10: USB disconnect, device number 4
> > > > [   14.271893] Bluetooth: hci0: Failed to send firmware data (-19)
> > > > [   14.271953] Bluetooth: hci0: sending frame failed (-19)
> > > > [   14.271976] Bluetooth: hci0: Intel reset sent to retry FW download
> > > > [   14.427765] Bluetooth: hci0: sending frame failed (-19)
> > > > [   14.427811] BUG: kernel NULL pointer dereference, address: 0000000000000070
> > > > [   14.427814] #PF: supervisor read access in kernel mode
> > > > [   14.427815] #PF: error_code(0x0000) - not-present page
> > > > [   14.427816] PGD 0 P4D 0
> > > > [   14.427819] Oops: 0000 [#1] PREEMPT SMP NOPTI
> > > > [   14.427820] CPU: 0 PID: 198 Comm: kworker/u41:0 Tainted: G           OE      6.8.2-arch2-1 #1 a430fb92f7ba43092b62bbe6bac995458d3d442d
> > > > [   14.427823] Hardware name: LENOVO 21CBCTO1WW/21CBCTO1WW, BIOS N3AET80W (1.45 ) 02/20/2024
> > > > [   14.427824] Workqueue: hci0 hci_power_on [bluetooth]
> > > > [   14.427874] RIP: 0010:btintel_read_debug_features+0x4d/0xf0 [btintel]
> > > > [   14.427880] Code: 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 48 8d 4c 24 07 c6 44 24 07 01 e8 de a9 ef ff 48 89 c3 48 3d 00 f0 ff ff 77 49 <83> 78 70 13 75 67 48 8b 80 d0 00 00 00 be 02 00 00 00 48 89 df 48
> > > > [   14.427881] RSP: 0018:ffffb7ba80aa3cc0 EFLAGS: 00010207
> > > > [   14.427882] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff965ac0ddaab0
> > > > [   14.427884] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff965ac0ddaaa8
> > > > [   14.427884] RBP: ffffb7ba80aa3cf0 R08: ffff965ac0ddaab0 R09: 0000000000000000
> > > > [   14.427885] R10: 0000000000000001 R11: 0000000000000100 R12: ffff965ac0dda000
> > > > [   14.427886] R13: ffff965a84746900 R14: ffff965a8208ca05 R15: ffff965ac0dda6d0
> > > > [   14.427887] FS:  0000000000000000(0000) GS:ffff9661bf400000(0000) knlGS:0000000000000000
> > > > [   14.427888] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   14.427889] CR2: 0000000000000070 CR3: 00000002bb620000 CR4: 0000000000f50ef0
> > > > [   14.427889] PKRU: 55555554
> > > > [   14.427890] Call Trace:
> > > > [   14.427891]  <TASK>
> > > > [   14.427893]  ? __die+0x23/0x70
> > > > [   14.427898]  ? page_fault_oops+0x171/0x4e0
> > > > [   14.427901]  ? __timer_delete_sync+0x7d/0xe0
> > > > [   14.427905]  ? exc_page_fault+0x7f/0x180
> > > > [   14.427908]  ? asm_exc_page_fault+0x26/0x30
> > > > [   14.427912]  ? btintel_read_debug_features+0x4d/0xf0 [btintel 6735e787c9ed982ceaa471c7ab364a390a1acb37]
> > > > [   14.427916]  btintel_register_devcoredump_support.isra.0+0x3e/0x110 [btintel 6735e787c9ed982ceaa471c7ab364a390a1acb37]
> > > > [   14.427921]  btintel_setup_combined+0x503/0x790 [btintel 6735e787c9ed982ceaa471c7ab364a390a1acb37]
> > > > [   14.427925]  hci_dev_open_sync+0x102/0xc20 [bluetooth 81a2e5f6c3a85f38dade670bfe91d861a9119613]
> > > > [   14.427953]  ? __schedule+0x3ee/0x1520
> > > > [   14.427956]  hci_dev_do_open+0x23/0x60 [bluetooth 81a2e5f6c3a85f38dade670bfe91d861a9119613]
> > > > [   14.427975]  hci_power_on+0x51/0x260 [bluetooth 81a2e5f6c3a85f38dade670bfe91d861a9119613]
> > > > [   14.427994]  process_one_work+0x183/0x370
> > > > [   14.427998]  worker_thread+0x3ab/0x4f0
> > > > [   14.428000]  ? __pfx_worker_thread+0x10/0x10
> > > > [   14.428001]  kthread+0xe5/0x120
> > > > [   14.428005]  ? __pfx_kthread+0x10/0x10
> > > > [   14.428006]  ret_from_fork+0x31/0x50
> > > > [   14.428009]  ? __pfx_kthread+0x10/0x10
> > > > [   14.428011]  ret_from_fork_asm+0x1b/0x30
> > > > [   14.428014]  </TASK>
> > > > [   14.428015] Modules linked in: nf_tables bnep tun btusb btrtl btintel btbcm btmtk bluetooth gpio_ljca i2c_ljca ecdh_generic hid_sensor_custom_intel_hinge hid_sensor_trigger industrialio_triggered_buffer kfifo_buf hid_sensor_iio_common industrialio hid_sensor_custom hid_sensor_hub intel_ishtp_hid joydev vfat fat ext4 crc16 mbcache jbd2 snd_ctl_led snd_soc_skl_hda_dsp snd_soc_hdac_hdmi snd_soc_intel_hda_dsp_common snd_sof_probes hid_multitouch hid_generic mousedev snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic snd_soc_dmic snd_sof_pci_intel_tgl snd_sof_intel_hda_common soundwire_intel snd_sof_intel_hda_mlink soundwire_cadence snd_sof_intel_hda snd_sof_pci snd_sof_xtensa_dsp snd_sof snd_sof_utils snd_soc_hdac_hda snd_hda_ext_core snd_soc_acpi_intel_match intel_uncore_frequency intel_uncore_frequency_common snd_soc_acpi intel_tcc_cooling iwlmvm soundwire_generic_allocation soundwire_bus x86_pkg_temp_thermal intel_powerclamp snd_soc_core mac80211 coretemp snd_compress ac97_bus snd_pcm_dmaengine kvm_intel
> > > > [   14.428048]  snd_hda_intel libarc4 ptp snd_intel_dspcfg pps_core snd_intel_sdw_acpi snd_hda_codec processor_thermal_device_pci kvm processor_thermal_device processor_thermal_wt_hint snd_hda_core processor_thermal_rfim iwlwifi iTCO_wdt irqbypass intel_rapl_msr processor_thermal_rapl mei_pxp snd_hwdep ucsi_acpi mei_hdcp mei_wdt rapl intel_rapl_common intel_pmc_bxt typec_ucsi iTCO_vendor_support intel_lpss_pci intel_cstate thinkpad_acpi mei_me think_lmi snd_pcm spi_nor processor_thermal_wt_req intel_ish_ipc typec ntfs3 i2c_i801 intel_lpss nxp_nci_i2c ledtrig_audio processor_thermal_power_floor cfg80211 wmi_bmof intel_uncore psmouse pcspkr firmware_attributes_class mtd thunderbolt mei snd_timer idma64 intel_ishtp i2c_smbus ov2740 roles platform_profile igen6_edac processor_thermal_mbox nxp_nci v4l2_fwnode nci snd intel_skl_int3472_tps68470 int3403_thermal v4l2_async intel_pmc_core soc_button_array tps68470_regulator nfc soundcore int340x_thermal_zone clk_tps68470 mei_vsc_hw rfkill videodev intel_vsec i2c_hid_acpi
> > > > [   14.428085]  intel_hid int3400_thermal i2c_hid pmt_telemetry mc pinctrl_tigerlake sparse_keymap acpi_pad acpi_thermal_rel intel_skl_int3472_discrete acpi_tad pmt_class mac_hid i2c_dev corefreqk(OE) sg crypto_user fuse loop nfnetlink ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted asn1_encoder tee dm_mod xe drm_ttm_helper gpu_sched drm_suballoc_helper drm_gpuvm drm_exec spi_ljca usb_ljca i915 crct10dif_pclmul crc32_pclmul crc32c_intel polyval_clmulni polyval_generic gf128mul ghash_clmulni_intel sha512_ssse3 serio_raw sha256_ssse3 atkbd i2c_algo_bit sha1_ssse3 drm_buddy libps2 vivaldi_fmap aesni_intel ttm nvme intel_gtt crypto_simd nvme_core cryptd drm_display_helper video xhci_pci spi_intel_pci i8042 nvme_auth xhci_pci_renesas spi_intel cec serio wmi
> > > > [   14.428121] CR2: 0000000000000070
> > > > [   14.428123] ---[ end trace 0000000000000000 ]---
> > > > [   14.428124] RIP: 0010:btintel_read_debug_features+0x4d/0xf0 [btintel]
> > > > [   14.428128] Code: 65 48 8b 04 25 28 00 00 00 48 89 44 24 08 31 c0 48 8d 4c 24 07 c6 44 24 07 01 e8 de a9 ef ff 48 89 c3 48 3d 00 f0 ff ff 77 49 <83> 78 70 13 75 67 48 8b 80 d0 00 00 00 be 02 00 00 00 48 89 df 48
> > > > [   14.428130] RSP: 0018:ffffb7ba80aa3cc0 EFLAGS: 00010207
> > > > [   14.428131] RAX: 0000000000000000 RBX: 0000000000000000 RCX: ffff965ac0ddaab0
> > > > [   14.428131] RDX: 0000000000000000 RSI: 0000000000000202 RDI: ffff965ac0ddaaa8
> > > > [   14.428132] RBP: ffffb7ba80aa3cf0 R08: ffff965ac0ddaab0 R09: 0000000000000000
> > > > [   14.428133] R10: 0000000000000001 R11: 0000000000000100 R12: ffff965ac0dda000
> > > > [   14.428133] R13: ffff965a84746900 R14: ffff965a8208ca05 R15: ffff965ac0dda6d0
> > > > [   14.428134] FS:  0000000000000000(0000) GS:ffff9661bf400000(0000) knlGS:0000000000000000
> > > > [   14.428135] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > > > [   14.428136] CR2: 0000000000000070 CR3: 00000002bb620000 CR4: 0000000000f50ef0
> > > > [   14.428137] PKRU: 55555554
> > > > [   14.428137] note: kworker/u41:0[198] exited with irqs disabled
> > > > [   16.434794] Bluetooth: hci0: command 0xfc09 tx timeout
> > > > 
> > > > 
> > > > Thanks for any suggestions and best regards
> > > > (and please Cc)
> > > > 
> > > > Norbert
> > > > 
> > > > --
> > > > PREINING Norbert                              https://www.preining.info
> > > > arXiv / Cornell University   +   IFMGA Guide   +   TU Wien  +  TeX Live
> > > > GPG: 0x860CDC13   fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
> > > > 
> > > > 
> > > 
> > > The BT woe should be fixed [1] in v6.8.3.
> > > 
> > > For other two things, would you be able to bisect?
> > > 
> > 
> > I noticed possible related USB problem/regression also with Linux 6.7.11, eg. my USB headset is not detected at all anymore when running Linux 6.7.11. USB headset works OK with Linux 6.7.9, so the regression is included/backported to also either in 6.7.10 or in 6.7.11.
> 
> It wound be nice to narrow this range down even further with bisection because there are too many changes between v6.7.9 and v6.7.11 that could cause this.
> 

I just tested more Fedora kernel rpms, and it seems Linux 6.7.10 still works OK,
but 6.7.11 has the regression (eg. USB devices not detected, laptop does not reboot 
as it gets stuck before resetting the machine).

So the regression was backported to Linux 6.7.11.


Thanks,

-- Pasi

Powered by blists - more mailing lists