Message-ID: <20250508051044.GA3689439@quokka>
Date: Thu, 8 May 2025 15:10:44 +1000
From: Peter Hutterer <peter.hutterer@...-t.net>
To: "Russell King (Oracle)" <linux@...linux.org.uk>
Cc: Dmitry Torokhov <dmitry.torokhov@...il.com>,
	Maxime Chevallier <maxime.chevallier@...tlin.com>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	Andrew Lunn <andrew@...n.ch>,
	Woojung Huh <woojung.huh@...rochip.com>,
	Vladimir Oltean <olteanv@...il.com>,
	Heiner Kallweit <hkallweit1@...il.com>,
	"David S. Miller" <davem@...emloft.net>,
	Eric Dumazet <edumazet@...gle.com>,
	Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
	linux-input@...r.kernel.org, linux-kernel@...r.kernel.org,
	Benjamin Tissoires <benjamin.tissoires@...hat.com>
Subject: Re: [BUG] Stuck key syndrome (was: Re: [PATCH net-next v2] net: dsa:
 microchip: Add SGMII port support to KSZ9477 switch)

On Wed, May 07, 2025 at 07:18:37PM +0100, Russell King (Oracle) wrote:
> On Wed, May 07, 2025 at 10:45:48AM -0700, Dmitry Torokhov wrote:
> > On Wed, May 07, 2025 at 01:51:26PM +0200, Maxime Chevallier wrote:
> > > On Wed, 7 May 2025 12:44:24 +0100
> > > "Russell King (Oracle)" <linux@...linux.org.uk> wrote:
> > > 
> > > > Hi Maxime,
> > > > 
> > > > On Wed, May 07, 2025 at 03:32:36PM +0200, Maxime Chevallier wrote:
> > > > > Hi Russell,
> > > > > 
> > > > > On Wed, 7 May 2025 10:59:21 +0100
> > > > > "Russell King (Oracle)" <linux@...linux.org.uk> wrote:
> > > > >   
> > > > > > On Wed, May 07, 2025 at 10:23:17AM +0100, Russell King (Oracle) wrote:  
> > > > > > > [Sorry for going off topic here - changed the Cc list, added Linus,
> > > > > > > changed the subject.]
> > > > > > > 
> > > > > > > On Wed, May 07, 2025 at 10:54:57AM +0200, Maxime Chevallier wrote:    
> > > > > > > > On Wed, 7 May 2025 09:31:48 +0100
> > > > > > > > "Russell King (Oracle)" <linux@...linux.org.uk> wrote:    
> > > > > > > > > [rest of the email got deleted because Linux / X11 / KDE got confused
> > > > > > > > > about the state the backspace key and decided it was going to be
> > > > > > > > > continuously pressed and doing nothing except shutting the laptop
> > > > > > > > > down would stop it.]    
> > > > > > > > 
> > > > > > > > Funny how I have the same exact issue on my laptop as well...     
> > > > > > > 
> > > > > > > I've had the "stuck key" behaviour with the HP Pavilion 15-au185sa
> > > > > > > laptop I had previously (normally with ctrl-F keys). However, hitting
> > > > > > > ctrl/shift/alt would stop it.
> > > > > > > 
> > > > > > > This is the first time I've seen the behaviour with the Carbon X1
> > > > > > > laptop, but this was way more severe. No key would stop it. Trying to
> > > > > > > move the focus using the trackpad/nipple had any effect. Meanwhile
> > > > > > > the email was being deleted one character at a time. So I shut the
> > 
> > If we indeed lost a key release event somewhere the way to "restore" it
> > is to hit the stuck key again. Then we should get press/release with
> > press most likely being ignored and release achieving the desired
> > result. Of course that will not help if embedded controller is confused.
> 
> I tried pressing every key...

ftr, any key autorepeat in XKB (or libxkbcommon) is cancelled when
another key arrives so this does indeed hint at a firmware/controller
issue.

> > > > > > The mysterious thing is "Keylock active" - clearly it isn't because I
> > > > > > can write this email typing on that very keyboard. However, I wonder
> > > > > > if it needs i8042_unlock=1 to set I8042_CTR_IGNKEYLOCK.
> > 
> > Just ignore this message, it is harmless and trying to flip the bit
> > might confuse the emulation even more. Maybe we should lower the
> > severity of it to debug.
> > 
> > That said I do not see it on my Carbon (neither v5 nor v12, can't check
> > v9 because it is at home)... What version of Carbon do you have? Do you
> > have up-to-date BIOS/EC?
> 
> Neither did I see a problem until today, and I've been using the laptop
> since October 2024, and this is the first time it's had an issue.
> 
> Looking at fwupd, it has an Intel ME update pending (1.32.2418 to
> 1.35.2557). I can't find a way to get any update history beyond
> that out of fwupd and fwupd doesn't seem to log to journald what
> it's doing.
> 
> > > > > It just happened to me as I was typing this very email (key 'd' got
> > > > > stuck, nothing could un-stick it, couldn't move the mouse cursor but
> > > > > mouse-click events did work, had to suspend/resume the laptop to fix
> > > > > that)
> > 
> > This is weird and suggests that the breakage happens up the stack from
> > the kernel (or down in the firmware). Mouse clicks and mouse movement is
> > delivered as part of a mouse packet, so if there are button clicks there
> > will also be movement, they are not separate. If the cursor is not
> > reacting that means desktop environment is not handling input properly.
> 
> So could we be looking at an Xorg bug?
> 
> > The kernel does drop input events if userspace is unable to read buffers
> > quickly enough. It notifies userspace by queuing special
> > EV_SYN/SYN_DROPPED event and userspace is supposed to query the full
> > device state upon receiving it to figure out what to do. I doubt we are
> > running into this with keyboards, but maybe we should add some logging
> > there to rule this out.
> > 
> > I'll add Peter and Benjamin to this thread in case they have ideas.
> 
> I'm thinking of leaving evtest running in a terminal, so its output
> can be inspected if it happens again. One issue though is the
> timestamps aren't readable, but I'm sure with a bit of perl
> post-processing that could be fixed.

$ screen sudo libinput record --autorestart=5 -o keyboard.yml

Select the keyboard devce and off you go.
This will create a keyboard.yml.$DATETIME file and after 5 seconds of no
events it will rotate to the next file. This means you can leave this
running for months and when it happens you stop typing for 5 seconds and
then look at the most recent (usually) two files for epiphanies.
Bonus points for ssh-ing in and killing the process remotely because
then you don't have interference of whatever

If the last event you get is a key down event the issue is in the
firmware and userspace does the right thing. In that case your 
logs will be full of EV_KEY KEY_FOO with a value of 2 (kernel repeat).

If the last event you get is a key up but the desktop still repeats
then the issue is in that part of the stack. In that case the log should
be empty after you stopped typing.

If you get a SYN_DROPPED and the desktop does the wrong thing the issue
is in libinput because it should handle that transparently (same with
the Xorg evdev driver if you're using that).

Optionally: --show-keycodes but adding this means you will leak
all your key presses into the logs (without this all alphanumeric keys
are recorded as KEY_A). So maybe only use that for entertainment value.

Optionally: provide the device node as arg (see below)

> That would allow an answer to "is it kernel or firmware" vs
> "userspace".
> 
> The problem is - if it's taken 7-ish months to show for me, it's
> likely that evtest won't be running when it next happens (as there
> will be needs to reboot for kernel upgrades/firmware upgrades in
> that time.) Really, it needs to be something like an automatically
> started at boot e.g. evtest inside a detached screen session.
 
Well, that bit I leave to you to figure out because it'll also require
scripting to find the keyboard's device node automatically so you can
pass it to libinput record via the .unit file, or cron job script, or
whatever.

Either way, this should at least make it easier to catch the issue when
it occurs.

Cheers,
  Peter

Powered by blists - more mailing lists