Message-ID: <1e13c61d-8581-4ece-b31c-7aa771ba7bc2@amazon.com>
Date: Tue, 3 Feb 2026 13:36:30 +0100
From: Alexander Graf <graf@...zon.com>
To: Thomas Gleixner <tglx@...nel.org>, <x86@...nel.org>
CC: <linux-kernel@...r.kernel.org>, <linux-doc@...r.kernel.org>, "Clemens
 Ladisch" <clemens@...isch.de>, Arnd Bergmann <arnd@...db.de>, "Greg
 Kroah-Hartman" <gregkh@...uxfoundation.org>, Dave Hansen
	<dave.hansen@...ux.intel.com>, Borislav Petkov <bp@...en8.de>, Ingo Molnar
	<mingo@...hat.com>, Jonathan Corbet <corbet@....net>, Paolo Bonzini
	<pbonzini@...hat.com>, Pasha Tatashin <pasha.tatashin@...een.com>,
	<nh-open-source@...zon.com>, Nicolas Saenz Julienne <nsaenz@...zon.es>,
	Hendrik Borghorst <hborghor@...zon.de>, Filippo Sironi <sironi@...zon.de>,
	David Woodhouse <dwmw@...zon.co.uk>, Jan Schönherr
	<jschoenh@...zon.de>, <ricardo.neri-calderon@...ux.intel.com>, Sasha Levin
	<sashal@...nel.org>
Subject: Re: [PATCH 2/2] hpet: Add HPET-based NMI watchdog support


On 03.02.26 11:32, Thomas Gleixner wrote:
> On Mon, Feb 02 2026 at 17:48, Alexander Graf wrote:
>> (Disclaimer: Some of this code was written with the help of Kiro, an AI
>> coding assistant)
> You could have sent your change log through AI too so it conforms with
> the change log rules ...


Maybe we should introduce an AGENTS.md file in Linux that tells the AI 
tool to do that automatically? These tools usually don't read README 
files. :)

Looks like - similar to the HPET watchdog - that never concluded though:

https://lore.kernel.org/lkml/20250813203647.06e49600@gandalf.local.home/

Sasha, are you going to resend your @README commit with a single 
AGENTS.md? FWIW that is pretty much what everything standardized on by now.


>
>> +#ifdef CONFIG_HARDLOCKUP_DETECTOR_HPET
>> +/*
>> + * HPET watchdog uses timer 0 routed to GSI 2 (legacy PIT IRQ line).
>> + * When using HPET as watchdog, we repurpose this line for NMI delivery.
>> + */
>> +#define HPET_WD_TIMER        0
>> +#define HPET_WD_GSI  2
>> +
>> +bool hpet_watchdog_initialized;
>> +static bool hpet_watchdog_ioapic_configured;
>> +static DEFINE_PER_CPU(u32, hpet_watchdog_next_tick);
>> +
>> +static int hpet_nmi_handler(unsigned int cmd, struct pt_regs *regs)
>> +{
>> +     u32 now, next, delta;
>> +
>> +     if (panic_in_progress())
>> +             return NMI_HANDLED;
>> +
>> +     /* Check if this NMI is from our HPET timer by comparing counter value */
>> +     now = hpet_readl(HPET_COUNTER);
> And both you and your AI assistant failed to read through the previous
> discussions on that topic and the 10+ failed attempts to make it work
> correctly.  Otherwise you would have figured out that reading HPET in
> the NMI handler is a patently bad idea.
>
> I'm not reiterating any of it as it's well documented in the LKML archive.


Thanks a bunch for the pointer. I had indeed missed the previous patch 
set submissions on the same topic. Those look a lot more sophisticated 
than the quick hacky version I built. Nice! Oh well, at least I 
(re)learned a few things about the HPET along the way.

Looking at the latest submission [1] (v7), I see patches but no reviews, 
no acks and no merges. Those patches also seem to address most of your 
concerns (obviously, since you reviewed them before :)). Reading the 
side conversation about it [2], it sounds like the buddy hardlockup 
detector is trying to fill the same gap as the HPET one and hence after 
that got merged, interest faded?

Let me reply the the other comments below regardless. Feel free to 
ignore - the conversation should move towards either the buddy or 
Ricardo's patch set.


[1] 
https://lore.kernel.org/lkml/20230413035844.GA31620@ranerica-svr.sc.intel.com/
[2] https://lore.kernel.org/lkml/ZFfb%2FbTi22RQwaol@tassilo/


>
>> +/*
>> + * On suspend, clear the configured flag so that the first CPU to come
>> + * online after resume will reconfigure the HPET timer and IO-APIC.
>> + *
>> + * We don't need to explicitly disable the watchdog here because:
>> + * 1. The HPET registers are reset by the hibernation/suspend process anyway
>> + * 2. The IO-APIC state is saved/restored by ioapic_syscore_ops, but we
>> + *    need to reconfigure it for NMI delivery after resume
> If it's saved/restored then what needs to be reconfigured?


I wasn't sure how much of the register state really gets saved/restored, 
especially in the HPET in both S3 and S4. So I figured I'd go the safe 
route and reprogram on resume always.


>
>> +static int __init hpet_watchdog_init(u32 channels)
>> +{
>> +     u32 cfg, i, route_cap;
>> +
>> +     if (channels <= HPET_WD_TIMER)
>> +             return 0;
>> +
>> +     /* Verify GSI 2 is available in the route capability bitmap */
> The legacy channels are always routed to GSIs. Why do you need GSI2?


2 because it's the usual HPET destination GSI, so I don't need to try 
and find an empty GSI.


> But why do you need to hijack the legacy 0 channel in the first place?
> As discussed before this can nicely use one of the extra channels (>2)
> which are available on any modern HPET implementation.


Mostly lazyness. I did not want to have to worry about implications of 
multiple components and subsystem (among which we expose bits to user 
space) can mess with the HPET at the same time, so I wanted it dedicated 
to the watchdog. But of course, we can absolutely share it if done 
cautiously. And then use a higher timer.


>
>> +     route_cap = hpet_readl(HPET_Tn_CFG(HPET_WD_TIMER) + 4);
>> +     if (!(route_cap & (1 << HPET_WD_GSI))) {
>> +             pr_info("HPET timer 0 cannot route to GSI %d\n", HPET_WD_GSI);
>> +             return 0;
>> +     }
>> +
>> +     /* Deactivate all timers */
>> +     for (i = 0; i < channels; i++) {
>> +             cfg = hpet_readl(HPET_Tn_CFG(i));
>> +             cfg &= ~(HPET_TN_ENABLE | HPET_TN_LEVEL | HPET_TN_FSB);
>> +             hpet_writel(cfg, HPET_Tn_CFG(i));
>> +     }
>> +
>> +     /* Configure HPET timer for periodic mode */
>> +     cfg = hpet_readl(HPET_Tn_CFG(HPET_WD_TIMER));
>> +     cfg &= ~(HPET_TN_ENABLE | HPET_TN_FSB);
>> +     cfg |= HPET_TN_PERIODIC | HPET_TN_32BIT | HPET_TN_SETVAL | HPET_TN_LEVEL;
> The HPET specification says about HPET_TN_LEVEL:
>
>     "The timer interrupt is level triggered. This means that a level-
>      triggered interrupt is generated. The interrupt will be held active until
>      it is cleared by writing to the bit in the General Interrupt Status
>      Register."
>
> This clearly has seen a lot of testing on real hardware.


Yikes, The TN_LEVEL slipped in last minute and I apparently did not 
properly revert it. This obviously needs to be edge triggered.


>
>> +     hpet_writel(cfg, HPET_Tn_CFG(HPET_WD_TIMER));
>> +
>> +     /* Route HPET timer to the GSI */
>> +     cfg = hpet_readl(HPET_Tn_CFG(HPET_WD_TIMER));
>> +     cfg &= ~(Tn_INT_ROUTE_CNF_MASK | HPET_CFG_ENABLE);
>> +     cfg |= (HPET_WD_GSI << Tn_INT_ROUTE_CNF_SHIFT) & Tn_INT_ROUTE_CNF_MASK;
>> +     hpet_writel(cfg, HPET_Tn_CFG(HPET_WD_TIMER));
> You need all of this muck because you did a shortcut in hpet_enable()
> which takes care of most things already. The previous attempts on this
> clearly took some effort to integrate this cleanly w/o duplicating code
> and introducing new bugs all over the place.
>
>> +void watchdog_hardlockup_enable(unsigned int cpu)
>> +{
>> +     if (!hpet_watchdog_ioapic_configured) {
>> +             /*
>> +              * First CPU online after resume - reconfigure HPET timer.
>> +              * This also sets hpet_watchdog_ioapic_configured = true.
>> +              */
>> +             watchdog_hardlockup_start();
>> +     }
>> +
>> +     if (num_online_cpus() == num_present_cpus()) {
>> +             ioapic_set_nmi(HPET_WD_GSI, true);
>> +             pr_info("switched to broadcast mode (all %d CPUs online)\n",
>> +                     num_online_cpus());
>> +     }
>> +}
>> +
>> +void watchdog_hardlockup_disable(unsigned int cpu)
>> +{
>> +     if (num_online_cpus() < num_present_cpus()) {
>> +             ioapic_set_nmi(HPET_WD_GSI, false);
>> +             pr_info("switched to CPU 0 only (%d CPUs online)\n",
>> +                     num_online_cpus() - 1);
> That's a truly useful lockup detector, which only runs on
> CPU0. Seriously?


I wanted to have a fully functional one with broadcast in the 
all-CPUs-online case. I was considering anything where not everything is 
online as more of a transitionary phase. Now, I see your argument on 
SMT=off. But if the other HPET patch set is not dead, maybe we could 
combine approaches and move to a broadcast mode when all CPUs are 
online, instead of the round robin? Not sure it's really a significant 
improvement though.


>
>> +     }
>> +}
>> +
>> +int __init watchdog_hardlockup_probe(void)
>> +{
>> +     return hpet_watchdog_mode ? 0 : -ENODEV;
>> +}
>> +#else
>> +static inline int hpet_watchdog_init(u32 channels) { return 0; }
>> +#endif /* CONFIG_HARDLOCKUP_DETECTOR_HPET */
>> +
>>   /**
>>    * hpet_enable - Try to setup the HPET timer. Returns 1 on success.
>>    */
>> @@ -1031,6 +1232,10 @@ int __init hpet_enable(void)
>>        /* This is the HPET channel number which is zero based */
>>        channels = ((id & HPET_ID_NUMBER) >> HPET_ID_NUMBER_SHIFT) + 1;
>>
>> +     /* If watchdog mode, hand off to watchdog driver */
>> +     if (hpet_watchdog_mode)
>> +             return hpet_watchdog_init(channels);
> And if that initialization fails for whatever reason the HPET is
> disfunct, but then all your hpet_is_watchdog() checks are false too and
> e.g. hpet_late_init() will fall flat on its nose.
>
>>        /*
>>         * The legacy routing mode needs at least two channels, tick timer
>>         * and the rtc emulation channel.
>> @@ -1122,6 +1327,9 @@ static __init int hpet_late_init(void)
>>   {
>>        int ret;
>>
>> +     if (hpet_is_watchdog())
>> +             return -ENODEV;
>> +
>>   #include <asm/hypervisor.h>
>>   #include <asm/apic.h>
>> @@ -31,6 +32,14 @@ struct clock_event_device *global_clock_event;
>>    */
>>   static bool __init use_pit(void)
>>   {
>> +     if (hpet_is_watchdog()) {
>> +             /*
>> +              * The PIT overlaps the HPET IRQ line which we configure to
>> +              * NMI in watchdog mode, rendering the PIT non functional.
>> +              */
>> +             return false;
>> +     }
> So your approach of enabling the HPET watchdog brute force on the
> command line ends up here because hpet_enable() returns 0. So now if
> apic_needs_pit() is true, then this unconditional enable results in a
> full boot fail.
> This clearly has been made "work" by the throw enough stuff at the wall
> and see what sticks approach.
>
> As it had been discussed before:
>
>     1) There is no reason to hijack channel 0 as this can be made work
>        nicely with the extra channels above channel 2 and MSI delivery
>
>     2) HPET read in the NMI handler is not going to happen and can be
>        solved by other means. A mostly working implementation exists
>        already in the mail archive.
>
>     3) Restricting it to CPU0 when not all CPUs are online is a
>        nonstarter. Think smt=off. Again, solutions for this have been
>        discussed and implemented.
>
>     4) Side channels into the interrupt configuration are not an option.
>        That has been properly integrated before...
>
> I'm definitely not impressed by this AI slop...


Like with any tool, the AI is only as good as its puppeteer :). Thanks 
for the insights! Super helpful. The most important one was the pointer 
to the existing patch set that I had completely missed.

At the end of the day, the end motivation is to get that one PMC back. 
Anything to make that happen works. I'll have a look at the buddy 
detector as well.


Thanks!

Alex




Amazon Web Services Development Center Germany GmbH
Tamara-Danz-Str. 13
10243 Berlin
Geschaeftsfuehrung: Christof Hellmis, Andreas Stieger
Eingetragen am Amtsgericht Charlottenburg unter HRB 257764 B
Sitz: Berlin
Ust-ID: DE 365 538 597

Powered by blists - more mailing lists