Message-ID: <Z1v9GKkghPjpnvp6@MBC02GN1V4Q05P>
Date: Fri, 13 Dec 2024 17:27:02 +0800
From: Richard Clark <richard.xnu.clark@...il.com>
To: Mark Kettenis <mark.kettenis@...all.nl>
Cc: Marc Zyngier <maz@...nel.org>, linux-arm-kernel@...ts.infradead.org,
	linux-kernel@...r.kernel.org, will@...nel.org,
	linux@...linux.org.uk, mark.rutland@....com,
	torvalds@...ux-foundation.org
Subject: Re: Question about interrupt prioriyt of ARM GICv3/4

Hi Mark,

On Thu, Dec 12, 2024 at 02:02:45PM +0100, Mark Kettenis wrote:
> > Date: Thu, 12 Dec 2024 10:12:32 +0000
> > From: Marc Zyngier <maz@...nel.org>
> 
> Hi Marc, Richard,
> 
> > On Thu, 12 Dec 2024 09:18:56 +0000,
> > richard clark <richard.xnu.clark@...il.com> wrote:
> > > 
> > > Hi M,
> > 
> > Hi r,
> > 
> > > 
> > > On Fri, Dec 6, 2024 at 5:37 PM Marc Zyngier <maz@...nel.org> wrote:
> > > >
> > > > On Fri, 06 Dec 2024 08:33:11 +0000,
> > > > richard clark <richard.xnu.clark@...il.com> wrote:
> > > > >
> > > > > Hi,
> > > > > Currently seems the GICv3/4 irqchip configures all the interrupts as
> > > > > the same priority, I am thinking about to minimize the latency of the
> > > > > interrupt for a particular device, e.g, the arm arch_timer in the RTL
> > > > > system. The question is,
> > > > > 1. Why don't we provide a /proc or /sys interface for the enduser to
> > > > > set the priority of a specific interrupt(SPI/PPI)?
> > > >
> > > > I'm afraid this really has nothing to do with any particular interrupt
> > > > architecture.
> > > >
> > > > Before thinking of exposing the interrupt priority to userspace, you
> > > > should look at what this translates into for the kernel once you allow
> > > > interrupts to be preempted by another one with a higher priority.
> > > >
> > > Interrupt priority doesn't necessarily mean the preemption, seems
> > > you're talking about the interrupt preemption harm according to your
> > > statement followed.Frankly I am just thinking higher priority will win
> > > the lower ones in case massive external interrupts received in the GIC
> > > level (you see I am still talking about GIC, not kernel)
> > 
> > As I stated at the end of my email, the GIC only gives guarantee that
> > you will ack the highest priority interrupt in finite time. Not right
> > when it is made pending. Yes, it has the concept of HPPI. But that
> > from the PoV of the CPU interface, not that of the distributor. Factor
> > in the Stream interface, and you realise that expecting to always ack
> > the highest priority pending interrupt is akin to expecting no
> > reordering of packets in a network.
> > 
> > > >
> > > > This means that at every point where you would normally see a
> > > > local_irq_save(), spinlock_irqsave() or equivalent, you would need to
> > > > explicitly specify the priority that you allow for preemption. You
> > > > should then make sure that any code that can be run during an
> > > > interrupt is reentrant. You need to define which data structures can
> > > > be manipulated at which priority level... The list goes on.
> > > >
> > > irqsave just masks the interrupt from the point of cpu, I don't think
> > > it will mess things up if preemption really happens (no? then what the
> > > side-effect is for the nested interrupt handling in the softirq part.
> > > damage the semantic of the lock primitives?)
> > > >
> > > > If you want a small taste of the complexity, just look at what
> > > > handling NMIs (or even pseudo-NMIs in the case of GICv3) means, and
> > > > generalise it to not just two, but an arbitrary range of priorities.
> > > >
> > > > If you want the full blown experience, look at the BSDs and their use
> > > > of spl*(). I don't think anyone has any plan to get there, and the RT
> > > > patches have shown that there is little need for it.
> > > >
> > > As supplement，the fiq is suggested to be used as an alternative to the
> > > higher priority in the RT area...
> > 
> > <PulpFiction>
> > FIQ's dead, baby. FIQ's dead.
> > </PulpFiction>
> 
> Hah, tell that to Apple! ;).
>
Suppose you're kiding, seems neither Apple employee nor working on its HW :)
> 
> > > > > 2. Is there any way to verify the higher priority interrupt will have
> > > > > more dominant to be selected to the CPU (IOW, the priority is really
> > > > > working) in case of multiple different interrupts asserted to the GIC
> > > > > at the same time(some debug registers of GIC like GICD_REEMPT_CNT :-)
> > > > > to record higher priority wins)?
> > > >
> > > > The GIC architecture makes no promise that the interrupt you
> > > > acknowledge is the highest priority pending interrupt, because this is
> > > > by definition a very racy process.
> > > >
> > > > Also, even on busy systems, you will very rarely see two interrupts
> > > > targeting the same CPU being made pending at the same time, so that
> > > > the interrupt delivery system would have to arbitrate between the two.
> > > > That's because interrupts are vanishingly rare in the grand scheme of
> > > > things.
> > > >
> > > 1. I am trying to stress the external interrupts to the core#0 via the
> > > stress-ng tool with one of interrupt being configured as higher
> > > priority to see the benchmark data, it's time consuming as you can
> > > image, still is in progress(BTW, I can't see any lockup similar hang
> > > in the system with a higher priority configured)
> > 
> > If you don't have preemption, I don't think anything wrong will
> > happen. But I don't expect any benefit either.
> 
> Based on my experience with OpenBSD, I'm not even sure there is much
> benefit even if you have preemtion.
>
is OpenBSD has this priority feature supported? or do you have some related perf
data on BSP...
> 
> And regarding anything wrong happening: there is an interesting bug in
> the RK3399 GIC integration where it gets the priorities wrong:
> 
> https://github.com/openbsd/src/blob/feb3ea439d8f49b3c0e33f54c34631a611b98e21/sys/arch/arm64/dev/agintc.c#L395
> 
> (that comment is my interpretation of what's happening; I might be
> misinterpreting what's really going on)
> 
> As far as I can tell the Linux code doesn't handle that quirk.
> Probably it doesn't matter because Linux only uses the priority
> mechanisms to implement pseudo-NMI functionality and/or doesn't do
> preemption of interrupts.
>
seems the BSP has the priority support but encounter the bug/quirk, correct me if I am wrong. Frankly I have no
time to read the code of your link

	r.

> 
> > > 2. This raises a very interesting question and I am also very curious
> > > about is, what is the purpose for the GIC to introduce the interrupt
> > > priority features, a placeholder feature reserved for the future? Ah,
> > > hardware prefer to provide more functionalities than its being
> > > actually used by software, any other justification except that?
> > 
> > You realise that the HW is not exclusively designed for Linux, right?
> > 
> > 	M.
> > 
> > -- 
> > Without deviation from the norm, progress is not possible.
> > 
> > 

Powered by blists - more mailing lists