[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <87o7gu7mo9.fsf@all.your.base.are.belong.to.us>
Date: Thu, 19 Oct 2023 15:43:18 +0200
From: Björn Töpel <bjorn@...nel.org>
To: Anup Patel <apatel@...tanamicro.com>,
Palmer Dabbelt <palmer@...belt.com>,
Paul Walmsley <paul.walmsley@...ive.com>,
Thomas Gleixner <tglx@...utronix.de>,
Marc Zyngier <maz@...nel.org>,
Rob Herring <robh+dt@...nel.org>,
Krzysztof Kozlowski <krzysztof.kozlowski+dt@...aro.org>,
Frank Rowand <frowand.list@...il.com>,
Conor Dooley <conor+dt@...nel.org>
Cc: Atish Patra <atishp@...shpatra.org>,
Andrew Jones <ajones@...tanamicro.com>,
Sunil V L <sunilvl@...tanamicro.com>,
Saravana Kannan <saravanak@...gle.com>,
Anup Patel <anup@...infault.org>,
linux-riscv@...ts.infradead.org, linux-kernel@...r.kernel.org,
devicetree@...r.kernel.org, Anup Patel <apatel@...tanamicro.com>
Subject: Re: [PATCH v10 00/15] Linux RISC-V AIA Support
Hi Anup,
Anup Patel <apatel@...tanamicro.com> writes:
> The RISC-V AIA specification is ratified as-per the RISC-V international
> process. The latest ratified AIA specifcation can be found at:
> https://github.com/riscv/riscv-aia/releases/download/1.0/riscv-interrupts-1.0.pdf
>
> At a high-level, the AIA specification adds three things:
> 1) AIA CSRs
> - Improved local interrupt support
> 2) Incoming Message Signaled Interrupt Controller (IMSIC)
> - Per-HART MSI controller
> - Support MSI virtualization
> - Support IPI along with virtualization
> 3) Advanced Platform-Level Interrupt Controller (APLIC)
> - Wired interrupt controller
> - In MSI-mode, converts wired interrupt into MSIs (i.e. MSI generator)
> - In Direct-mode, injects external interrupts directly into HARTs
Thanks for working on the AIA support! I had a look at the series, and
have some concerns about interrupt ID abstraction.
A bit of background, for readers not familiar with the AIA details.
IMSIC allows for 2047 unique MSI ("msi-irq") sources per hart, and
each MSI is dedicated to a certain hart. The series takes the approach
to say that there are, e.g., 2047 interrupts ("lnx-irq") globally.
Each lnx-irq consists of #harts * msi-irq -- a slice -- and in the
slice only *one* msi-irq is acutally used.
This scheme makes affinity changes more robust, because the interrupt
sources on "other" harts are pre-allocated. On the other hand it
requires to propagate irq masking to other harts via IPIs (this is
mostly done up setup/tear down). It's also wasteful, because msi-irqs
are hogged, and cannot be used.
Contemporary storage/networking drivers usually uses queues per core
(or a sub-set of cores). The current scheme wastes a lot of msi-irqs.
If we instead used a scheme where "msi-irq == lnx-irq", instead of
"lnq-irq = {hart 0;msi-irq x , ... hart N;msi-irq x}", there would be
a lot MSIs for other users. 1-1 vs 1-N. E.g., if a storage device
would like to use 5 queues (5 cores) on a 128 core system, the current
scheme would consume 5 * 128 MSIs, instead of just 5.
On the plus side:
* Changing interrupts affinity will never fail, because the interrupts
on each hart is pre-allocated.
On the negative side:
* Wasteful interrupt usage, and a system can potientially "run out" of
interrupts. Especially for many core systems.
* Interrupt masking need to proagate to harts via IPIs (there's no
broadcast csr in IMSIC), and a more complex locking scheme IMSIC
Summary:
The current series caps the number of global interrupts to maximum
2047 MSIs for all cores (whole system). A better scheme, IMO, would be
to expose 2047 * #harts unique MSIs.
I think this could simplify/remove(?) the locking as well.
I realize that the series in v10, and coming with a change like this
now might be a bit of a pain...
Finally, another question related to APLIC/IMSIC. AFAIU the memory map
of the IMSIC regions are constrained by the APLIC, which requires a
certain layout for MSI forwarding (group/hart/guest bits). Say that a
system doesn't have an APLIC, couldn't the layout requirement be
simplified?
Again, thanks for the hard work!
Björn
Powered by blists - more mailing lists