| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <Y3V24PrgL0aN9P08@iweiny-desk3>
Date: Wed, 16 Nov 2022 15:48:48 -0800
From: Ira Weiny <ira.weiny@...el.com>
To: Jonathan Cameron <Jonathan.Cameron@...wei.com>
CC: Dan Williams <dan.j.williams@...el.com>,
Davidlohr Bueso <dave@...olabs.net>,
Bjorn Helgaas <helgaas@...nel.org>,
Alison Schofield <alison.schofield@...el.com>,
Vishal Verma <vishal.l.verma@...el.com>,
"Ben Widawsky" <bwidawsk@...nel.org>,
Steven Rostedt <rostedt@...dmis.org>,
<linux-kernel@...r.kernel.org>, <linux-cxl@...r.kernel.org>
Subject: Re: [PATCH 01/11] cxl/pci: Add generic MSI-X/MSI irq support
On Wed, Nov 16, 2022 at 02:53:41PM +0000, Jonathan Cameron wrote:
> On Thu, 10 Nov 2022 10:57:48 -0800
> ira.weiny@...el.com wrote:
>
> > From: Davidlohr Bueso <dave@...olabs.net>
> >
> > Currently the only CXL features targeted for irq support require their
> > message numbers to be within the first 16 entries. The device may
> > however support less than 16 entries depending on the support it
> > provides.
> >
> > Attempt to allocate these 16 irq vectors. If the device supports less
> > then the PCI infrastructure will allocate that number. Store the number
> > of vectors actually allocated in the device state for later use
> > by individual functions.
> See later patch review, but I don't think we need to store the number
> allocated because any vector is guaranteed to be below that point
Only as long as we stick to those functions which are guaranteed to be under
16. If a device supports more than 16 and code is added to try and enable that
irq this base support will not cover that.
> (QEMU code is wrong on this at the momemt, but there are very few vectors
> so it hasn't mattered yet).
How so? Does the spec state that a device must report at least 16 vectors?
>
> Otherwise, pcim fun deals with some of the cleanup you are doing again
> here for us so can simplify this somewhat. See inline.
Yea it is broken.
>
> Jonathan
>
>
>
> >
> > Upon successful allocation, users can plug in their respective isr at
> > any point thereafter, for example, if the irq setup is not done in the
> > PCI driver, such as the case of the CXL-PMU.
> >
> > Cc: Bjorn Helgaas <helgaas@...nel.org>
> > Cc: Jonathan Cameron <Jonathan.Cameron@...wei.com>
> > Co-developed-by: Ira Weiny <ira.weiny@...el.com>
> > Signed-off-by: Ira Weiny <ira.weiny@...el.com>
> > Signed-off-by: Davidlohr Bueso <dave@...olabs.net>
> >
> > ---
> > Changes from Ira
> > Remove reviews
> > Allocate up to a static 16 vectors.
> > Change cover letter
> > ---
> > drivers/cxl/cxlmem.h | 3 +++
> > drivers/cxl/cxlpci.h | 6 ++++++
> > drivers/cxl/pci.c | 32 ++++++++++++++++++++++++++++++++
> > 3 files changed, 41 insertions(+)
> >
> > diff --git a/drivers/cxl/cxlmem.h b/drivers/cxl/cxlmem.h
> > index 88e3a8e54b6a..b7b955ded3ac 100644
> > --- a/drivers/cxl/cxlmem.h
> > +++ b/drivers/cxl/cxlmem.h
> > @@ -211,6 +211,7 @@ struct cxl_endpoint_dvsec_info {
> > * @info: Cached DVSEC information about the device.
> > * @serial: PCIe Device Serial Number
> > * @doe_mbs: PCI DOE mailbox array
> > + * @nr_irq_vecs: Number of MSI-X/MSI vectors available
> > * @mbox_send: @dev specific transport for transmitting mailbox commands
> > *
> > * See section 8.2.9.5.2 Capacity Configuration and Label Storage for
> > @@ -247,6 +248,8 @@ struct cxl_dev_state {
> >
> > struct xarray doe_mbs;
> >
> > + int nr_irq_vecs;
> > +
> > int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd);
> > };
> >
> > diff --git a/drivers/cxl/cxlpci.h b/drivers/cxl/cxlpci.h
> > index eec597dbe763..b7f4e2f417d3 100644
> > --- a/drivers/cxl/cxlpci.h
> > +++ b/drivers/cxl/cxlpci.h
> > @@ -53,6 +53,12 @@
> > #define CXL_DVSEC_REG_LOCATOR_BLOCK_ID_MASK GENMASK(15, 8)
> > #define CXL_DVSEC_REG_LOCATOR_BLOCK_OFF_LOW_MASK GENMASK(31, 16)
> >
> > +/*
> > + * NOTE: Currently all the functions which are enabled for CXL require their
> > + * vectors to be in the first 16. Use this as the max.
> > + */
> > +#define CXL_PCI_REQUIRED_VECTORS 16
> > +
> > /* Register Block Identifier (RBI) */
> > enum cxl_regloc_type {
> > CXL_REGLOC_RBI_EMPTY = 0,
> > diff --git a/drivers/cxl/pci.c b/drivers/cxl/pci.c
> > index faeb5d9d7a7a..62e560063e50 100644
> > --- a/drivers/cxl/pci.c
> > +++ b/drivers/cxl/pci.c
> > @@ -428,6 +428,36 @@ static void devm_cxl_pci_create_doe(struct cxl_dev_state *cxlds)
> > }
> > }
> >
> > +static void cxl_pci_free_irq_vectors(void *data)
> > +{
> > + pci_free_irq_vectors(data);
> > +}
> > +
> > +static void cxl_pci_alloc_irq_vectors(struct cxl_dev_state *cxlds)
> > +{
> > + struct device *dev = cxlds->dev;
> > + struct pci_dev *pdev = to_pci_dev(dev);
> > + int nvecs;
> > + int rc;
> > +
> > + nvecs = pci_alloc_irq_vectors(pdev, 1, CXL_PCI_REQUIRED_VECTORS,
> > + PCI_IRQ_MSIX | PCI_IRQ_MSI);
> > + if (nvecs < 0) {
> > + dev_dbg(dev, "Not enough interrupts; use polling instead.\n");
> > + return;
> > + }
> > +
> > + rc = devm_add_action_or_reset(dev, cxl_pci_free_irq_vectors, pdev);
> The pci managed code always gives me a headache because there is a lot of magic
> under the hood if you ever called pcim_enable_device() which we did.
>
> Chasing through
>
> pci_alloc_irq_vectors_affinity()->
> either
> __pci_enable_msix_range()
> or
> __pci_enable_msi_range()
>
> they are similar
> pci_setup_msi_context()
> pci_setup_msi_release()
> adds pcmi_msi_release devm action.
> and that frees the vectors for us.
> So we don't need to do it here.
:-/
So what is the point of pci_free_irq_vectors()? This is very confusing to have
a function not called pcim_* [pci_alloc_irq_vectors()] do 'pcim stuff'.
Ok I'll drop this extra because I see it now.
>
>
> > + if (rc) {
> > + dev_dbg(dev, "Device managed call failed; interrupts disabled.\n");
> > + /* some got allocated, clean them up */
> > + cxl_pci_free_irq_vectors(pdev);
> We could just leave them lying around for devm cleanup to sweep up eventually
> or free them as you have done here.
And besides this extra call is flat out broken. cxl_pci_free_irq_vectors() is
already called at this point if devm_add_action_or_reset() failed... But I see
this is not required.
I do plan to add a big ol' comment as to why we don't need to mirror the call
with the corresponding 'free'.
I'll respin,
Ira
>
> > + return;
> > + }
> > +
> > + cxlds->nr_irq_vecs = nvecs;
> > +}
> > +
> > static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> > {
> > struct cxl_register_map map;
> > @@ -494,6 +524,8 @@ static int cxl_pci_probe(struct pci_dev *pdev, const struct pci_device_id *id)
> > if (rc)
> > return rc;
> >
> > + cxl_pci_alloc_irq_vectors(cxlds);
> > +
> > cxlmd = devm_cxl_add_memdev(cxlds);
> > if (IS_ERR(cxlmd))
> > return PTR_ERR(cxlmd);
>
Powered by blists - more mailing lists