Message-ID: <aSnH3C8s5xVSk_ti@linaro.org>
Date: Fri, 28 Nov 2025 17:03:40 +0100
From: Stephan Gerhold <stephan.gerhold@...aro.org>
To: Jonathan Marek <jonathan@...ek.ca>
Cc: linux-arm-msm@...r.kernel.org, Bjorn Andersson <andersson@...nel.org>,
	Konrad Dybcio <konradybcio@...nel.org>,
	Rob Herring <robh@...nel.org>,
	Krzysztof Kozlowski <krzk+dt@...nel.org>,
	Conor Dooley <conor+dt@...nel.org>,
	Sibi Sankar <sibi.sankar@....qualcomm.com>,
	Abel Vesa <abel.vesa@...aro.org>,
	Rajendra Nayak <quic_rjendra@...cinc.com>,
	"open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS" <devicetree@...r.kernel.org>,
	open list <linux-kernel@...r.kernel.org>
Subject: Re: [PATCH] arm64: dts: qcom: x1e: bus is 40-bits (fix 64GB models)

On Fri, Nov 28, 2025 at 09:39:52AM -0500, Jonathan Marek wrote:
> On 11/28/25 5:26 AM, Stephan Gerhold wrote:
> > On Thu, Nov 27, 2025 at 04:29:42PM -0500, Jonathan Marek wrote:
> > > Unlike the phone SoCs this was copied from, x1e has a 40-bit physical bus.
> > > The upper address space is used to support more than 32GB of memory.
> > > 
> > > This fixes issues when DMA buffers are allocated outside the 36-bit range.
> > > 
> > > Fixes: af16b00578a7 ("arm64: dts: qcom: Add base X1E80100 dtsi and the QCP dts")
> > > Signed-off-by: Jonathan Marek <jonathan@...ek.ca>
> > > ---
> > >   arch/arm64/boot/dts/qcom/x1e80100.dtsi | 4 ++--
> > >   1 file changed, 2 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/arch/arm64/boot/dts/qcom/x1e80100.dtsi b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > > index cff34d1c74b60..cd34ce5dfd63a 100644
> > > --- a/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > > +++ b/arch/arm64/boot/dts/qcom/x1e80100.dtsi
> > > @@ -792,8 +792,8 @@ soc: soc@0 {
> > >   		#address-cells = <2>;
> > >   		#size-cells = <2>;
> > > -		dma-ranges = <0 0 0 0 0x10 0>;
> > > -		ranges = <0 0 0 0 0x10 0>;
> > > +		dma-ranges = <0 0 0 0 0x100 0>;
> > > +		ranges = <0 0 0 0 0x100 0>;
> > 
> > Could you clarify which "issues" (crashes?) you are referring to?
> > 
> > We need to distinguish two distinct use cases here, which are both
> > (somewhat) supported upstream: Running in EL1 with the Gunyah hypervisor
> > with the regular DTB and in EL2 with the x1-el2.dtbo applied.
> > 
> > # EL2 with x1-el2.dtbo
> > 
> > For EL2, I think the 40-bit dma-ranges should indeed work correctly, so
> > we could add your proposed change inside x1-el2.dtso. I'm not sure which
> > issues we are fixing with that though (besides correctness of the
> > hardware description). In EL2, all DMA devices should be behind an
> > IOMMU. In this case, the dma-ranges limit the size of the I/O virtual
> > addresses (DMA addresses) that are given to the devices. The IOMMU maps
> > the DMA buffers to arbitrary physical memory addresses (including
> > outside of the 36-bit range, dma-ranges limits only the DMA address).
> > 
> > I would expect that applying your change effectively just enlarges the
> > I/O virtual address space, which will then be 40-bit instead of just
> > 36-bit. For most devices, even 32-bit of virtual address space should be
> > enough. A larger address space will only be applied for drivers that
> > explicitly request a larger DMA mask (e.g. the nvme driver).
> > 
> > We can make this change for correctness, but given that it is only about
> > the IOVA space, there shouldn't be much functional difference.
> > 
> > # EL1 with Gunyah hypervisor
> > 
> > For EL1, the hypervisor firmware used on most retail laptops limits the
> > usable DMA memory in the SMMUs to the physical 36-bit range. You are
> > right that laptops with 64 GiB memory are essentially unusable in EL1
> > without disabling the physical memory outside the 36-bit range, but
> > applying this patch would make it even worse.
> > 
> > There are two separate cases:
> > 
> >   - For devices behind the SMMUv2, the situation should be the same as
> >     above. Increased IOVA space, but no effect on physical address range.
> >     This is what is currently causing crashes with 64 GiB RAM in EL1.
> > 
> >   - Devices behind the SMMUv3 (PCIe) do not have an IOMMU assigned when
> >     running in EL1. In this case, the 36-bit dma-ranges prevents PCIe
> >     devices from using memory outside the 36-bit range. They will fall
> >     back to bounce buffers in that case. Applying your patch will disable
> >     that, making it even more likely to crash than before.
> > 
> > Given that x1e80100.dtsi / hamoa.dtsi primarily models the EL1 setup
> > with Gunyah hypervisor, I don't think it makes sense to apply this patch
> > as-is. It will just make it even more likely to crash than before.
> > I suggest adding these overrides in x1-el2.dtso, with the expected
> > limited effect I described above.
> > 
> > Thanks,
> > Stephan
> > 
> 
> I am using EL2.
> 
> Without this patch, DMA buffers allocated in the upper 36-bit physical range
> will try to use bounce buffers. The dma range from the dts is compared
> against the physical address, not the virtual address.

I don't think this is the case for the dma-iommu layer. I debugged a
crash caused by USB in EL1 on a 64 GiB device earlier this year and it
was happily using buffers above the 36-bit physical range without using
bounce buffers. There is some code inside dma-iommu for using swiotlb,
but it's used only for "untrusted" PCI devices and some edge cases with
unaligned/small buffers.

> 
> The crash I see is display driver crashes/freezes once a buffer is allocated
> in the upper 36-bit range and it tries to use bounce buffers. This can
> happens very quickly under load.
> 

You could be right about the MSM display driver though, since that
bypasses dma-iommu and manages the IOMMU itself. I stared at the code a
bit and I'm not immediately seeing where it would end up calling into
swiotlb, but it might be hidden somewhere in the endless nesting.

> The same crash would happen for EL1 as well. I wasn't aware of the EL1
> broken firmware when I sent this patch, but instead of display freezing I
> guess the behavior would a hard reset now, which is a bit worse but still
> unusable unles display/gpu driver is disabled.
> 
> This patch is correct and should be applied regardless of broken-firmware
> EL1 cases (where 64GB isn't usable anyway), but I guess the Fixes tag
> can/should be dropped.
> 

Please clarify the commit message a bit and mention the two separate use
cases (EL1 and EL2). I'll leave it up to Bjorn/Konrad to decide whether
to merge it. At the end you are right and using 64 GiB RAM in EL1 is
kind of a lost cause anyway.

Thanks,
Stephan

Powered by blists - more mailing lists