Message-ID: <1a0c7282-63e0-4add-8e38-3abe3e0a8e2f@linaro.org>
Date: Thu, 29 Aug 2024 15:42:14 +0200
From: Neil Armstrong <neil.armstrong@...aro.org>
To: Baruch Siach <baruch@...s.co.il>, Christoph Hellwig <hch@....de>,
 Marek Szyprowski <m.szyprowski@...sung.com>,
 Catalin Marinas <catalin.marinas@....com>, Will Deacon <will@...nel.org>
Cc: Robin Murphy <robin.murphy@....com>, iommu@...ts.linux.dev,
 linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
 linuxppc-dev@...ts.ozlabs.org, linux-s390@...r.kernel.org,
 Petr Tesařík <petr@...arici.cz>,
 Ramon Fried <ramon@...reality.ai>, Elad Nachman <enachman@...vell.com>,
 linux-arm-msm <linux-arm-msm@...r.kernel.org>
Subject: Re: [PATCH v6 RESED 1/2] dma: replace zone_dma_bits by zone_dma_limit

Hi,

On 11/08/2024 09:09, Baruch Siach wrote:
> From: Catalin Marinas <catalin.marinas@....com>
> 
> Hardware DMA limit might not be power of 2. When RAM range starts above
> 0, say 4GB, DMA limit of 30 bits should end at 5GB. A single high bit
> can not encode this limit.
> 
> Use plain address for DMA zone limit.
> 
> Since DMA zone can now potentially span beyond 4GB physical limit of
> DMA32, make sure to use DMA zone for GFP_DMA32 allocations in that case.
> 
> Signed-off-by: Catalin Marinas <catalin.marinas@....com>
> Co-developed-by: Baruch Siach <baruch@...s.co.il>
> Signed-off-by: Baruch Siach <baruch@...s.co.il>
> ---
>   arch/arm64/mm/init.c       | 30 +++++++++++++++---------------
>   arch/powerpc/mm/mem.c      |  5 ++++-
>   arch/s390/mm/init.c        |  2 +-
>   include/linux/dma-direct.h |  2 +-
>   kernel/dma/direct.c        |  6 +++---
>   kernel/dma/pool.c          |  4 ++--
>   kernel/dma/swiotlb.c       |  6 +++---
>   7 files changed, 29 insertions(+), 26 deletions(-)
> 

<snip>

This change breaks the Qualcomm SM8550-HDK boot since next-20240826.
It doesn't affect SM8550-QRD or other similar SoCs like SM8650 or SM8450.
The last CI run on next-20240828 can be found at:
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/pipelines/100936

SM8550-HDK boot log:
https://git.codelinaro.org/linaro/qcomlt/ci/staging/cdba-tester/-/jobs/165617

bisect log:
# bad: [b18bbfc14a38b5234e09c2adcf713e38063a7e6e] Add linux-next specific files for 20240829
# good: [5be63fc19fcaa4c236b307420483578a56986a37] Linux 6.11-rc5
git bisect start 'FETCH_HEAD' 'v6.11-rc5'
# bad: [dc09f0263a0accf41821d260f4bf7ad9a4f7b7d8] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/bpf/bpf-next.git
git bisect bad dc09f0263a0accf41821d260f4bf7ad9a4f7b7d8
# bad: [97c7e618752776e03f50311400bb73c01489fb17] Merge branch 'for-next' of git://github.com/Xilinx/linux-xlnx.git
git bisect bad 97c7e618752776e03f50311400bb73c01489fb17
# good: [27ad8eb339a5e3f96aed5f3a3b5901994ce7856d] Merge branch 'clang-format' of https://github.com/ojeda/linux.git
git bisect good 27ad8eb339a5e3f96aed5f3a3b5901994ce7856d
# bad: [fd34d49f6d8dbd157b17b675dc51a145cdad580c] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/khilman/linux-omap.git
git bisect bad fd34d49f6d8dbd157b17b675dc51a145cdad580c
# bad: [29f35e8cf8bff7c69a740edb1cf3d62d211f5a43] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/amlogic/linux.git
git bisect bad 29f35e8cf8bff7c69a740edb1cf3d62d211f5a43
# good: [41860d49473c0c09dc0a2a4d148047f97aaa2539] perf sched: Use perf_tool__init()
git bisect good 41860d49473c0c09dc0a2a4d148047f97aaa2539
# good: [6236ebe07131a7746d870f1d8eb3637a8df13e70] perf daemon: Fix the build on more 32-bit architectures
git bisect good 6236ebe07131a7746d870f1d8eb3637a8df13e70
# good: [92b0d033c80c882e6be26dfeeb2f24c53bdeeee6] Merge branches 'for-next/acpi', 'for-next/misc', 'for-next/perf', 'for-next/selftests' and 'for-next/timers' into for-next/core
git bisect good 92b0d033c80c882e6be26dfeeb2f24c53bdeeee6
# bad: [5db893307df81f0c1dd6b6f167dc263e88ba855a] Merge branch 'for-next' of git://git.kernel.org/pub/scm/linux/kernel/git/rmk/linux.git
git bisect bad 5db893307df81f0c1dd6b6f167dc263e88ba855a
# bad: [f65eaf209abc5f4761623aaa1e110bdb6de124ed] Merge branch 'for-next' of git://git.infradead.org/users/hch/dma-mapping.git
git bisect bad f65eaf209abc5f4761623aaa1e110bdb6de124ed
# bad: [f69e342eec008e1bab772d3963c3dd9979293e13] dma-mapping: call ->unmap_page and ->unmap_sg unconditionally
git bisect bad f69e342eec008e1bab772d3963c3dd9979293e13
# bad: [ba0fb44aed47693cc2482427f63ba6cd19051327] dma-mapping: replace zone_dma_bits by zone_dma_limit
git bisect bad ba0fb44aed47693cc2482427f63ba6cd19051327
# good: [fa3c109a6d302b56437a6412c5f3044c3e12de03] dma-mapping: use bit masking to check VM_DMA_COHERENT
git bisect good fa3c109a6d302b56437a6412c5f3044c3e12de03
# first bad commit: [ba0fb44aed47693cc2482427f63ba6cd19051327] dma-mapping: replace zone_dma_bits by zone_dma_limit

Boot log with earlycon:
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x411fd461]
[    0.000000] Linux version 6.11.0-rc5-next-20240829-07694-gb36219e63903 (narmstrong@...akeen.starnux.net) (aarch64-none-linux-gnu-gcc (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 13.2.1 20231009, GNU ld (Arm GNU Toolchain 13.2.rel1 (Build arm-13.7)) 2.41.0.20231009) #397 SMP PREEMPT Thu Aug 29 15:25:39 CEST 2024
[    0.000000] KASLR enabled
[    0.000000] random: crng init done
[    0.000000] Machine model: Qualcomm Technologies, Inc. SM8550 HDK
[    0.000000] printk: debug: ignoring loglevel setting.
[    0.000000] efi: UEFI not found.
[    0.000000] [Firmware Bug]: Kernel image misaligned at boot, please fix your bootloader!
[    0.000000] OF: reserved mem: 0x0000000080000000..0x00000000809fffff (10240 KiB) nomap non-reusable hyp-region@...00000
[    0.000000] OF: reserved mem: 0x0000000080a00000..0x0000000080dfffff (4096 KiB) nomap non-reusable cpusys-vm-region@...00000
[    0.000000] OF: reserved mem: 0x0000000080e00000..0x00000000811cffff (3904 KiB) nomap non-reusable hyp-tags-region@...00000
[    0.000000] OF: reserved mem: 0x00000000811d0000..0x00000000811fffff (192 KiB) nomap non-reusable hyp-tags-reserved-region@...d0000
[    0.000000] OF: reserved mem: 0x0000000081a00000..0x0000000081c5ffff (2432 KiB) nomap non-reusable xbl-dt-log-merged-region@...00000
[    0.000000] OF: reserved mem: 0x0000000081c60000..0x0000000081c7ffff (128 KiB) nomap non-reusable aop-cmd-db-region@...60000
[    0.000000] OF: reserved mem: 0x0000000081c80000..0x0000000081cf3fff (464 KiB) nomap non-reusable aop-config-merged-region@...80000
[    0.000000] OF: reserved mem: 0x0000000081d00000..0x0000000081efffff (2048 KiB) nomap non-reusable smem@...00000
[    0.000000] OF: reserved mem: 0x0000000081f00000..0x0000000081f1ffff (128 KiB) nomap non-reusable adsp-mhi-region@...00000
[    0.000000] OF: reserved mem: 0x0000000082600000..0x00000000826fffff (1024 KiB) nomap non-reusable global-sync-region@...00000
[    0.000000] OF: reserved mem: 0x0000000082700000..0x00000000827fffff (1024 KiB) nomap non-reusable tz-stat-region@...00000
[    0.000000] OF: reserved mem: 0x0000000082800000..0x0000000086dfffff (71680 KiB) nomap non-reusable cdsp-secure-heap-region@...00000
[    0.000000] OF: reserved mem: 0x000000008a800000..0x000000009affffff (270336 KiB) nomap non-reusable mpss-region@...00000
[    0.000000] OF: reserved mem: 0x000000009b000000..0x000000009b07ffff (512 KiB) nomap non-reusable q6-mpss-dtb-region@...00000
[    0.000000] OF: reserved mem: 0x000000009b080000..0x000000009b08ffff (64 KiB) nomap non-reusable ipa-fw-region@...80000
[    0.000000] OF: reserved mem: 0x000000009b090000..0x000000009b099fff (40 KiB) nomap non-reusable ipa-gsi-region@...90000
[    0.000000] OF: reserved mem: 0x000000009b09a000..0x000000009b09bfff (8 KiB) nomap non-reusable gpu-micro-code-region@...9a000
[    0.000000] OF: reserved mem: 0x000000009b100000..0x000000009b27ffff (1536 KiB) nomap non-reusable spss-region@...00000
[    0.000000] OF: reserved mem: 0x000000009b280000..0x000000009b2dffff (384 KiB) nomap non-reusable spu-tz-shared-region@...80000
[    0.000000] OF: reserved mem: 0x000000009b2e0000..0x000000009b2fffff (128 KiB) nomap non-reusable spu-modem-shared-region@...e0000
[    0.000000] OF: reserved mem: 0x000000009b300000..0x000000009bafffff (8192 KiB) nomap non-reusable camera-region@...00000
[    0.000000] OF: reserved mem: 0x000000009bb00000..0x000000009c1fffff (7168 KiB) nomap non-reusable video-region@...00000
[    0.000000] OF: reserved mem: 0x000000009c200000..0x000000009c8fffff (7168 KiB) nomap non-reusable cvp-region@...00000
[    0.000000] OF: reserved mem: 0x000000009c900000..0x000000009e8fffff (32768 KiB) nomap non-reusable cdsp-region@...00000
[    0.000000] OF: reserved mem: 0x000000009e900000..0x000000009e97ffff (512 KiB) nomap non-reusable q6-cdsp-dtb-region@...00000
[    0.000000] OF: reserved mem: 0x000000009e980000..0x000000009e9fffff (512 KiB) nomap non-reusable q6-adsp-dtb-region@...80000
[    0.000000] OF: reserved mem: 0x000000009ea00000..0x00000000a2a7ffff (66048 KiB) nomap non-reusable adspslpi-region@...00000
[    0.000000] OF: reserved mem: 0x00000000d4a80000..0x00000000d4cfffff (2560 KiB) nomap non-reusable rmtfs-region@...80000
[    0.000000] OF: reserved mem: 0x00000000d4d00000..0x00000000d7ffffff (52224 KiB) nomap non-reusable mpss-dsm-region@...00000
[    0.000000] OF: reserved mem: 0x00000000d8000000..0x00000000d80fffff (1024 KiB) nomap non-reusable tz-reserved-region@...00000
[    0.000000] OF: reserved mem: 0x00000000d8100000..0x00000000d813ffff (256 KiB) nomap non-reusable xbl-sc-region@...00000
[    0.000000] OF: reserved mem: 0x00000000d8140000..0x00000000d82fffff (1792 KiB) nomap non-reusable cpucp-fw-region@...40000
[    0.000000] OF: reserved mem: 0x00000000d8300000..0x00000000d87fffff (5120 KiB) nomap non-reusable qtee-region@...00000
[    0.000000] OF: reserved mem: 0x00000000d8800000..0x00000000e11fffff (141312 KiB) nomap non-reusable ta-region@...00000
[    0.000000] OF: reserved mem: 0x00000000e1200000..0x00000000e393ffff (40192 KiB) nomap non-reusable tz-tags-region@...00000
[    0.000000] OF: reserved mem: 0x00000000e6440000..0x00000000e66b8fff (2532 KiB) nomap non-reusable hwfence-shbuf-region@...40000
[    0.000000] OF: reserved mem: 0x00000000f3600000..0x00000000f80edfff (76728 KiB) nomap non-reusable trust-ui-vm-region@...00000
[    0.000000] OF: reserved mem: 0x00000000f80ee000..0x00000000f80eefff (4 KiB) nomap non-reusable trust-ui-vm-dump-region@...ee000
[    0.000000] OF: reserved mem: 0x00000000f80ef000..0x00000000f80f7fff (36 KiB) nomap non-reusable trust-ui-vm-qrt-region@...ef000
[    0.000000] OF: reserved mem: 0x00000000f80f8000..0x00000000f80fbfff (16 KiB) nomap non-reusable trust-ui-vm-vblk0-ring-region@...f8000
[    0.000000] OF: reserved mem: 0x00000000f80fc000..0x00000000f80fffff (16 KiB) nomap non-reusable trust-ui-vm-vblk1-ring-region@...fc000
[    0.000000] OF: reserved mem: 0x00000000f8100000..0x00000000f81fffff (1024 KiB) nomap non-reusable trust-ui-vm-swiotlb-region@...00000
[    0.000000] OF: reserved mem: 0x00000000f8400000..0x00000000fcbfffff (73728 KiB) nomap non-reusable oem-vm-region@...00000
[    0.000000] OF: reserved mem: 0x00000000fcc00000..0x00000000fcc03fff (16 KiB) nomap non-reusable oem-vm-vblk0-ring-region@...00000
[    0.000000] OF: reserved mem: 0x00000000fcc04000..0x00000000fcd03fff (1024 KiB) nomap non-reusable oem-vm-swiotlb-region@...04000
[    0.000000] OF: reserved mem: 0x00000000fce00000..0x00000000ff6fffff (41984 KiB) nomap non-reusable hyp-ext-tags-region@...00000
[    0.000000] OF: reserved mem: 0x00000000ff700000..0x00000000ff7fffff (1024 KiB) nomap non-reusable hyp-ext-reserved-region@...00000
[    0.000000] earlycon: qcom_geni0 at MMIO 0x0000000000a9c000 (options '115200n8')
[    0.000000] printk: legacy bootconsole [qcom_geni0] enabled
[    0.000000] NUMA: Faking a node at [mem 0x0000000080000000-0x0000000affffffff]
[    0.000000] NODE_DATA(0) allocated [mem 0xafe933200-0xafe93583f]
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x0000000affffffff]
[    0.000000]   DMA32    empty
[    0.000000]   Normal   empty
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x0000000080000000-0x0000000080dfffff]
[    0.000000]   node   0: [mem 0x00000000811d0000-0x00000000811fffff]
[    0.000000]   node   0: [mem 0x0000000081200000-0x00000000819fffff]
[    0.000000]   node   0: [mem 0x0000000081a00000-0x0000000081cf3fff]
[    0.000000]   node   0: [mem 0x0000000081cf4000-0x0000000081cfffff]
[    0.000000]   node   0: [mem 0x0000000081d00000-0x0000000081f1ffff]
[    0.000000]   node   0: [mem 0x0000000081f20000-0x00000000825fffff]
[    0.000000]   node   0: [mem 0x0000000082600000-0x0000000086dfffff]
[    0.000000]   node   0: [mem 0x0000000086e00000-0x000000008a7fffff]
[    0.000000]   node   0: [mem 0x000000008a800000-0x000000009b09bfff]
[    0.000000]   node   0: [mem 0x000000009b09c000-0x000000009b0fffff]
[    0.000000]   node   0: [mem 0x000000009b100000-0x00000000a2a7ffff]
[    0.000000]   node   0: [mem 0x00000000a2a80000-0x00000000d4a7ffff]
[    0.000000]   node   0: [mem 0x00000000d4a80000-0x00000000d7ffffff]
[    0.000000]   node   0: [mem 0x00000000d8140000-0x00000000d815ffff]
[    0.000000]   node   0: [mem 0x00000000e1bb0000-0x00000000e393ffff]
[    0.000000]   node   0: [mem 0x00000000e3940000-0x00000000e643ffff]
[    0.000000]   node   0: [mem 0x00000000e6440000-0x00000000e66b8fff]
[    0.000000]   node   0: [mem 0x00000000e66b9000-0x00000000f35fffff]
[    0.000000]   node   0: [mem 0x00000000f3600000-0x00000000f81fffff]
[    0.000000]   node   0: [mem 0x00000000f8200000-0x00000000f83fffff]
[    0.000000]   node   0: [mem 0x00000000f8400000-0x00000000fcd03fff]
[    0.000000]   node   0: [mem 0x00000000fcd04000-0x00000000fcdfffff]
[    0.000000]   node   0: [mem 0x00000000fce00000-0x00000000ff7fffff]
[    0.000000]   node   0: [mem 0x00000000ff800000-0x00000000ffffffff]
[    0.000000]   node   0: [mem 0x0000000880000000-0x00000008b99fffff]
[    0.000000]   node   0: [mem 0x00000008c0000000-0x0000000affffffff]
[    0.000000] Initmem setup node 0 [mem 0x0000000080000000-0x0000000affffffff]
[    0.000000] On node 0, zone DMA: 976 pages in unavailable ranges
[    0.000000] On node 0, zone DMA: 320 pages in unavailable ranges
[    0.000000] On node 0, zone DMA: 39504 pages in unavailable ranges
[    0.000000] On node 0, zone DMA: 26112 pages in unavailable ranges
[    0.000000] cma: Reserved 32 MiB at 0x0000000880000000 on node -1
[    0.000000] psci: probing for conduit method from DT.
[    0.000000] psci: PSCIv1.1 detected in firmware.
[    0.000000] psci: Using standard PSCI v0.2 function IDs
[    0.000000] psci: MIGRATE_INFO_TYPE not supported.
[    0.000000] psci: SMC Calling Convention v1.3
[    0.000000] psci: OSI mode supported.
[    0.000000] percpu: Embedded 25 pages/cpu s61656 r8192 d32552 u102400
[    0.000000] pcpu-alloc: s61656 r8192 d32552 u102400 alloc=25*4096
[    0.000000] pcpu-alloc: [0] 0 [0] 1 [0] 2 [0] 3 [0] 4 [0] 5 [0] 6 [0] 7
[    0.000000] Detected PIPT I-cache on CPU0
[    0.000000] CPU features: detected: Address authentication (architected QARMA5 algorithm)
[    0.000000] CPU features: detected: GIC system register CPU interface
[    0.000000] CPU features: detected: Spectre-v4
[    0.000000] CPU features: detected: ARM erratum 2457168
[    0.000000] CPU features: detected: ARM erratum 2658417
[    0.000000] CPU features: detected: ARM errata 2966298, 3117295
[    0.000000] alternatives: applying boot alternatives
[    0.000000] Kernel command line: earlycon root= allow_mismatched_32bit_el0 clk_ignore_unused pd_ignore_unused systemd.mask=rmtfs.service ignore_loglevel -- androidboot.verifiedbootstate=orange androidboot.keymaster=1  androidboot.bootdevice=1d84000.ufshc androidboot.fstab_suffix=default androidboot.boot_devices=soc/1d84000.ufshc androidboot.serialno=2a7a254f androidboot.baseband=apq androidboot.force_normal_boot=1
[    0.000000] Dentry cache hash table entries: 2097152 (order: 12, 16777216 bytes, linear)
[    0.000000] Inode-cache hash table entries: 1048576 (order: 11, 8388608 bytes, linear)
[    0.000000] Fallback order for Node 0: 0
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 3078816
[    0.000000] Policy zone: DMA
[    0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off
[    0.000000] software IO TLB: SWIOTLB bounce buffer size adjusted to 11MB
[    0.000000] software IO TLB: area num 8.
[    0.000000] software IO TLB: SWIOTLB bounce buffer size roundup to 16MB
[    0.000000] software IO TLB: mapped [mem 0x0000000aeff38000-0x0000000af0f38000] (16MB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: 	RCU event tracing is enabled.
[    0.000000] rcu: 	RCU restricting CPUs from NR_CPUS=512 to nr_cpu_ids=8.
[    0.000000] 	Trampoline variant of Tasks RCU enabled.
[    0.000000] 	Tracing variant of Tasks RCU enabled.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies.
[    0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=8
[    0.000000] RCU Tasks: Setting shift to 3 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=8.
[    0.000000] RCU Tasks Trace: Setting shift to 3 and lim to 1 rcu_task_cb_adjust=1 rcu_task_cpu_ids=8.
[    0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0
[    0.000000] GICv3: 988 SPIs implemented
[    0.000000] GICv3: 0 Extended SPIs implemented
[    0.000000] Root IRQ handler: gic_handle_irq
[    0.000000] GICv3: GICv3 features: 16 PPIs, DirectLPI
[    0.000000] GICv3: GICD_CTRL.DS=1, SCR_EL3.FIQ=0
[    0.000000] GICv3: Enabling SGIs without active state
[    0.000000] GICv3: CPU0: found redistributor 0 region 0:0x0000000017180000
[    0.000000] ITS [mem 0x17140000-0x1715ffff]
[    0.000000] ITS@...000000017140000: Devices Table too large, reduce ids 32->19
[    0.000000] ITS@...000000017140000: Devices too large, reduce ITS pages 1024->256
[    0.000000] ITS@...000000017140000: allocated 131072 Devices @81500000 (indirect, esz 8, psz 4K, shr 1)
[    0.000000] ITS@...000000017140000: allocated 4096 Interrupt Collections @8145b000 (flat, esz 1, psz 4K, shr 1)
[    0.000000] GICv3: using LPI property table @0x0000000081470000
[    0.000000] GICv3: CPU0: using allocated LPI pending table @0x0000000081480000
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] arch_timer: cp15 and mmio timer(s) running at 19.20MHz (virt/virt).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[    0.000000] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[    0.008355] arm-pv: using stolen time PV
[    0.012595] Console: colour dummy device 80x25
[    0.017182] printk: legacy console [tty0] enabled
[    0.022031] printk: legacy bootconsole [qcom_geni0] disabled
<----------------->

System handler takes over, I'll analyze the system firmware logs to understand which access made the crash.

I tested the "arm64: mm: fix DMA zone when dma-ranges is missing" patch at [1], but it doesn't help.

[1] https://lore.kernel.org/all/d8e92b14181fc815773ac4f4bba70b5d48bc390e.1724838684.git.baruch@tkos.co.il/

Neil

#regzbot introduced: ba0fb44aed47693cc2482427f63ba6cd19051327

Powered by blists - more mailing lists