| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <2g2lp3bkadc4wpeslmdoexpidoiqzt7vejar5xhjx5ayt3uox3@dqdyfzn6khn6>
Date: Wed, 4 Dec 2024 16:58:34 +0100
From: Thierry Reding <thierry.reding@...il.com>
To: Robin Murphy <robin.murphy@....com>
Cc: Furong Xu <0x1207@...il.com>, Jakub Kicinski <kuba@...nel.org>,
Jon Hunter <jonathanh@...dia.com>, netdev@...r.kernel.org, linux-stm32@...md-mailman.stormreply.com,
linux-arm-kernel@...ts.infradead.org, linux-kernel@...r.kernel.org,
Alexandre Torgue <alexandre.torgue@...s.st.com>, Jose Abreu <joabreu@...opsys.com>,
"David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
Paolo Abeni <pabeni@...hat.com>, Maxime Coquelin <mcoquelin.stm32@...il.com>, xfr@...look.com,
Suraj Jaiswal <quic_jsuraj@...cinc.com>, Thierry Reding <treding@...dia.com>,
"linux-tegra@...r.kernel.org" <linux-tegra@...r.kernel.org>, Will Deacon <will@...nel.org>
Subject: Re: [PATCH net v1] net: stmmac: TSO: Fix unbalanced DMA map/unmap
for non-paged SKB data
On Wed, Dec 04, 2024 at 02:06:00PM +0000, Robin Murphy wrote:
> On 2024-12-04 1:57 pm, Thierry Reding wrote:
> > On Tue, Dec 03, 2024 at 11:16:37AM +0800, Furong Xu wrote:
> > > On Mon, 2 Dec 2024 18:34:25 -0800, Jakub Kicinski <kuba@...nel.org> wrote:
> > >
> > > > On Tue, 3 Dec 2024 10:03:31 +0800 Furong Xu wrote:
> > > > > I requested Jon to provide more info about "Tx DMA map failed" in previous
> > > > > reply, and he does not respond yet.
> > > >
> > > > What does it mean to provide "more info" about a print statement from
> > > > the driver? Is there a Kconfig which he needs to set to get more info?
> > > > Perhaps you should provide a debug patch he can apply on his tree, that
> > > > will print info about (1) which buffer mapping failed (head or frags);
> > > > (2) what the physical address was of the buffer that couldn't be mapped.
> > >
> > > A debug patch to print info about buffer makes no sense here.
> > > Both Tegra186 Jetson TX2(tegra186-p2771-0000) and Tegra194 Jetson AGX Xavier
> > > (tegra194-p2972-0000) enable IOMMU/SMMU for stmmac in its device-tree node,
> > > buffer info should be investigated with detailed IOMMU/SMMU debug info from
> > > drivers/iommu/arm/arm-smmu/arm-smmu-nvidia.c together.
> > >
> > > I am not an expert in IOMMU, so I cannot help more.
> > >
> > > Without the help from Jon, our only choice is revert as you said.
> >
> > I was able to reproduce this locally and I get this splat:
> >
> > --- >8 ---
> > [ 228.179234] WARNING: CPU: 0 PID: 0 at drivers/iommu/io-pgtable-arm.c:346 __arm_lpae_map+0x388/0x4e4
> > [ 228.188300] Modules linked in: snd_soc_tegra210_mixer snd_soc_tegra210_admaif snd_soc_tegra_pcm snd_soc_tegra186_asrc snd_soc_tegra210_ope snd_soc_tegra210_adx snd_soc_tegra210_mvc snd_soc_tegra210_dmic snd_soc_tegra186_dspk snd_soc_tegra210_sfc snd_soc_tegra210_amx snd_soc_tegra210_i2s tegra_drm drm_dp_aux_bus cec drm_display_helper drm_client_lib tegra210_adma snd_soc_tegra210_ahub drm_kms_helper snd_hda_codec_hdmi snd_hda_tegra snd_soc_tegra_audio_graph_card at24 snd_hda_codec ina3221 snd_soc_audio_graph_card snd_soc_simple_card_utils tegra_bpmp_thermal tegra_xudc snd_hda_core tegra_aconnect host1x fuse drm backlight ipv6
> > [ 228.243750] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G S 6.13.0-rc1-next-20241203 #30
> > [ 228.253412] Tainted: [S]=CPU_OUT_OF_SPEC
> > [ 228.257336] Hardware name: nvidia NVIDIA P2771-0000-500/NVIDIA P2771-0000-500, BIOS 2025.01-rc3-00040-g36352ae2e68e-dirty 01/01/2025
> > [ 228.269239] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
> > [ 228.276205] pc : __arm_lpae_map+0x388/0x4e4
> > [ 228.280398] lr : __arm_lpae_map+0x120/0x4e4
> > [ 228.284587] sp : ffff8000800037f0
> > [ 228.287901] x29: ffff800080003800 x28: 0000000000000002 x27: 0000000000000001
> > [ 228.295050] x26: 0000000000000001 x25: 0000000111580000 x24: 0000000000001000
> > [ 228.302197] x23: 000000ffffc72000 x22: 0000000000000ec0 x21: 0000000000000003
> > [ 228.309342] x20: 0000000000000001 x19: ffff00008574b000 x18: 0000000000000001
> > [ 228.316486] x17: 0000000000000000 x16: 0000000000000001 x15: ffff800080003ad0
> > [ 228.323631] x14: ffff00008574d000 x13: 0000000000000000 x12: 0000000000000001
> > [ 228.330775] x11: 0000000000000001 x10: 0000000000000001 x9 : 0000000000001000
> > [ 228.337921] x8 : ffff00008674c390 x7 : ffff00008674c000 x6 : 0000000000000003
> > [ 228.345066] x5 : 0000000000000003 x4 : 0000000000000001 x3 : 0000000000000002
> > [ 228.352209] x2 : 0000000000000001 x1 : 0000000000000000 x0 : ffff00008574b000
> > [ 228.359356] Call trace:
> > [ 228.361807] __arm_lpae_map+0x388/0x4e4 (P)
> > [ 228.366002] __arm_lpae_map+0x120/0x4e4 (L)
> > [ 228.370198] __arm_lpae_map+0x120/0x4e4
> > [ 228.374042] __arm_lpae_map+0x120/0x4e4
> > [ 228.377886] __arm_lpae_map+0x120/0x4e4
> > [ 228.381730] arm_lpae_map_pages+0x108/0x250
> > [ 228.385922] arm_smmu_map_pages+0x40/0x120
> > [ 228.390029] __iommu_map+0xfc/0x1bc
> > [ 228.393525] iommu_map+0x38/0xc0
> > [ 228.396759] __iommu_dma_map+0xb4/0x1a4
> > [ 228.400604] iommu_dma_map_page+0x14c/0x27c
> > [ 228.404795] dma_map_page_attrs+0x1fc/0x280
> > [ 228.408987] stmmac_tso_xmit+0x2d0/0xbac
> > [ 228.412920] stmmac_xmit+0x230/0xd14
> > [ 228.416505] dev_hard_start_xmit+0x94/0x11c
> > [ 228.420697] sch_direct_xmit+0x8c/0x380
> > [ 228.424540] __qdisc_run+0x11c/0x66c
> > [ 228.428121] net_tx_action+0x168/0x228
> > [ 228.431875] handle_softirqs+0x100/0x244
> > [ 228.435809] __do_softirq+0x14/0x20
> > [ 228.439303] ____do_softirq+0x10/0x20
> > [ 228.442972] call_on_irq_stack+0x24/0x64
> > [ 228.446903] do_softirq_own_stack+0x1c/0x40
> > [ 228.451091] __irq_exit_rcu+0xd4/0x10c
> > [ 228.454847] irq_exit_rcu+0x10/0x1c
> > [ 228.458343] el1_interrupt+0x38/0x68
> > [ 228.461927] el1h_64_irq_handler+0x18/0x24
> > [ 228.466032] el1h_64_irq+0x6c/0x70
> > [ 228.469438] default_idle_call+0x28/0x58 (P)
> > [ 228.473718] default_idle_call+0x24/0x58 (L)
> > [ 228.477998] do_idle+0x1fc/0x260
> > [ 228.481234] cpu_startup_entry+0x34/0x3c
> > [ 228.485163] rest_init+0xdc/0xe0
> > [ 228.488401] console_on_rootfs+0x0/0x6c
> > [ 228.492250] __primary_switched+0x88/0x90
> > [ 228.496270] ---[ end trace 0000000000000000 ]---
> > [ 228.500950] dwc-eth-dwmac 2490000.ethernet: Tx dma map failed
> > --- >8 ---
> >
> > This looks to be slightly different from what Jon was seeing. Looking at
> > the WARN_ON() that triggers this, it seems like for some reason the page
> > is getting mapped twice.
> >
> > Not exactly sure why that would be happening, so adding Robin and Will,
> > maybe they can shed some light on this from the ARM SMMU side.
> >
> > Robin, Will, any idea who could be the culprit here? Is this a map/unmap
> > imbalance or something else entirely?
>
> If valid PTEs are getting left behind in the pagetable, that would indicate
> that a previous dma_unmap_page() was called with a size smaller than its
> original dma_map_page(). Throwing CONFIG_DMA_API_DEBUG at it should
> hopefully shed more light.
Bull's-eye! DMA_API_DEBUG does flag this:
--- >8 ---
[ 60.469121] DMA-API: dwc-eth-dwmac 2490000.ethernet: device driver tries to free DMA memory it has not allocated [device add ress=0x000000ffffcf65c0] [size=66 bytes]
1
[ 60.486534] WARNING: CPU: 0 PID: 0 at kernel/dma/debug.c:972 check_unmap+0x564/0x8f0
[ 60.494493] Modules linked in: snd_soc_tegra210_admaif snd_soc_tegra_pcm snd_soc_tegra210_ope snd_soc_tegra186_asrc snd_soc_tegra210_amx snd_soc_tegra210_mv
c snd_soc_tegra210_mixer snd_soc_tegra210_dmic snd_soc_tegra210_sfc snd_soc_tegra186_dspk snd_soc_tegra210_i2s snd_soc_tegra210_adx tegra_drm drm_dp_aux_bus ce
c drm_display_helper drm_client_lib drm_kms_helper tegra210_adma snd_soc_tegra210_ahub snd_hda_codec_hdmi snd_soc_tegra_audio_graph_card snd_hda_tegra snd_soc_
audio_graph_card snd_hda_codec snd_soc_simple_card_utils at24 snd_hda_core ina3221 tegra_aconnect tegra_xudc tegra_bpmp_thermal host1x fuse drm backlight ipv6
[ 60.549857] CPU: 0 UID: 0 PID: 0 Comm: swapper/0 Tainted: G S 6.13.0-rc1-next-20241203 #31
[ 60.559504] Tainted: [S]=CPU_OUT_OF_SPEC
[ 60.563423] Hardware name: nvidia NVIDIA P2771-0000-500/NVIDIA P2771-0000-500, BIOS 2025.01-rc3-00040-g36352ae2e68e-dirty 01/01/2025
[ 60.575317] pstate: 60000005 (nZCv daif -PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[ 60.582273] pc : check_unmap+0x564/0x8f0
[ 60.586197] lr : check_unmap+0x564/0x8f0
[ 60.590117] sp : ffff800080003b50
[ 60.593428] x29: ffff800080003b50 x28: ffff0000825309c0 x27: ffff0000825313c0
[ 60.600562] x26: 00000000000001b9 x25: 0000000000000001 x24: ffff00008011b410
[ 60.607696] x23: ffff800082059ec0 x22: 0000000000000000 x21: ffff8000825b25c8
[ 60.614829] x20: ffff800080003bc0 x19: 000000ffffcf65c0 x18: 0000000000000006
[ 60.621962] x17: 645b206465746163 x16: 0000000000000000 x15: 0720072007200720
[ 60.629095] x14: ffff800082074960 x13: 0720072007200720 x12: 0720072007200720
[ 60.636229] x11: ffff800082074960 x10: 0000000000000299 x9 : ffff8000820cc960
[ 60.643362] x8 : 0000000000017fe8 x7 : 00000000fffff000 x6 : ffff8000820cc960
[ 60.650496] x5 : 0000000000000000 x4 : 0000000000000000 x3 : 0000000000000000
[ 60.657629] x2 : 0000000000000000 x1 : 0000000000000000 x0 : ffff800082064900
[ 60.664763] Call trace:
[ 60.667206] check_unmap+0x564/0x8f0 (P)
[ 60.671131] check_unmap+0x564/0x8f0 (L)
[ 60.675055] debug_dma_unmap_page+0xac/0xc0
[ 60.679239] dma_unmap_page_attrs+0xf4/0x200
[ 60.683508] stmmac_tx_clean.constprop.0+0x1ac/0x7bc
[ 60.688474] stmmac_napi_poll_tx+0xdc/0x168
[ 60.692657] __napi_poll+0x38/0x180
[ 60.696148] net_rx_action+0x158/0x2c0
[ 60.699897] handle_softirqs+0x100/0x244
[ 60.703821] __do_softirq+0x14/0x20
[ 60.707309] ____do_softirq+0x10/0x20
[ 60.710970] call_on_irq_stack+0x24/0x64
[ 60.714891] do_softirq_own_stack+0x1c/0x40
[ 60.719072] __irq_exit_rcu+0xd4/0x10c
[ 60.722821] irq_exit_rcu+0x10/0x1c
[ 60.726308] el1_interrupt+0x38/0x68
[ 60.729886] el1h_64_irq_handler+0x18/0x24
[ 60.733980] el1h_64_irq+0x6c/0x70
[ 60.737381] default_idle_call+0x28/0x58 (P)
[ 60.741652] default_idle_call+0x24/0x58 (L)
[ 60.745920] do_idle+0x1fc/0x260
[ 60.749149] cpu_startup_entry+0x34/0x3c
[ 60.753068] rest_init+0xdc/0xe0
[ 60.756296] console_on_rootfs+0x0/0x6c
[ 60.760135] __primary_switched+0x88/0x90
[ 60.764146] ---[ end trace 0000000000000000 ]---
--- >8 ---
This doesn't match the location from earlier, but at least there's
something afoot here that needs fixing. I suppose this could simply be
hiding any subsequent errors, so once this is fixed we might see other
similar issues.
Furong, I can look into this some more, but I'm not at all familiar with
this part of the driver, so I don't really know where this could be
originating from. Any pointers would be appreciated. Also, if you think
there's anything I should try, I do have this setup that I can test on
locally.
Thierry
Download attachment "signature.asc" of type "application/pgp-signature" (834 bytes)
Powered by blists - more mailing lists