Date: Fri, 28 Jun 2024 07:32:56 +0900
From: Yunseong Kim <yskelg@...il.com>
To: Paolo Abeni <pabeni@...hat.com>
Cc: Taehee Yoo <ap420073@...il.com>, Pedro Tammela <pctammela@...atatu.com>,
 Austin Kim <austindh.kim@...il.com>, MichelleJin <shjy180909@...il.com>,
 linux-kernel@...r.kernel.org, linux-trace-kernel@...r.kernel.org,
 netdev@...r.kernel.org, pbuk5246@...il.com, stable@...r.kernel.org,
 Yeoreum Yun <yeoreum.yun@....com>, Steven Rostedt <rostedt@...dmis.org>,
 Masami Hiramatsu <mhiramat@...nel.org>,
 Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
 Takashi Iwai <tiwai@...e.de>, "David S. Miller" <davem@...emloft.net>,
 Thomas Hellström <thomas.hellstrom@...ux.intel.com>,
 "Rafael J. Wysocki" <rafael@...nel.org>, Jamal Hadi Salim
 <jhs@...atatu.com>, Cong Wang <xiyou.wangcong@...il.com>,
 Jiri Pirko <jiri@...nulli.us>, Eric Dumazet <edumazet@...gle.com>,
 Jakub Kicinski <kuba@...nel.org>
Subject: Re: [PATCH v4] tracing/net_sched: NULL pointer dereference in
 perf_trace_qdisc_reset()

Hi Paolo,

On 6/27/24 6:14 오후, Paolo Abeni wrote:
> On Tue, 2024-06-25 at 02:33 +0900, yskelg@...il.com wrote:
>> From: Yunseong Kim <yskelg@...il.com>
>>
>> In the TRACE_EVENT(qdisc_reset) NULL dereference occurred from
>>
>>  qdisc->dev_queue->dev <NULL> ->name
>>
>> This situation simulated from bunch of veths and Bluetooth disconnection
>> and reconnection.
>>
>> During qdisc initialization, qdisc was being set to noop_queue.
>> In veth_init_queue, the initial tx_num was reduced back to one,
>> causing the qdisc reset to be called with noop, which led to the kernel
>> panic.
>>
>> I've attached the GitHub gist link that C converted syz-execprogram
>> source code and 3 log of reproduced vmcore-dmesg.
>>
>>  https://gist.github.com/yskelg/cc64562873ce249cdd0d5a358b77d740
>>
>> Yeoreum and I use two fuzzing tool simultaneously.
>>
>> One process with syz-executor : https://github.com/google/syzkaller
>>
>>  $ ./syz-execprog -executor=./syz-executor -repeat=1 -sandbox=setuid \
>>     -enable=none -collide=false log1
>>
>> The other process with perf fuzzer:
>>  https://github.com/deater/perf_event_tests/tree/master/fuzzer
>>
>>  $ perf_event_tests/fuzzer/perf_fuzzer
>>
>> I think this will happen on the kernel version.
>>
>>  Linux kernel version +v6.7.10, +v6.8, +v6.9 and it could happen in v6.10.
>>
>> This occurred from 51270d573a8d. I think this patch is absolutely
>> necessary. Previously, It was showing not intended string value of name.
>>
>> I've reproduced 3 time from my fedora 40 Debug Kernel with any other module
>> or patched.
>>
>>  version: 6.10.0-0.rc2.20240608gitdc772f8237f9.29.fc41.aarch64+debug
>>

>> [ 5301.595872] KASAN: null-ptr-deref in range [0x0000000000000130-0x0000000000000137]
>> [ 5301.595877] Mem abort info:
>> [ 5301.595881]   ESR = 0x0000000096000006
>> [ 5301.595885]   EC = 0x25: DABT (current EL), IL = 32 bits
>> [ 5301.595889]   SET = 0, FnV = 0
>> [ 5301.595893]   EA = 0, S1PTW = 0
>> [ 5301.595896]   FSC = 0x06: level 2 translation fault
>> [ 5301.595900] Data abort info:
>> [ 5301.595903]   ISV = 0, ISS = 0x00000006, ISS2 = 0x00000000
>> [ 5301.595907]   CM = 0, WnR = 0, TnD = 0, TagAccess = 0
>> [ 5301.595911]   GCS = 0, Overlay = 0, DirtyBit = 0, Xs = 0
>> [ 5301.595915] [dfff800000000026] address between user and kernel address ranges
>> [ 5301.595971] Internal error: Oops: 0000000096000006 [#1] SMP
>> …
>> [ 5301.596076] CPU: 2 PID: 102769 Comm:
>> syz-executor.3 Kdump: loaded Tainted:
>>  G        W         -------  ---  6.10.0-0.rc2.20240608gitdc772f8237f9.29.fc41.aarch64+debug #1
>> [ 5301.596080] Hardware name: VMware, Inc. VMware20,1/VBSA,
>>  BIOS VMW201.00V.21805430.BA64.2305221830 05/22/2023
>> [ 5301.596082] pstate: 01400005 (nzcv daif +PAN -UAO -TCO +DIT -SSBS BTYPE=--)
>> [ 5301.596085] pc : strnlen+0x40/0x88
>> [ 5301.596114] lr : trace_event_get_offsets_qdisc_reset+0x6c/0x2b0
>> [ 5301.596124] sp : ffff8000beef6b40
>> [ 5301.596126] x29: ffff8000beef6b40 x28: dfff800000000000 x27: 0000000000000001
>> [ 5301.596131] x26: 6de1800082c62bd0 x25: 1ffff000110aa9e0 x24: ffff800088554f00
>> [ 5301.596136] x23: ffff800088554ec0 x22: 0000000000000130 x21: 0000000000000140
>> [ 5301.596140] x20: dfff800000000000 x19: ffff8000beef6c60 x18: ffff7000115106d8
>> [ 5301.596143] x17: ffff800121bad000 x16: ffff800080020000 x15: 0000000000000006
>> [ 5301.596147] x14: 0000000000000002 x13: ffff0001f3ed8d14 x12: ffff700017ddeda5
>> [ 5301.596151] x11: 1ffff00017ddeda4 x10: ffff700017ddeda4 x9 : ffff800082cc5eec
>> [ 5301.596155] x8 : 0000000000000004 x7 : 00000000f1f1f1f1 x6 : 00000000f2f2f200
>> [ 5301.596158] x5 : 00000000f3f3f3f3 x4 : ffff700017dded80 x3 : 00000000f204f1f1
>> [ 5301.596162] x2 : 0000000000000026 x1 : 0000000000000000 x0 : 0000000000000130
>> [ 5301.596166] Call trace:
>> [ 5301.596175]  strnlen+0x40/0x88
>> [ 5301.596179]  trace_event_get_offsets_qdisc_reset+0x6c/0x2b0
>> [ 5301.596182]  perf_trace_qdisc_reset+0xb0/0x538
>> [ 5301.596184]  __traceiter_qdisc_reset+0x68/0xc0
>> [ 5301.596188]  qdisc_reset+0x43c/0x5e8
>> [ 5301.596190]  netif_set_real_num_tx_queues+0x288/0x770
>> [ 5301.596194]  veth_init_queues+0xfc/0x130 [veth]
>> [ 5301.596198]  veth_newlink+0x45c/0x850 [veth]
>> [ 5301.596202]  rtnl_newlink_create+0x2c8/0x798
>> [ 5301.596205]  __rtnl_newlink+0x92c/0xb60
>> [ 5301.596208]  rtnl_newlink+0xd8/0x130
>> [ 5301.596211]  rtnetlink_rcv_msg+0x2e0/0x890
>> [ 5301.596214]  netlink_rcv_skb+0x1c4/0x380
>> [ 5301.596225]  rtnetlink_rcv+0x20/0x38
>> [ 5301.596227]  netlink_unicast+0x3c8/0x640
>> [ 5301.596231]  netlink_sendmsg+0x658/0xa60
>> [ 5301.596234]  __sock_sendmsg+0xd0/0x180
>> [ 5301.596243]  __sys_sendto+0x1c0/0x280
>> [ 5301.596246]  __arm64_sys_sendto+0xc8/0x150
>> [ 5301.596249]  invoke_syscall+0xdc/0x268
>> [ 5301.596256]  el0_svc_common.constprop.0+0x16c/0x240
>> [ 5301.596259]  do_el0_svc+0x48/0x68
>> [ 5301.596261]  el0_svc+0x50/0x188
>> [ 5301.596265]  el0t_64_sync_handler+0x120/0x130
>> [ 5301.596268]  el0t_64_sync+0x194/0x198
>> [ 5301.596272] Code: eb15001f 54000120 d343fc02 12000801 (38f46842)
>> [ 5301.596285] SMP: stopping secondary CPUs
>> [ 5301.597053] Starting crashdump kernel...
>> [ 5301.597057] Bye!
>>
>> After applying our patch, I didn't find any kernel panic errors.
>>
>> We've found a simple reproducer
>>
>>  # echo 1 > /sys/kernel/debug/tracing/events/qdisc/qdisc_reset/enable
>>
>>  # ip link add veth0 type veth peer name veth1
>>
>>  Error: Unknown device type.

>>
>> Fixes: 51270d573a8d ("tracing/net_sched: Fix tracepoints that save qdisc_dev() as a string")
>> Link: https://lore.kernel.org/lkml/20240229143432.273b4871@gandalf.local.home/t/
>> Cc: netdev@...r.kernel.org
>> Cc: stable@...r.kernel.org # +v6.7.10, +v6.8
>> Tested-by: Yunseong Kim <yskelg@...il.com>
>> Signed-off-by: Yunseong Kim <yskelg@...il.com>
>> Signed-off-by: Yeoreum Yun <yeoreum.yun@....com>
> 
> I took the liberty of dropping the stable tag when applying, because as
> noted from Petro this patch will not address the issue on v6.7.10 nor
> on +v6.8, as they lack the upstream commit 2c92ca849fcc
> ("tracing/treewide: Remove second parameter of __assign_str()") and a
> we need slightly different fix on such trees, including the
> TP_fast_assign() chunk.
> 
> Could you please take care of such backport?
> 
> Thanks,
> 
> Paolo

Thank you Paolo for your hard work.

Yes, I'll try Pedro's review for +v6.7.10, test it and submit a backport
patch.

Link:
https://lore.kernel.org/all/06d0ea61-47ee-4e54-9dfa-a711c5bc07d0@mojatatu.com/


Warm regards,
Yunseong Kim

Powered by blists - more mailing lists