Message-ID: <e2f9da4e-919d-4a98-919d-167726ef6bc7@gmail.com>
Date: Sat, 22 Jun 2024 21:01:55 +0900
From: Yunseong Kim <yskelg@...il.com>
To: Taehee Yoo <ap420073@...il.com>
Cc: Jakub Kicinski <kuba@...nel.org>, Pedro Tammela <pctammela@...atatu.com>,
 netdev@...r.kernel.org, stable@...r.kernel.org,
 Steven Rostedt <rostedt@...dmis.org>, Masami Hiramatsu
 <mhiramat@...nel.org>, Mathieu Desnoyers <mathieu.desnoyers@...icios.com>,
 Takashi Iwai <tiwai@...e.de>, "David S. Miller" <davem@...emloft.net>,
 Thomas Hellström <thomas.hellstrom@...ux.intel.com>,
 "Rafael J. Wysocki" <rafael@...nel.org>, Jamal Hadi Salim
 <jhs@...atatu.com>, Cong Wang <xiyou.wangcong@...il.com>,
 Jiri Pirko <jiri@...nulli.us>, Eric Dumazet <edumazet@...gle.com>,
 Paolo Abeni <pabeni@...hat.com>, Austin Kim <austindh.kim@...il.com>,
 shjy180909@...il.com, linux-kernel@...r.kernel.org,
 linux-trace-kernel@...r.kernel.org, ppbuk5246@...il.com,
 Yeoreum Yun <yeoreum.yun@....com>, virtualization@...ts.linux.dev
Subject: Re: [PATCH v3] tracing/net_sched: NULL pointer dereference in
 perf_trace_qdisc_reset()

Hi,

On 6/22/24 3:12 오후, Yunseong Kim wrote:
> Hi Taehee,
> 
> On 6/22/24 2:50 오후, Taehee Yoo wrote:
>> On Sat, Jun 22, 2024 at 1:58 PM <yskelg@...il.com> wrote:
>>>
>>> From: Yunseong Kim <yskelg@...il.com>
>>>
>>
>> Hi Yunseong,
>> Thanks a lot for this work!
> 
> Thank you Taehee for reviewing our patch. It's greatly appreciated.
> 
>>> During qdisc initialization, qdisc was being set to noop_queue.
>>> In veth_init_queue, the initial tx_num was reduced back to one,
>>> causing the qdisc reset to be called with noop, which led to the kernel panic.
>>>
>>> I've attached the GitHub gist link that C converted syz-execprogram
>>> source code and 3 log of reproduced vmcore-dmesg.
>>>
>>>  https://gist.github.com/yskelg/cc64562873ce249cdd0d5a358b77d740
>>>
>>> Yeoreum and I use two fuzzing tool simultaneously.
>>>
>>> One process with syz-executor : https://github.com/google/syzkaller
>>>
>>>  $ ./syz-execprog -executor=./syz-executor -repeat=1 -sandbox=setuid \
>>>     -enable=none -collide=false log1
>>>
>>> The other process with perf fuzzer:
>>>  https://github.com/deater/perf_event_tests/tree/master/fuzzer
>>>
>>>  $ perf_event_tests/fuzzer/perf_fuzzer
>>>
>>> I think this will happen on the kernel version.
>>>
>>>  Linux kernel version +v6.7.10, +v6.8, +v6.9 and it could happen in v6.10.
>>>
>>> This occurred from 51270d573a8d. I think this patch is absolutely
>>> necessary. Previously, It was showing not intended string value of name.
> 
> 
>> I found a simple reproducer, please use the below command to test this patch.
>>
>> echo 1 > /sys/kernel/debug/tracing/events/enable
>> ip link add veth0 type veth peer name veth1

After applying our patch, I didn't find any message or kernel panic errors.

 # echo 1 > /sys/kernel/debug/tracing/events/qdisc/qdisc_reset/enable
 # ip link add veth0 type veth peer name veth1
 Error: Unknown device type.

However, without our patch applied, I tested Tahee's command line on the
upstream 6.10.0-rc3 kernel using the qdisc_reset event and the ip command.

 $ echo 1 > /sys/kernel/debug/tracing/events/qdisc/qdisc_reset/enable

 $ ip link add veth0 type veth peer name veth1

This make always kernel panic.

Linux version: 6.10.0-rc3

[    0.000000] Linux version 6.10.0-rc3-00164-g44ef20baed8e-dirty
(paran@...ora) (gcc (GCC) 14.1.1 20240522 (Red Hat 14.1.1-4), GNU ld
version 2.41-34.fc40) #20 SMP PREEMPT Sat Jun 15 16:51:25 KST 2024

Kernel panic message:

[  615.236484] Internal error: Oops: 0000000096000005 [#1] PREEMPT SMP
[  615.237250] Dumping ftrace buffer:
[  615.237679]    (ftrace buffer empty)
[  615.238097] Modules linked in: veth crct10dif_ce virtio_gpu
virtio_dma_buf drm_shmem_helper drm_kms_helper zynqmp_fpga xilinx_can
xilinx_spi xilinx_selectmap xilinx_core xilinx_pr_decoupler versal_fpga
uvcvideo uvc videobuf2_vmalloc videobuf2_memops videobuf2_v4l2 videodev
videobuf2_common mc usbnet deflate zstd ubifs ubi rcar_canfd rcar_can
omap_mailbox ntb_msi_test ntb_hw_epf lattice_sysconfig_spi
lattice_sysconfig ice40_spi gpio_xilinx dwmac_altr_socfpga mdio_regmap
stmmac_platform stmmac pcs_xpcs dfl_fme_region dfl_fme_mgr dfl_fme_br
dfl_afu dfl fpga_region fpga_bridge can can_dev br_netfilter bridge stp
llc atl1c ath11k_pci mhi ath11k_ahb ath11k qmi_helpers ath10k_sdio
ath10k_pci ath10k_core ath mac80211 libarc4 cfg80211 drm fuse backlight ipv6
Jun 22 02:36:5[3   6k152.62-4sm98k4-0k]v  kCePUr:n e1l :P IUDn:a b4le6
8t oC ohmma: nidpl eN oketr nteali nptaedg i6n.g1 0re.0q-urecs3t- 0at0
1v6i4r-tgu4a4le fa2d0dbraeeds0se-dir tyd f#f2f08
  615.252376] Hardware name: linux,dummy-virt (DT)
[  615.253220] pstate: 80400005 (Nzcv daif +PAN -UAO -TCO -DIT -SSBS
BTYPE=--)
[  615.254433] pc : strnlen+0x6c/0xe0
[  615.255096] lr : trace_event_get_offsets_qdisc_reset+0x94/0x3d0
[  615.256088] sp : ffff800080b269a0
[  615.256615] x29: ffff800080b269a0 x28: ffffc070f3f98500 x27:
0000000000000001
[  615.257831] x26: 0000000000000010 x25: ffffc070f3f98540 x24:
ffffc070f619cf60
[  615.259020] x23: 0000000000000128 x22: 0000000000000138 x21:
dfff800000000000
[  615.260241] x20: ffffc070f631ad00 x19: 0000000000000128 x18:
ffffc070f448b800
[  615.261454] x17: 0000000000000000 x16: 0000000000000001 x15:
ffffc070f4ba2a90
[  615.262635] x14: ffff700010164d73 x13: 1ffff80e1e8d5eb3 x12:
1ffff00010164d72
[  615.263877] x11: ffff700010164d72 x10: dfff800000000000 x9 :
ffffc070e85d6184
[  615.265047] x8 : ffffc070e4402070 x7 : 000000000000f1f1 x6 :
000000001504a6d3
[  615.266336] x5 : ffff28ca21122140 x4 : ffffc070f5043ea8 x3 :
0000000000000000
[  615.267528] x2 : 0000000000000025 x1 : 0000000000000000 x0 :
0000000000000000
[  615.268747] Call trace:
[  615.269180]  strnlen+0x6c/0xe0
[  615.269767]  trace_event_get_offsets_qdisc_reset+0x94/0x3d0
[  615.270716]  trace_event_raw_event_qdisc_reset+0xe8/0x4e8
[  615.271667]  __traceiter_qdisc_reset+0xa0/0x140
[  615.272499]  qdisc_reset+0x554/0x848
[  615.273134]  netif_set_real_num_tx_queues+0x360/0x9a8
[  615.274050]  veth_init_queues+0x110/0x220 [veth]
[  615.275110]  veth_newlink+0x538/0xa50 [veth]
[  615.276172]  __rtnl_newlink+0x11e4/0x1bc8
[  615.276944]  rtnl_newlink+0xac/0x120
[  615.277657]  rtnetlink_rcv_msg+0x4e4/0x1370
[  615.278409]  netlink_rcv_skb+0x25c/0x4f0
[  615.279122]  rtnetlink_rcv+0x48/0x70
[  615.279769]  netlink_unicast+0x5a8/0x7b8
[  615.280462]  netlink_sendmsg+0xa70/0x1190

> The perf event is activated by perf_fuzzer, and it's indeed a similar
> environment with veth.
> 
>> In my machine, the splat looks like:
>>
>> BUG: kernel NULL pointer dereference, address: 0000000000000130
>> #PF: supervisor read access in kernel mode

>> Thanks a lot!
>> Taehee Yoo
> 
> I think this bug might cause inconvenience for developers working on net
> devices driver in a virtual machine when they use tracing.
> 
> I'm appreciate your effort in reproducing it.
> 
> Warm Regards,
> Yunseong Kim
I believe our patch can prevent panics in all stable kernels released
after +v6.7.10.

Warm Regards,
Yunseong Kim

Powered by blists - more mailing lists