| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <CAMj1kXH6STkFX-SocCiqRgFwkQFZEG=DW6hu6H9W7Egxm2icrw@mail.gmail.com>
Date: Fri, 30 Jun 2023 13:32:33 +0200
From: Ard Biesheuvel <ardb@...nel.org>
To: Tetsuo Handa <penguin-kernel@...ove.sakura.ne.jp>
Cc: Alexander Potapenko <glider@...gle.com>, Boris Pismenny <borisp@...dia.com>,
John Fastabend <john.fastabend@...il.com>, Jakub Kicinski <kuba@...nel.org>, herbert@...dor.apana.org.au,
linux-crypto@...r.kernel.org, syzkaller-bugs@...glegroups.com,
syzbot <syzbot+828dfc12440b4f6f305d@...kaller.appspotmail.com>,
Eric Biggers <ebiggers@...nel.org>, Aviad Yehezkel <aviadye@...dia.com>,
Daniel Borkmann <daniel@...earbox.net>, netdev@...r.kernel.org,
"David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
Paolo Abeni <pabeni@...hat.com>
Subject: Re: [PATCH] net: tls: enable __GFP_ZERO upon tls_init()
On Fri, 30 Jun 2023 at 13:12, Tetsuo Handa
<penguin-kernel@...ove.sakura.ne.jp> wrote:
>
> On 2023/06/30 19:18, Ard Biesheuvel wrote:
> > On Fri, 30 Jun 2023 at 12:11, Alexander Potapenko <glider@...gle.com> wrote:
> >>
> >> On Fri, Jun 30, 2023 at 12:02 PM Ard Biesheuvel <ardb@...nel.org> wrote:
> >>>
> >>> On Fri, 30 Jun 2023 at 11:53, Tetsuo Handa
> >>> <penguin-kernel@...ove.sakura.ne.jp> wrote:
> >>>>
> >>>> On 2023/06/30 18:36, Ard Biesheuvel wrote:
> >>>>> Why are you sending this now?
> >>>>
> >>>> Just because this is currently top crasher and I can reproduce locally.
> >>>>
> >>>>> Do you have a reproducer for this issue?
> >>>>
> >>>> Yes. https://syzkaller.appspot.com/text?tag=ReproC&x=12931621900000 works.
> >>>>
> >>>
> >>> Could you please share your kernel config and the resulting kernel log
> >>> when running the reproducer? I'll try to reproduce locally as well,
> >>> and see if I can figure out what is going on in the crypto layer
> >>
> >> The config together with the repro is available at
> >> https://syzkaller.appspot.com/bug?extid=828dfc12440b4f6f305d, see the
> >> latest row of the "Crashes" table that contains a C repro.
>
> Kernel is commit e6bc8833d80f of https://github.com/google/kmsan/commits/master .
That commit does not exist in that repo. Does it matter?
> Config is available in the dashboard page, but a smaller one is available at
> https://I-love.SAKURA.ne.jp/tmp/config-6.4.0-rc7-kmsan .
>
Thanks - I'll try to rebuild v6.4-rc7 with that.
> I'm using a debug printk() patch shown below.
>
> ----------------------------------------
> diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
> index 1a53c8f481e9..b32bb015995c 100644
> --- a/net/tls/tls_sw.c
> +++ b/net/tls/tls_sw.c
> @@ -1210,7 +1210,8 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page,
> if (!sk_stream_memory_free(sk))
> goto wait_for_sndbuf;
> alloc_payload:
> - ret = tls_alloc_encrypted_msg(sk, required_size);
> + ret = tls_alloc_encrypted_msg(sk, required_size); /////
> + pr_info("required_size=%d ret=%d\n", required_size, ret);
> if (ret) {
> if (ret != -ENOSPC)
> goto wait_for_memory;
> @@ -1232,7 +1233,9 @@ static int tls_sw_do_sendpage(struct sock *sk, struct page *page,
>
> tls_ctx->pending_open_record_frags = true;
> if (full_record || eor || sk_msg_full(msg_pl)) {
> - ret = bpf_exec_tx_verdict(msg_pl, sk, full_record,
> + pr_info("full_record=%d eor=%d sk_msg_full(msg_pl)=%d copied=%d\n",
> + full_record, eor, sk_msg_full(msg_pl), copied);
> + ret = bpf_exec_tx_verdict(msg_pl, sk, full_record, /////
> record_type, &copied, flags);
> if (ret) {
> if (ret == -EINPROGRESS)
> ----------------------------------------
>
> Output (on Ubuntu 22.04 on Oracle VM VirtualBox) is shown below.
> Please check tendency of the sum of required_size= values up to the full_record= line.
> It seems that the value of required_size= might vary depending on the timings, but
> the sum of the values seems to have some rule.
>
> 4125+8221+12317+16413=41076 (the lower 4 bits are 0100)
> 2461+6557+10653+14749+16413=50833 (the lower 4 bits are 0001)
> 2461+6573+10669+14765+16413=50881 (the lower 4 bits are 0001)
>
> KMSAN reports this problem when the lower 4 bits became 0001 for the second time.
> Unless KMSAN's reporting is asynchronous, maybe the reason of "for the second time"
> part is that the previous state is relevant...
>
> ----------------------------------------
> [ 157.471712][ T3414] required_size=4125 ret=0
> [ 157.475879][ T3414] required_size=8221 ret=0
> [ 157.480471][ T3414] required_size=12317 ret=0
> [ 157.484604][ T3414] required_size=16413 ret=0
> [ 157.490499][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096
> [ 157.513772][ T3414] required_size=4125 ret=0
> [ 157.523782][ T3414] required_size=8221 ret=0
> [ 157.533658][ T3414] required_size=12317 ret=0
> [ 157.539579][ T3414] required_size=16413 ret=0
> [ 157.543785][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096
> [ 157.572869][ T3414] required_size=4125 ret=0
> [ 157.579350][ T3414] required_size=8221 ret=0
> [ 157.584699][ T3414] required_size=12317 ret=0
> [ 157.591756][ T3414] required_size=16413 ret=0
> [ 157.595891][ T3414] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=4096
> [ 157.790734][ T3424] required_size=2461 ret=0
> [ 157.800725][ T3424] required_size=6557 ret=0
> [ 157.804560][ T3424] required_size=10653 ret=0
> [ 157.808433][ T3424] required_size=14749 ret=0
> [ 157.810125][ T3424] required_size=16413 ret=0
> [ 157.829564][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1664
> [ 157.848397][ T3424] required_size=2461 ret=0
> [ 157.854875][ T3424] required_size=6573 ret=0
> [ 157.860883][ T3424] required_size=10669 ret=0
> [ 157.865463][ T3424] required_size=14765 ret=0
> [ 157.871794][ T3424] required_size=16413 ret=0
> [ 157.877333][ T3424] full_record=1 eor=0 sk_msg_full(msg_pl)=0 copied=1648
> [ 157.885187][ T3424] =====================================================
> [ 157.887262][ T3424] BUG: KMSAN: uninit-value in aes_encrypt+0x1692/0x1fa0
> [ 157.887262][ T3424] aes_encrypt+0x1692/0x1fa0
> [ 157.887262][ T3424] aesti_encrypt+0xe1/0x160
> [ 157.887262][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0
> [ 157.887262][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0
> [ 157.887262][ T3424] shash_ahash_finup+0x79d/0xd00
> [ 157.887262][ T3424] shash_async_finup+0xbf/0x110
> [ 157.887262][ T3424] crypto_ahash_finup+0x244/0x500
> [ 157.887262][ T3424] crypto_ccm_auth+0x14df/0x15a0
> [ 157.887262][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0
> [ 157.887262][ T3424] crypto_aead_encrypt+0x116/0x1a0
> [ 157.887262][ T3424] tls_push_record+0x2bbe/0x3bf0
> [ 157.887262][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530
> [ 157.887262][ T3424] tls_sw_do_sendpage+0x1779/0x21f0
> [ 157.887262][ T3424] tls_sw_sendpage+0x247/0x2b0
> [ 157.887262][ T3424] inet_sendpage+0x1de/0x2f0
> [ 157.887262][ T3424] kernel_sendpage+0x4cc/0x940
> [ 158.004827][ T3424] sock_sendpage+0x162/0x220
> [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0
> [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010
> [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0
> [ 158.004827][ T3424] do_splice+0x26d8/0x32f0
> [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0
> [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200
> [ 158.004827][ T3424] do_syscall_64+0x41/0x90
> [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 158.004827][ T3424]
> [ 158.004827][ T3424] Uninit was stored to memory at:
> [ 158.004827][ T3424] __crypto_xor+0x285/0x1700
> [ 158.004827][ T3424] crypto_cbcmac_digest_update+0x2ba/0x5a0
> [ 158.004827][ T3424] shash_ahash_finup+0x79d/0xd00
> [ 158.004827][ T3424] shash_async_finup+0xbf/0x110
> [ 158.004827][ T3424] crypto_ahash_finup+0x244/0x500
> [ 158.004827][ T3424] crypto_ccm_auth+0x14df/0x15a0
> [ 158.004827][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0
> [ 158.004827][ T3424] crypto_aead_encrypt+0x116/0x1a0
> [ 158.004827][ T3424] tls_push_record+0x2bbe/0x3bf0
> [ 158.004827][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530
> [ 158.004827][ T3424] tls_sw_do_sendpage+0x1779/0x21f0
> [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0
> [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0
> [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940
> [ 158.004827][ T3424] sock_sendpage+0x162/0x220
> [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0
> [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010
> [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0
> [ 158.004827][ T3424] do_splice+0x26d8/0x32f0
> [ 158.004827][ T3424] __se_sys_splice+0x81f/0xba0
> [ 158.004827][ T3424] __x64_sys_splice+0x1a1/0x200
> [ 158.004827][ T3424] do_syscall_64+0x41/0x90
> [ 158.004827][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 158.004827][ T3424]
> [ 158.004827][ T3424] Uninit was created at:
> [ 158.004827][ T3424] __alloc_pages+0x925/0x1050
> [ 158.004827][ T3424] alloc_pages+0xe30/0x11b0
> [ 158.004827][ T3424] skb_page_frag_refill+0x362/0x910
> [ 158.004827][ T3424] sk_page_frag_refill+0xa2/0x1c0
> [ 158.004827][ T3424] sk_msg_alloc+0x278/0x1560
> [ 158.004827][ T3424] tls_sw_do_sendpage+0xbec/0x21f0
> [ 158.004827][ T3424] tls_sw_sendpage+0x247/0x2b0
> [ 158.004827][ T3424] inet_sendpage+0x1de/0x2f0
> [ 158.004827][ T3424] kernel_sendpage+0x4cc/0x940
> [ 158.004827][ T3424] sock_sendpage+0x162/0x220
> [ 158.004827][ T3424] pipe_to_sendpage+0x3df/0x4f0
> [ 158.004827][ T3424] __splice_from_pipe+0x5c7/0x1010
> [ 158.004827][ T3424] generic_splice_sendpage+0x1c6/0x2a0
> [ 158.260226][ T3424] do_splice+0x26d8/0x32f0
> [ 158.260226][ T3424] __se_sys_splice+0x81f/0xba0
> [ 158.260226][ T3424] __x64_sys_splice+0x1a1/0x200
> [ 158.260226][ T3424] do_syscall_64+0x41/0x90
> [ 158.260226][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 158.260226][ T3424]
> [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Not tainted 6.4.0-rc7-ge6bc8833d80f-dirty #26
> [ 158.260226][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
> [ 158.260226][ T3424] =====================================================
> [ 158.260226][ T3424] Disabling lock debugging due to kernel taint
> [ 158.260226][ T3424] Kernel panic - not syncing: kmsan.panic set ...
> [ 158.260226][ T3424] CPU: 7 PID: 3424 Comm: a.out Tainted: G B 6.4.0-rc7-ge6bc8833d80f-dirty #26
> [ 158.320898][ T3424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
> [ 158.334186][ T3424] Call Trace:
> [ 158.334186][ T3424] <TASK>
> [ 158.334186][ T3424] dump_stack_lvl+0x1f6/0x280
> [ 158.334186][ T3424] dump_stack+0x29/0x30
> [ 158.334186][ T3424] panic+0x4e7/0xc60
> [ 158.334186][ T3424] ? add_taint+0x185/0x210
> [ 158.334186][ T3424] kmsan_report+0x2d1/0x2e0
> [ 158.334186][ T3424] ? __msan_warning+0x98/0x120
> [ 158.334186][ T3424] ? aes_encrypt+0x1692/0x1fa0
> [ 158.334186][ T3424] ? aesti_encrypt+0xe1/0x160
> [ 158.334186][ T3424] ? crypto_cipher_encrypt_one+0x1d1/0x2e0
> [ 158.334186][ T3424] ? crypto_cbcmac_digest_update+0x3ff/0x5a0
> [ 158.334186][ T3424] ? shash_ahash_finup+0x79d/0xd00
> [ 158.334186][ T3424] ? shash_async_finup+0xbf/0x110
> [ 158.334186][ T3424] ? crypto_ahash_finup+0x244/0x500
> [ 158.334186][ T3424] ? crypto_ccm_auth+0x14df/0x15a0
> [ 158.334186][ T3424] ? crypto_ccm_encrypt+0x2ad/0x8b0
> [ 158.334186][ T3424] ? crypto_aead_encrypt+0x116/0x1a0
> [ 158.334186][ T3424] ? tls_push_record+0x2bbe/0x3bf0
> [ 158.334186][ T3424] ? bpf_exec_tx_verdict+0x5ba/0x2530
> [ 158.334186][ T3424] ? tls_sw_do_sendpage+0x1779/0x21f0
> [ 158.334186][ T3424] ? tls_sw_sendpage+0x247/0x2b0
> [ 158.334186][ T3424] ? inet_sendpage+0x1de/0x2f0
> [ 158.334186][ T3424] ? kernel_sendpage+0x4cc/0x940
> [ 158.334186][ T3424] ? sock_sendpage+0x162/0x220
> [ 158.334186][ T3424] ? pipe_to_sendpage+0x3df/0x4f0
> [ 158.334186][ T3424] ? __splice_from_pipe+0x5c7/0x1010
> [ 158.334186][ T3424] ? generic_splice_sendpage+0x1c6/0x2a0
> [ 158.334186][ T3424] ? do_splice+0x26d8/0x32f0
> [ 158.334186][ T3424] ? __se_sys_splice+0x81f/0xba0
> [ 158.334186][ T3424] ? __x64_sys_splice+0x1a1/0x200
> [ 158.334186][ T3424] ? do_syscall_64+0x41/0x90
> [ 158.334186][ T3424] ? entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 158.334186][ T3424] ? filter_irq_stacks+0xb9/0x230
> [ 158.334186][ T3424] ? __stack_depot_save+0x22/0x490
> [ 158.334186][ T3424] ? kmsan_internal_set_shadow_origin+0x66/0xe0
> [ 158.334186][ T3424] ? kmsan_internal_chain_origin+0x110/0x120
> [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0
> [ 158.334186][ T3424] __msan_warning+0x98/0x120
> [ 158.334186][ T3424] aes_encrypt+0x1692/0x1fa0
> [ 158.334186][ T3424] aesti_encrypt+0xe1/0x160
> [ 158.334186][ T3424] crypto_cipher_encrypt_one+0x1d1/0x2e0
> [ 158.334186][ T3424] ? aesti_set_key+0xb0/0xb0
> [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0
> [ 158.334186][ T3424] crypto_cbcmac_digest_update+0x3ff/0x5a0
> [ 158.334186][ T3424] ? crypto_cbcmac_digest_init+0x140/0x140
> [ 158.334186][ T3424] shash_ahash_finup+0x79d/0xd00
> [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0
> [ 158.334186][ T3424] shash_async_finup+0xbf/0x110
> [ 158.334186][ T3424] crypto_ahash_finup+0x244/0x500
> [ 158.334186][ T3424] ? shash_async_final+0x3d0/0x3d0
> [ 158.334186][ T3424] crypto_ccm_auth+0x14df/0x15a0
> [ 158.334186][ T3424] crypto_ccm_encrypt+0x2ad/0x8b0
> [ 158.334186][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0
> [ 158.334186][ T3424] ? crypto_ccm_setauthsize+0x100/0x100
> [ 158.334186][ T3424] crypto_aead_encrypt+0x116/0x1a0
> [ 158.653332][ T3424] tls_push_record+0x2bbe/0x3bf0
> [ 158.653332][ T3424] bpf_exec_tx_verdict+0x5ba/0x2530
> [ 158.653332][ T3424] ? _printk+0x181/0x1b0
> [ 158.653332][ T3424] ? tls_sw_do_sendpage+0xc81/0x21f0
> [ 158.653332][ T3424] tls_sw_do_sendpage+0x1779/0x21f0
> [ 158.653332][ T3424] tls_sw_sendpage+0x247/0x2b0
> [ 158.653332][ T3424] ? tls_sw_do_sendpage+0x21f0/0x21f0
> [ 158.653332][ T3424] inet_sendpage+0x1de/0x2f0
> [ 158.653332][ T3424] ? inet_sendmsg+0x1d0/0x1d0
> [ 158.653332][ T3424] kernel_sendpage+0x4cc/0x940
> [ 158.653332][ T3424] sock_sendpage+0x162/0x220
> [ 158.653332][ T3424] pipe_to_sendpage+0x3df/0x4f0
> [ 158.653332][ T3424] ? sock_fasync+0x240/0x240
> [ 158.653332][ T3424] __splice_from_pipe+0x5c7/0x1010
> [ 158.653332][ T3424] ? generic_splice_sendpage+0x2a0/0x2a0
> [ 158.653332][ T3424] generic_splice_sendpage+0x1c6/0x2a0
> [ 158.653332][ T3424] ? iter_file_splice_write+0x1a30/0x1a30
> [ 158.653332][ T3424] do_splice+0x26d8/0x32f0
> [ 158.653332][ T3424] ? kmsan_get_shadow_origin_ptr+0x4d/0xa0
> [ 158.653332][ T3424] ? __se_sys_splice+0x292/0xba0
> [ 158.653332][ T3424] ? __msan_metadata_ptr_for_load_8+0x24/0x40
> [ 158.653332][ T3424] ? filter_irq_stacks+0xb9/0x230
> [ 158.653332][ T3424] __se_sys_splice+0x81f/0xba0
> [ 158.870673][ T3424] __x64_sys_splice+0x1a1/0x200
> [ 158.870673][ T3424] do_syscall_64+0x41/0x90
> [ 158.870673][ T3424] entry_SYSCALL_64_after_hwframe+0x63/0xcd
> [ 158.870673][ T3424] RIP: 0033:0x7f6bbd51ea3d
> [ 158.895223][ T3424] Code: 5b 41 5c c3 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d c3 a3 0f 00 f7 d8 64 89 01 48
> [ 158.895223][ T3424] RSP: 002b:00007f6bbd731e08 EFLAGS: 00000246 ORIG_RAX: 0000000000000113
> [ 158.895223][ T3424] RAX: ffffffffffffffda RBX: 000055ccd9ea6080 RCX: 00007f6bbd51ea3d
> [ 158.895223][ T3424] RDX: 0000000000000004 RSI: 0000000000000000 RDI: 0000000000000003
> [ 158.895223][ T3424] RBP: 000055ccd9ea41f4 R08: 00080000fffffffc R09: 0000000000000000
> [ 158.895223][ T3424] R10: 0000000000000000 R11: 0000000000000246 R12: 0100000000000000
> [ 158.895223][ T3424] R13: e65b75b4ec4292eb R14: f2300cdb85a45425 R15: 000055ccd9ea6088
> [ 159.041467][ T3424] </TASK>
> [ 159.041467][ T3424] Kernel Offset: disabled
> [ 159.041467][ T3424] Rebooting in 10 seconds..
> ----------------------------------------
>
> >
> > Could you explain why that bug contains ~50 reports that seem entirely
> > unrelated? AIUI, this actual issue has not been reproduced since
> > 2020??
>
> Multiple different bugs are reported as the same problem.
> Reproducer is available for only bpf_exec_tx_verdict() one, and the reproducer still works.
>
> >
> >
> >> Config: https://syzkaller.appspot.com/text?tag=KernelConfig&x=ee5f7a0b2e48ed66
> >> Report: https://syzkaller.appspot.com/text?tag=CrashReport&x=1325260d900000
> >> Syz repro: https://syzkaller.appspot.com/text?tag=ReproSyz&x=11af973e900000
> >> C repro: https://syzkaller.appspot.com/text?tag=ReproC&x=163a1e45900000
> >>
> >> The bug is reproducible for me locally as well (and Tetsuo's patch
> >> makes it disappear, although I have no opinion on its correctness).
> >
> > What I'd like to do is run a kernel plus initrd locally in OVMF and
> > reproduce the issue - can I do that without all the syzkaller
> > machinery?
>
> I'm using Ubuntu 22.04 on Oracle VM VirtualBox.
> I don't know if this can be reproduced with kernel plus initrd only. But
> since the C reproducer is standalone, syzkaller machinery is not involved.
>
Powered by blists - more mailing lists