lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Thu, 24 Dec 2020 18:46:18 +0000
From:   Ignat Korchagin <ignat@...udflare.com>
To:     Herbert Xu <herbert@...dor.apana.org.au>
Cc:     "Maciej S. Szmigiero" <mail@...iej.szmigiero.name>,
        Alasdair G Kergon <agk@...hat.com>,
        Mike Snitzer <snitzer@...hat.com>,
        device-mapper development <dm-devel@...hat.com>,
        dm-crypt@...ut.de, linux-kernel <linux-kernel@...r.kernel.org>,
        Eric Biggers <ebiggers@...nel.org>,
        Damien Le Moal <Damien.LeMoal@....com>,
        Mikulas Patocka <mpatocka@...hat.com>,
        kernel-team <kernel-team@...udflare.com>,
        Nobuto Murata <nobuto.murata@...onical.com>,
        Chris Mason <clm@...com>, Josef Bacik <josef@...icpanda.com>,
        David Sterba <dsterba@...e.com>, linux-btrfs@...r.kernel.org,
        linux-crypto <linux-crypto@...r.kernel.org>
Subject: Re: dm-crypt with no_read_workqueue and no_write_workqueue + btrfs
 scrub = BUG()

On Wed, Dec 23, 2020 at 8:57 PM Herbert Xu <herbert@...dor.apana.org.au> wrote:
>
> On Wed, Dec 23, 2020 at 04:37:34PM +0100, Maciej S. Szmigiero wrote:
> >
> > It looks like to me that the skcipher API might not be safe to
> > call from a softirq context, after all.
>
> skcipher is safe to use in a softirq.  The problem is only in
> dm-crypt where it tries to allocate memory with GFP_NOIO.

Hm.. After eliminating the GFP_NOIO (as well as some other sleeping
paths) from dm-crypt softirq code I still hit an occasional crash in
my extreme setup (QEMU with 1 CPU and cryptd_max_cpu_qlen set to 1)
(decoded with stacktrace_decode.sh):

[   89.324723] BUG: kernel NULL pointer dereference, address: 0000000000000008
[   89.325713] #PF: supervisor write access in kernel mode
[   89.326460] #PF: error_code(0x0002) - not-present page
[   89.327211] PGD 0 P4D 0
[   89.327589] Oops: 0002 [#1] PREEMPT SMP PTI
[   89.328200] CPU: 0 PID: 21 Comm: kworker/0:1 Not tainted 5.10.0+ #79
[   89.329109] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996),
BIOS 0.0.0 02/06/2015
[   89.330284] Workqueue: cryptd cryptd_queue_worker
[   89.330999] RIP: 0010:crypto_dequeue_request
(/cfsetup_build/./include/linux/list.h:112
/cfsetup_build/./include/linux/list.h:135
/cfsetup_build/./include/linux/list.h:146
/cfsetup_build/crypto/algapi.c:957)
[ 89.331757] Code: e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 e8 c2 88 fe ff 4c
8b 23 48 c7 c6 e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 49 8b 14 24 49 8b 44
24 08 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
48
All code
========
   0: e9 c9 d0 a8 48        jmpq   0x48a8d0ce
   5: c7 c7 f9 c9 d0 a8    mov    $0xa8d0c9f9,%edi
   b: e8 c2 88 fe ff        callq  0xfffffffffffe88d2
  10: 4c 8b 23              mov    (%rbx),%r12
  13: 48 c7 c6 e9 c9 d0 a8 mov    $0xffffffffa8d0c9e9,%rsi
  1a: 48 c7 c7 f9 c9 d0 a8 mov    $0xffffffffa8d0c9f9,%rdi
  21: 49 8b 14 24          mov    (%r12),%rdx
  25: 49 8b 44 24 08        mov    0x8(%r12),%rax
  2a:* 48 89 42 08          mov    %rax,0x8(%rdx) <-- trapping instruction
  2e: 48 89 10              mov    %rdx,(%rax)
  31: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
  38: 00 ad de
  3b: 49 89 04 24          mov    %rax,(%r12)
  3f: 48                    rex.W

Code starting with the faulting instruction
===========================================
   0: 48 89 42 08          mov    %rax,0x8(%rdx)
   4: 48 89 10              mov    %rdx,(%rax)
   7: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
   e: 00 ad de
  11: 49 89 04 24          mov    %rax,(%r12)
  15: 48                    rex.W
[   89.334414] RSP: 0018:ffffba64c00bbe68 EFLAGS: 00010246
[   89.335165] RAX: 0000000000000000 RBX: ffff9b9d6fc28d88 RCX: 0000000000000000
[   89.336182] RDX: 0000000000000000 RSI: ffffffffa8d0c9e9 RDI: ffffffffa8d0c9f9
[   89.337204] RBP: 0000000000000000 R08: ffffffffa906e708 R09: 0000000000000058
[   89.338208] R10: ffffffffa9068720 R11: 00000000fffffc00 R12: ffff9b9a43797478
[   89.339216] R13: 0000000000000020 R14: ffff9b9d6fc28e00 R15: 0000000000000000
[   89.340231] FS:  0000000000000000(0000) GS:ffff9b9d6fc00000(0000)
knlGS:0000000000000000
[   89.341376] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   89.342207] CR2: 0000000000000008 CR3: 000000014cd76002 CR4: 0000000000170ef0
[   89.343238] Call Trace:
[   89.343609] cryptd_queue_worker (/cfsetup_build/crypto/cryptd.c:172)
[   89.344218] process_one_work
(/cfsetup_build/./arch/x86/include/asm/preempt.h:26
/cfsetup_build/kernel/workqueue.c:2284)
[   89.344821] ? rescuer_thread (/cfsetup_build/kernel/workqueue.c:2364)
[   89.345399] worker_thread
(/cfsetup_build/./include/linux/list.h:282
/cfsetup_build/kernel/workqueue.c:2422)
[   89.345923] ? rescuer_thread (/cfsetup_build/kernel/workqueue.c:2364)
[   89.346504] kthread (/cfsetup_build/kernel/kthread.c:292)
[   89.346986] ? kthread_create_worker_on_cpu
(/cfsetup_build/kernel/kthread.c:245)
[   89.347713] ret_from_fork (/cfsetup_build/arch/x86/entry/entry_64.S:302)
[   89.348255] Modules linked in:
[   89.348708] CR2: 0000000000000008
[   89.349197] ---[ end trace b7e9618b4122ed3b ]---
[   89.349863] RIP: 0010:crypto_dequeue_request
(/cfsetup_build/./include/linux/list.h:112
/cfsetup_build/./include/linux/list.h:135
/cfsetup_build/./include/linux/list.h:146
/cfsetup_build/crypto/algapi.c:957)
[ 89.350606] Code: e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 e8 c2 88 fe ff 4c
8b 23 48 c7 c6 e9 c9 d0 a8 48 c7 c7 f9 c9 d0 a8 49 8b 14 24 49 8b 44
24 08 <48> 89 42 08 48 89 10 48 b8 00 01 00 00 00 00 ad de 49 89 04 24
48
All code
========
   0: e9 c9 d0 a8 48        jmpq   0x48a8d0ce
   5: c7 c7 f9 c9 d0 a8    mov    $0xa8d0c9f9,%edi
   b: e8 c2 88 fe ff        callq  0xfffffffffffe88d2
  10: 4c 8b 23              mov    (%rbx),%r12
  13: 48 c7 c6 e9 c9 d0 a8 mov    $0xffffffffa8d0c9e9,%rsi
  1a: 48 c7 c7 f9 c9 d0 a8 mov    $0xffffffffa8d0c9f9,%rdi
  21: 49 8b 14 24          mov    (%r12),%rdx
  25: 49 8b 44 24 08        mov    0x8(%r12),%rax
  2a:* 48 89 42 08          mov    %rax,0x8(%rdx) <-- trapping instruction
  2e: 48 89 10              mov    %rdx,(%rax)
  31: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
  38: 00 ad de
  3b: 49 89 04 24          mov    %rax,(%r12)
  3f: 48                    rex.W

Code starting with the faulting instruction
===========================================
   0: 48 89 42 08          mov    %rax,0x8(%rdx)
   4: 48 89 10              mov    %rdx,(%rax)
   7: 48 b8 00 01 00 00 00 movabs $0xdead000000000100,%rax
   e: 00 ad de
  11: 49 89 04 24          mov    %rax,(%r12)
  15: 48                    rex.W
[   89.353266] RSP: 0018:ffffba64c00bbe68 EFLAGS: 00010246
[   89.354003] RAX: 0000000000000000 RBX: ffff9b9d6fc28d88 RCX: 0000000000000000
[   89.355048] RDX: 0000000000000000 RSI: ffffffffa8d0c9e9 RDI: ffffffffa8d0c9f9
[   89.356063] RBP: 0000000000000000 R08: ffffffffa906e708 R09: 0000000000000058
[   89.357082] R10: ffffffffa9068720 R11: 00000000fffffc00 R12: ffff9b9a43797478
[   89.358088] R13: 0000000000000020 R14: ffff9b9d6fc28e00 R15: 0000000000000000
[   89.359127] FS:  0000000000000000(0000) GS:ffff9b9d6fc00000(0000)
knlGS:0000000000000000
[   89.360296] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   89.361129] CR2: 0000000000000008 CR3: 000000014cd76002 CR4: 0000000000170ef0
[   89.362160] Kernel panic - not syncing: Fatal exception in interrupt
[   89.363145] Kernel Offset: 0x26000000 from 0xffffffff81000000
(relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[   89.364730] ---[ end Kernel panic - not syncing: Fatal exception in
interrupt ]---

This happens when running dm-crypt with no_read_workqueues on top of
an emulated NVME in QEMU (NVME driver "completes" IO in IRQ context).
Somehow sending decryption requests to cryptd in some fashion in
softirq context corrupts the crypto queue it seems.

Regards,
Ignat


> Cheers,
> --
> Email: Herbert Xu <herbert@...dor.apana.org.au>
> Home Page: http://gondor.apana.org.au/~herbert/
> PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Powered by blists - more mailing lists