linux-kernel - Re: [PATCH][V3] nbd: add multi-connection support

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <4b5da874-dd62-b683-563d-717d710156e4@grimberg.me>
Date:   Tue, 11 Oct 2016 12:00:18 +0300
From:   Sagi Grimberg <sagi@...mberg.me>
To:     Josef Bacik <jbacik@...com>, axboe@...com,
        linux-block@...r.kernel.org, linux-kernel@...r.kernel.org,
        kernel-team@...com, nbd-general@...ts.sourceforge.net, w@...r.be
Subject: Re: [PATCH][V3] nbd: add multi-connection support


> NBD can become contended on its single connection.  We have to serialize all
> writes and we can only process one read response at a time.  Fix this by
> allowing userspace to provide multiple connections to a single nbd device.  This
> coupled with block-mq drastically increases performance in multi-process cases.
> Thanks,

Hey Josef,

I gave this patch a tryout and I'm getting a kernel paging request when
running multi-threaded write workload [1].

I have 2 VMs on my laptop: each is assigned with 2 cpus. I connected
the client to the server via 2 connections and ran:
fio --group_reporting --rw=randwrite --bs=4k --numjobs=2 --iodepth=128 
--runtime=60 --time_based --loops=1 --ioengine=libaio --direct=1 
--invalidate=1 --randrepeat=1 --norandommap --exitall --name task_nbd0 
--filename=/dev/nbd0

The server backend is null_blk btw:
./nbd-server 1022 /dev/nullb0

nbd-client:
./nbd-client -C 2 192.168.100.3 1022 /dev/nbd0

[1]:
[  171.813649] BUG: unable to handle kernel paging request at 
0000000235363130
[  171.816015] IP: [<ffffffffc0645e39>] nbd_queue_rq+0x319/0x580 [nbd]
[  171.816015] PGD 7a080067 PUD 0
[  171.816015] Oops: 0000 [#1] SMP
[  171.816015] Modules linked in: nbd(O) rpcsec_gss_krb5 nfsv4 ib_iser 
iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi 
snd_hda_codec_generic ppdev kvm_intel cirrus snd_hda_intel ttm kvm 
irqbypass drm_kms_helper snd_hda_codec drm snd_hda_core snd_hwdep joydev 
input_leds fb_sys_fops snd_pcm serio_raw syscopyarea snd_timer 
sysfillrect snd sysimgblt soundcore i2c_piix4 nfsd ib_umad parport_pc 
auth_rpcgss nfs_acl rdma_ucm nfs rdma_cm iw_cm lockd grace ib_cm 
configfs sunrpc ib_uverbs mac_hid fscache ib_core lp parport psmouse 
floppy e1000 pata_acpi [last unloaded: nbd]
[  171.816015] CPU: 0 PID: 196 Comm: kworker/0:1H Tainted: G           O 
    4.8.0-rc4+ #61
[  171.816015] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), 
BIOS Bochs 01/01/2011
[  171.816015] Workqueue: kblockd blk_mq_run_work_fn
[  171.816015] task: ffff8f0b37b23280 task.stack: ffff8f0b37bf0000
[  171.816015] RIP: 0010:[<ffffffffc0645e39>]  [<ffffffffc0645e39>] 
nbd_queue_rq+0x319/0x580 [nbd]
[  171.816015] RSP: 0018:ffff8f0b37bf3c20  EFLAGS: 00010206
[  171.816015] RAX: 0000000235363130 RBX: 0000000000000000 RCX: 
0000000000000200
[  171.816015] RDX: 0000000000000200 RSI: ffff8f0b37b23b48 RDI: 
ffff8f0b37b23280
[  171.816015] RBP: ffff8f0b37bf3cc8 R08: 0000000000000001 R09: 
0000000000000000
[  171.816015] R10: 0000000000000000 R11: ffff8f0b37f21000 R12: 
0000000023536303
[  171.816015] R13: 0000000000000000 R14: 0000000023536313 R15: 
ffff8f0b37f21000
[  171.816015] FS:  0000000000000000(0000) GS:ffff8f0b3d200000(0000) 
knlGS:0000000000000000
[  171.816015] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  171.816015] CR2: 0000000235363130 CR3: 00000000789b7000 CR4: 
00000000000006f0
[  171.816015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[  171.816015] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 
0000000000000400
[  171.816015] Stack:
[  171.816015]  ffff8f0b00000000 ffff8f0b37a79480 ffff8f0b378513c8 
0000000000000282
[  171.816015]  ffff8f0b37b28428 ffff8f0b37a795f0 ffff8f0b37f21500 
00000a0023536313
[  171.816015]  ffffea0001c69080 0000000000000000 ffff8f0b37b28280 
1395602537b23280
[  171.816015] Call Trace:
[  171.816015]  [<ffffffffb8426840>] __blk_mq_run_hw_queue+0x260/0x390
[  171.816015]  [<ffffffffb84269b2>] blk_mq_run_work_fn+0x12/0x20
[  171.816015]  [<ffffffffb80aae21>] process_one_work+0x1f1/0x6b0
[  171.816015]  [<ffffffffb80aada2>] ? process_one_work+0x172/0x6b0
[  171.816015]  [<ffffffffb80ab32e>] worker_thread+0x4e/0x490
[  171.816015]  [<ffffffffb80ab2e0>] ? process_one_work+0x6b0/0x6b0
[  171.816015]  [<ffffffffb80ab2e0>] ? process_one_work+0x6b0/0x6b0
[  171.816015]  [<ffffffffb80b1f41>] kthread+0x101/0x120
[  171.816015]  [<ffffffffb88d4ecf>] ret_from_fork+0x1f/0x40
[  171.816015]  [<ffffffffb80b1e40>] ? kthread_create_on_node+0x250/0x250