Date:   Sun, 22 Apr 2018 23:21:24 +0200
From:   Andre Tomt <andre@...t.net>
To:     netdev <netdev@...r.kernel.org>
Cc:     davejwatson@...com, Tariq Toukan <tariqt@...lanox.com>
Subject: kTLS in combination with mlx4 is very unstable

Hello!

kTLS looks fun, so I decided to play with it. It is quite spiffy - 
however with mlx4 I get kernel crashes I'm not seeing when testing on ixgbe.

For testing I'm using a git build of the "stream reflector" cubemap[1] 
configured with kTLS and 8 worker threads running on 4 physical cores, 
loading it up with a ~13Mbps MPEG-TS stream pulled from satelite TV.

The kernel seems to get increasingly unstable as I load it up with 
client connections. At about 9Gbps and 700 connections, it is okay at 
least for a while - it might run fine for say 45 minutes. Once it gets 
to 20 - 30Gbps, the kernel will usually start spewing OOPSes within 
minutes and the traffic drops.

Some bad interaction between mlx4 and kTLS?

Hardware is a quad core Xeon-D 1520 using a dual port Mellanox 
ConnectX-3 VPI with a single 40Gbps ethernet link configured. Mellanox 
mlx4 driver settings are kernel.org upstream defaults. Interface is 
configured with FQ qdisc and sockets are using BBR congestion control.

Tested on kernel 4.14.34, 4.15.17, and 4.16.2 - 4.16.3.

[1] https://git.sesse.net/?p=cubemap

First OOPS (from 4.16.3):
> [  660.467358] BUG: stack guard page was hit at 00000000b136e403 (stack is 00000000ded3f179..00000000835ee6c5)
> [  660.467422] kernel stack overflow (double-fault): 0000 [#1] SMP PTI
> [  660.467457] Modules linked in: coretemp intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp iTCO_wdt gpio_ich iTCO_vendor_support kvm_intel mxm_wmi xfs libcrc32c kvm crc32c_generic irqbypass nls_iso8859_1 crct10dif_pclmul crc32_pclmul nls_cp437 ghash_clmulni_intel vfat fat aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_pch_thermal mei_me sg mei lpc_ich mfd_core evdev ipmi_si ipmi_devintf ipmi_msghandler wmi acpi_pad tls ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid hid mlx4_ib mlx4_en ib_core sd_mod ast i2c_algo_bit ttm drm_kms_helper syscopyarea sysfillrect ehci_pci xhci_pci sysimgblt fb_sys_fops ahci libahci xhci_hcd ehci_hcd libata crc32c_intel nvme drm usbcore scsi_mod mlx4_core ixgbe i2c_core mdio usb_common devlink hwmon nvme_core rtc_cmos
> [  660.467856] CPU: 4 PID: 660 Comm: cubemap Not tainted 4.16.0-1 #1
> [  660.467890] Hardware name: Supermicro Super Server/X10SDV-4C-TLN2F, BIOS 1.2c 09/19/2017
> [  660.467939] RIP: 0010:__kmalloc+0x7/0x1f0
> [  660.467962] RSP: 0018:ffffabafc27b8000 EFLAGS: 00010206
> [  660.467992] RAX: 000000000000000d RBX: 0000000000000010 RCX: ffffabafc27b8070
> [  660.468030] RDX: ffff98a0d0235490 RSI: 0000000001080020 RDI: 000000000000001d
> [  660.468069] RBP: 000000000000000d R08: ffff98a0d5be4860 R09: ffff98a0ec299180
> [  660.468106] R10: ffffabafc27b80b8 R11: 0000000000000010 R12: 0000000000000010
> [  660.468145] R13: ffff98a0ec299180 R14: ffff98a0ec299180 R15: 0000000000000000
> [  660.468184] FS:  00007f8a35ffb700(0000) GS:ffff98a17fd00000(0000) knlGS:0000000000000000
> [  660.468227] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  660.468258] CR2: ffffabafc27b7ff8 CR3: 00000004698ee001 CR4: 00000000003606e0
> [  660.468297] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [  660.468334] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [  660.468373] Call Trace:
> [  660.468401]  gcmaes_encrypt.constprop.5+0x137/0x240 [aesni_intel]
> [  660.468439]  ? generic_gcmaes_encrypt+0x5f/0x80 [aesni_intel]
> [  660.468476]  ? gcmaes_wrapper_encrypt+0x36/0x80 [aesni_intel]
> [  660.468511]  ? tls_push_record+0x1d3/0x390 [tls]
> [  660.468537]  ? tls_push_record+0x1d3/0x390 [tls]
> [  660.468565]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.468593]  ? do_tcp_sendpages+0x8d/0x580
> [  660.468618]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.468643]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.468671]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.468697]  ? do_tcp_sendpages+0x8d/0x580
> [  660.468722]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.468748]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.468776]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.468802]  ? do_tcp_sendpages+0x8d/0x580
> [  660.468826]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.468852]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.468880]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.468906]  ? do_tcp_sendpages+0x8d/0x580
> [  660.468931]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.468957]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.470165]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.471363]  ? do_tcp_sendpages+0x8d/0x580
> [  660.472555]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.473713]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.474838]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.475927]  ? do_tcp_sendpages+0x8d/0x580
> [  660.476977]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.477999]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.478968]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.479902]  ? do_tcp_sendpages+0x8d/0x580
> [  660.480790]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.481644]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.482483]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.483301]  ? do_tcp_sendpages+0x8d/0x580
> [  660.484099]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.484891]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.485674]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.486455]  ? do_tcp_sendpages+0x8d/0x580
> [  660.487220]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.487890]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.488328]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.488748]  ? do_tcp_sendpages+0x8d/0x580
> [  660.489167]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.489565]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.489970]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.490370]  ? do_tcp_sendpages+0x8d/0x580
> [  660.490771]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.491165]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.491550]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.491914]  ? do_tcp_sendpages+0x8d/0x580
> [  660.492274]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.492641]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.493008]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.493374]  ? do_tcp_sendpages+0x8d/0x580
> [  660.493787]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.494177]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.494585]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.494972]  ? do_tcp_sendpages+0x8d/0x580
> [  660.495359]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.495742]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.496128]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.496512]  ? do_tcp_sendpages+0x8d/0x580
> [  660.496901]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.497301]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.497697]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.498096]  ? do_tcp_sendpages+0x8d/0x580
> [  660.498490]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.498884]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.499291]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.499700]  ? do_tcp_sendpages+0x8d/0x580
> [  660.500103]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.500511]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.500909]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.501326]  ? do_tcp_sendpages+0x8d/0x580
> [  660.501737]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.502131]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.502525]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.502928]  ? do_tcp_sendpages+0x8d/0x580
> [  660.503331]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.503724]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.504127]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.504547]  ? do_tcp_sendpages+0x8d/0x580
> [  660.504949]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.505348]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.505769]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.506207]  ? do_tcp_sendpages+0x8d/0x580
> [  660.506622]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.507030]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.507435]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.507841]  ? do_tcp_sendpages+0x8d/0x580
> [  660.508518]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.509261]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.510011]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.510187] BUG: stack guard page was hit at 00000000e0315e51 (stack is 00000000bea6f919..0000000005fc5eb4)
> [  660.510473] BUG: stack guard page was hit at 000000004b958a15 (stack is 000000001f2af2d1..000000006295a4b1)
> [  660.510758]  ? do_tcp_sendpages+0x8d/0x580
> [  660.513094]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.513886]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.514680]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.515487]  ? do_tcp_sendpages+0x8d/0x580
> [  660.515750] BUG: stack guard page was hit at 00000000bc93cf0d (stack is 0000000031a15c9c..0000000029a82776)
> [  660.516295]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.518017]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.518883]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.519752]  ? do_tcp_sendpages+0x8d/0x580
> [  660.519816] BUG: stack guard page was hit at 000000002d1db286 (stack is 00000000b5bb06d4..000000007a29c8f2)
> [  660.520544]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.522315]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.523162]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.524006]  ? do_tcp_sendpages+0x8d/0x580
> [  660.524849]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.525695]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.526545]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.527399]  ? do_tcp_sendpages+0x8d/0x580
> [  660.528247]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.529099]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.529955]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.530797]  ? do_tcp_sendpages+0x8d/0x580
> [  660.531643]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.532010] BUG: stack guard page was hit at 0000000027abda92 (stack is 00000000aadcb221..00000000a587b67b)
> [  660.532535]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.534511]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.535506]  ? do_tcp_sendpages+0x8d/0x580
> [  660.536500]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.537495]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.538493]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.539482]  ? do_tcp_sendpages+0x8d/0x580
> [  660.540462]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.541447]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.542430]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.543411]  ? do_tcp_sendpages+0x8d/0x580
> [  660.544395]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.545382]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.546365]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.547347]  ? do_tcp_sendpages+0x8d/0x580
> [  660.548334]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.549318]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.550300]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.551284]  ? do_tcp_sendpages+0x8d/0x580
> [  660.552267]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.553250]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.554205]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.555158]  ? do_tcp_sendpages+0x8d/0x580
> [  660.556083]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.557009]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.557936]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.558862]  ? do_tcp_sendpages+0x8d/0x580
> [  660.559786]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.560681]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.561547]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.562413]  ? do_tcp_sendpages+0x8d/0x580
> [  660.563279]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.564143]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.564979]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.565783]  ? do_tcp_sendpages+0x8d/0x580
> [  660.566587]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.567392]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.568197]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.569000]  ? do_tcp_sendpages+0x8d/0x580
> [  660.569804]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.570609]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.571415]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.572218]  ? do_tcp_sendpages+0x8d/0x580
> [  660.573023]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.573830]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.574634]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.575437]  ? do_tcp_sendpages+0x8d/0x580
> [  660.576210]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.576953]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.577698]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.578441]  ? do_tcp_sendpages+0x8d/0x580
> [  660.579183]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.579929]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.580673]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.581417]  ? do_tcp_sendpages+0x8d/0x580
> [  660.582159]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.582904]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.583649]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.584394]  ? do_tcp_sendpages+0x8d/0x580
> [  660.585137]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.585882]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.586628]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.587372]  ? do_tcp_sendpages+0x8d/0x580
> [  660.588115]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.588861]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.589605]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.590350]  ? do_tcp_sendpages+0x8d/0x580
> [  660.591093]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.591840]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.592585]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.593328]  ? do_tcp_sendpages+0x8d/0x580
> [  660.594072]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.594816]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.595563]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.596308]  ? do_tcp_sendpages+0x8d/0x580
> [  660.597050]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.597794]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.598540]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.599281]  ? do_tcp_sendpages+0x8d/0x580
> [  660.600025]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.600772]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.601517]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.602260]  ? do_tcp_sendpages+0x8d/0x580
> [  660.603003]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.603750]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.604495]  ? tls_write_space+0x6a/0x80 [tls]
> [  660.605238]  ? do_tcp_sendpages+0x8d/0x580
> [  660.605981]  ? tls_push_sg+0x74/0x130 [tls]
> [  660.606726]  ? tls_push_record+0x24a/0x390 [tls]
> [  660.607474]  ? tls_sw_sendpage+0x14a/0x390 [tls]
> [  660.608214]  ? direct_splice_actor+0x40/0x40
> [  660.608951]  ? inet_sendpage+0x40/0xf0
> [  660.609689]  ? kernel_sendpage+0x1a/0x30
> [  660.610426]  ? sock_sendpage+0x20/0x30
> [  660.611161]  ? pipe_to_sendpage+0x5f/0x70
> [  660.611898]  ? __splice_from_pipe+0x80/0x180
> [  660.612637]  ? generic_file_splice_read+0x100/0x150
> [  660.613382]  ? direct_splice_actor+0x40/0x40
> [  660.614128]  ? splice_from_pipe+0x4f/0x70
> [  660.614871]  ? direct_splice_actor+0x35/0x40
> [  660.615619]  ? splice_direct_to_actor+0xce/0x1d0
> [  660.616368]  ? generic_pipe_buf_nosteal+0x10/0x10
> [  660.617122]  ? do_splice_direct+0x8c/0xa0
> [  660.617876]  ? do_sendfile+0x19d/0x380
> [  660.618626]  ? SyS_sendfile64+0x4c/0x90
> [  660.619376]  ? do_syscall_64+0x7a/0x390
> [  660.620121]  ? do_page_fault+0x31/0x130
> [  660.620863]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
> [  660.621618] Code: 24 08 4c 89 e9 48 89 de e8 d7 66 63 00 4d 8b 17 5a 4d 85 d2 75 d7 e9 e9 fe ff ff 48 89 c5 e9 e1 fe ff ff 90 0f 1f 44 00 00 41 57 <41> 56 41 55 41 54 55 53 48 81 ff 00 20 00 00 0f 87 a4 01 00 00 
> [  660.623316] RIP: __kmalloc+0x7/0x1f0 RSP: ffffabafc27b8000
> [  660.624168] ---[ end trace 7f6206177c0cc58f ]---

Powered by blists - more mailing lists