netdev - Re: kTLS broken somewhere between 4.18 and 5.0

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Date:   Fri, 31 May 2019 00:41:35 +0200
From:   Andre Tomt <andre@...t.net>
To:     John Fastabend <john.fastabend@...il.com>,
        "Steinar H. Gunderson" <steinar+kernel@...derson.no>,
        netdev@...r.kernel.org, Daniel Borkmann <daniel@...earbox.net>
Subject: Re: kTLS broken somewhere between 4.18 and 5.0

On 07.05.2019 16:45, John Fastabend wrote:
> Andre Tomt wrote:
>> On 14.04.2019 22:40, John Fastabend wrote:
>>> On 4/13/19 6:56 PM, Andre Tomt wrote:
>>>> On 13.04.2019 17:34, Steinar H. Gunderson wrote:
>>>>> Hi,
>>>>>
>>>>> I've been using kTLS for a while, with my video reflector Cubemap
>>>>> (https://git.sesse.net/?p=cubemap). After I upgraded my server from
>>>>> 4.18.11 to 5.0.6, seemingly I've started seeing corruption. The data sent
>>>>> with send() (HTTP headers, HLS playlists) appears to be fine, but sendfile()
>>>>> (actual video data, from a file on tmpfs) is not; after ~20 kB of data
>>>>> (19626 in one test here), the data appears to be randomly corrupted. Diffing
>>>>> non-TLS (good) and TLS (bad) video data:
> 
> [...]
> 
>> Hi John
>>
>> Have you had any luck tracking this down?
>>
>> Just gave net.git a spin and it is still serving up corrupted data when
>> ktls is active and using sendfile.  FWIW I only tested without ktls
>> offload capable hardware (ie in software mode) and no bpf. Same sendfile
>> usage on a non-ktls socket works fine.
> 
> Hi Andre, I should have a series to address this in the next few days. I
> still need to resolve a couple corner cases. Hopefully, by next week we
> can get bpf tree working for this case.

current linus master, net.git master and bpf.git master are all still 
not working right, so I took a closer look.

It seems to only happen if sendfile writes more than a maximum tls 
record size worth of data. If I clamp the sendfile calls to 16384 bytes 
at a time everything works fine.

Not sure if sendfile triggers record splitting, but could that be what 
is broken? The bisected commit does touch that part quite extensively.

I made a fest input that just repeats 0-9 a-z 10 times for each 
character in a loop and got the following corruption post-decryption:

> 00004f74: 6969 6969 6969 6969 6969  iiiiiiiiii
> 00004f7e: 6a6a 6a6a 6a6a 6a6a 6a6a  jjjjjjjjjj
> 00004f88: 6b6b 6b6b 6b6b 6b6b 6b6b  kkkkkkkkkk
> 00004f92: 6c6c 6c6c 6c6c 6c6c 6c6c  llllllllll
> 00004f9c: 6d6d 6d6d 6d6d 6d6d 6d6d  mmmmmmmmmm
> 00004fa6: 6e6e 6e6e 6e6e 6e6e 6e6e  nnnnnnnnnn
> 00004fb0: 6f6f 6f6f 6f6f 6f6f 6f6f  oooooooooo
> 00004fba: 7070 7070 7070 7070 7070  pppppppppp
> 00004fc4: 7171 7171 7171 7171 7171  qqqqqqqqqq
> 00004fce: 7272 6d6d 6d6d 6d6d 6e6e  rrmmmmmmnn <- uh oh, goes backwards?
> 00004fd8: 6e6e 6e6e 6e6e 6e6e 6f6f  nnnnnnnnoo
> 00004fe2: 6f6f 6f6f 6f6f 6f6f 7070  oooooooopp
> 00004fec: 7070 7070 7070 7070 7171  ppppppppqq
> 00004ff6: 7171 7171 7171 7171 7272  qqqqqqqqrr
> 00005000: 7272 7272 7272 7272 7373  rrrrrrrrss
> 0000500a: 7373 7373 7373 7373 7474  sssssssstt
> 00005014: 7474 7474 7474 7474 7575  ttttttttuu
> 0000501e: 7575 7575 7575 7575 7676  uuuuuuuuvv
> 00005028: 7676 7676 7676 7676 7777  vvvvvvvvww