netdev - Re: kTLS broken somewhere between 4.18 and 5.0

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <a69bb5f9-45d2-354e-0422-e3d37d4e427d@tomt.net>
Date:   Thu, 2 May 2019 16:15:18 +0200
From:   Andre Tomt <andre@...t.net>
To:     John Fastabend <john.fastabend@...il.com>,
        "Steinar H. Gunderson" <steinar+kernel@...derson.no>,
        netdev@...r.kernel.org, Daniel Borkmann <daniel@...earbox.net>
Subject: Re: kTLS broken somewhere between 4.18 and 5.0

On 14.04.2019 22:40, John Fastabend wrote:
> On 4/13/19 6:56 PM, Andre Tomt wrote:
>> On 13.04.2019 17:34, Steinar H. Gunderson wrote:
>>> Hi,
>>>
>>> I've been using kTLS for a while, with my video reflector Cubemap
>>> (https://git.sesse.net/?p=cubemap). After I upgraded my server from
>>> 4.18.11 to 5.0.6, seemingly I've started seeing corruption. The data sent
>>> with send() (HTTP headers, HLS playlists) appears to be fine, but sendfile()
>>> (actual video data, from a file on tmpfs) is not; after ~20 kB of data
>>> (19626 in one test here), the data appears to be randomly corrupted. Diffing
>>> non-TLS (good) and TLS (bad) video data:
>>>
>>>     00004c70: fa 70 c5 71 b5 f5 b7 ac 74 b0 ca 80 02 4c 06 3f  .p.q....t....L.?
>>>     00004c80: 5c 5b 0c b3 e0 a0 c3 21 93 d3 6e 65 36 70 0a 27  \[.....!..ne6p.'
>>>     00004c90: 84 67 16 2c 95 c0 55 e1 04 76 52 10 50 5d 00 26  .g.,..U..vR.P].&
>>>
>>>    -00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 18 c0 06 20 02 54  ...p~...^~... .T
>>>    +00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 0a 60 9f 1f 97 f2  ...p~...^~.`....
>>>
>>>    -00004cb0: 1e 4c c1 71 7d 0b 91 28 23 98 09 ae c4 95 ae 7f  .L.q}..(#.......
>>>    +00004cb0: 6e 17 50 03 67 fa 2f 83 b0 88 eb fc 54 f2 0b 00  n.P.g./.....T...
>>>
>>>    -00004cc0: a2 92 20 b8 f2 b6 72 2a e8 7e d7 27 99 65 56 70  .. ...r*.~.'.eVp
>>>    +00004cc0: 6c 9e a1 02 b4 30 11 25 d7 58 b0 0c c0 6c e1 bd  l....0.%.X...l..
>>>
>>> It never appears to get back into sync after that. Interestingly, it is
>>> _consistently_ wrong; if I download the same fragment multiple times, it
>>> breaks at the same place and gives the same garbage (but different fragments
>>> give different divergence points). Tested with both wget and Chrome.
>>> Does anyone know what could be wrong?
>>>
>>> (It is, unfortunately, not easy for me to reboot this server at will, so a
>>> bisect could be hard.)
>>>
>>> Please Cc me on any replies, I'm not subscribed to netdev.
>>>
>>> /* Steinar */
>>>
>>
>>
>> Reproduced and bisected, the problem showed up in v4.20-rc1. Unfortunately the commit seems to have some significant dependencies so I was unable to verify by reverting it on 4.20.
>>
> 
> 
> Hi thanks I'll take a look this evening or first thing tomorrow. I have
> a couple other fixes queued up for ktls as well so I'll see if we can get
> a fix for this included in that series.
> 
> Thanks!
> John

Hi John

Have you had any luck tracking this down?

Just gave net.git a spin and it is still serving up corrupted data when 
ktls is active and using sendfile.  FWIW I only tested without ktls 
offload capable hardware (ie in software mode) and no bpf. Same sendfile 
usage on a non-ktls socket works fine.