Message-ID: <4cdfca7f-bee0-56f5-e512-8ad2e4e6dfcf@tomt.net>
Date:   Sun, 14 Apr 2019 03:56:28 +0200
From:   Andre Tomt <andre@...t.net>
To:     "Steinar H. Gunderson" <steinar+kernel@...derson.no>,
        netdev@...r.kernel.org, John Fastabend <john.fastabend@...il.com>,
        Daniel Borkmann <daniel@...earbox.net>
Subject: Re: kTLS broken somewhere between 4.18 and 5.0

On 13.04.2019 17:34, Steinar H. Gunderson wrote:
> Hi,
> 
> I've been using kTLS for a while, with my video reflector Cubemap
> (https://git.sesse.net/?p=cubemap). After I upgraded my server from
> 4.18.11 to 5.0.6, seemingly I've started seeing corruption. The data sent
> with send() (HTTP headers, HLS playlists) appears to be fine, but sendfile()
> (actual video data, from a file on tmpfs) is not; after ~20 kB of data
> (19626 in one test here), the data appears to be randomly corrupted. Diffing
> non-TLS (good) and TLS (bad) video data:
> 
>    00004c70: fa 70 c5 71 b5 f5 b7 ac 74 b0 ca 80 02 4c 06 3f  .p.q....t....L.?
>    00004c80: 5c 5b 0c b3 e0 a0 c3 21 93 d3 6e 65 36 70 0a 27  \[.....!..ne6p.'
>    00004c90: 84 67 16 2c 95 c0 55 e1 04 76 52 10 50 5d 00 26  .g.,..U..vR.P].&
> 
>   -00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 18 c0 06 20 02 54  ...p~...^~... .T
>   +00004ca0: 0c b8 84 70 7e ed 12 8f 5e 7e 0a 60 9f 1f 97 f2  ...p~...^~.`....
> 
>   -00004cb0: 1e 4c c1 71 7d 0b 91 28 23 98 09 ae c4 95 ae 7f  .L.q}..(#.......
>   +00004cb0: 6e 17 50 03 67 fa 2f 83 b0 88 eb fc 54 f2 0b 00  n.P.g./.....T...
> 
>   -00004cc0: a2 92 20 b8 f2 b6 72 2a e8 7e d7 27 99 65 56 70  .. ...r*.~.'.eVp
>   +00004cc0: 6c 9e a1 02 b4 30 11 25 d7 58 b0 0c c0 6c e1 bd  l....0.%.X...l..
> 
> It never appears to get back into sync after that. Interestingly, it is
> _consistently_ wrong; if I download the same fragment multiple times, it
> breaks at the same place and gives the same garbage (but different fragments
> give different divergence points). Tested with both wget and Chrome.
> Does anyone know what could be wrong?
> 
> (It is, unfortunately, not easy for me to reboot this server at will, so a
> bisect could be hard.)
> 
> Please Cc me on any replies, I'm not subscribed to netdev.
> 
> /* Steinar */
> 


Reproduced and bisected, the problem showed up in v4.20-rc1. 
Unfortunately the commit seems to have some significant dependencies so 
I was unable to verify by reverting it on 4.20.

Adding John and Daniel.


d3b18ad31f93d0b6bae105c679018a1ba7daa9ca is the first bad commit
commit d3b18ad31f93d0b6bae105c679018a1ba7daa9ca
Author: John Fastabend <john.fastabend@...il.com>
Date:   Sat Oct 13 02:46:01 2018 +0200

     tls: add bpf support to sk_msg handling

     This work adds BPF sk_msg verdict program support to kTLS
     allowing BPF and kTLS to be combined together. Previously kTLS
     and sk_msg verdict programs were mutually exclusive in the
     ULP layer which created challenges for the orchestrator when
     trying to apply TCP based policy, for example. To resolve this,
     leveraging the work from previous patches that consolidates
     the use of sk_msg, we can finally enable BPF sk_msg verdict
     programs so they continue to run after the kTLS socket is
     created. No change in behavior when kTLS is not used in
     combination with BPF, the kselftest suite for kTLS also runs
     successfully.

     Joint work with Daniel.

     Signed-off-by: John Fastabend <john.fastabend@...il.com>
     Signed-off-by: Daniel Borkmann <daniel@...earbox.net>
     Signed-off-by: Alexei Starovoitov <ast@...nel.org>

:040000 040000 107a1f08fa7d54610292047bbd360a6bf9fff78a 
1ef18a0495f094c6d771a371f6c05f849daff512 M	include
:040000 040000 56e6ebd0c6dc0a5aa8d371332cae6bff6cdcc1ff 
44ddc12947dce93b449e468ed6862d475c33f32b M	net

$ git bisect log
git bisect start '--' 'net' 'include/net'
# good: [84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d] Linux 4.19
git bisect good 84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d
# bad: [8fe28cb58bcb235034b64cbbb7550a8a43fd88be] Linux 4.20
git bisect bad 8fe28cb58bcb235034b64cbbb7550a8a43fd88be
# bad: [aa563d7bca6e882ec2bdae24603c8f016401a144] iov_iter: Separate 
type from direction and use accessor functions
git bisect bad aa563d7bca6e882ec2bdae24603c8f016401a144
# good: [85dd3da43dd59b9220d9cba4f933a3dc0ea6faa5] cfg80211: combine 
wdev/netdev unregister code
git bisect good 85dd3da43dd59b9220d9cba4f933a3dc0ea6faa5
# good: [9000a457a0c84883874a844ef94adf26f633f3b4] Merge 
git://git.kernel.org/pub/scm/linux/kernel/git/pablo/nf-next
git bisect good 9000a457a0c84883874a844ef94adf26f633f3b4
# good: [1a3aea2534f4f3083f29b2b047aa83a9d6c777a4] net: bridge: fix a 
possible memory leak in __vlan_add
git bisect good 1a3aea2534f4f3083f29b2b047aa83a9d6c777a4
# bad: [cb167893f41e21e6bd283d78e53489289dc0592d] net: Plumb support for 
filtering ipv4 and ipv6 multicast route dumps
git bisect bad cb167893f41e21e6bd283d78e53489289dc0592d
# bad: [5ef0ae84f02a4dbe0e09f89c6481ac13649cb19b] bpf: Fix IPv6 dport 
byte-order in bpf_sk_lookup
git bisect bad 5ef0ae84f02a4dbe0e09f89c6481ac13649cb19b
# good: [604326b41a6fb9b4a78b6179335decee0365cd8c] bpf, sockmap: convert 
to generic sk_msg interface
git bisect good 604326b41a6fb9b4a78b6179335decee0365cd8c
# good: [924ad65ed01ee0eec5d2a3280c01c394343d6df7] tls: replace poll 
implementation with read hook
git bisect good 924ad65ed01ee0eec5d2a3280c01c394343d6df7
# bad: [8a615c6b0352a9ec56151b6c95d68e0a2eef5cf0] bpf: Allow sk_lookup 
with IPv6 module
git bisect bad 8a615c6b0352a9ec56151b6c95d68e0a2eef5cf0
# bad: [d3b18ad31f93d0b6bae105c679018a1ba7daa9ca] tls: add bpf support 
to sk_msg handling
git bisect bad d3b18ad31f93d0b6bae105c679018a1ba7daa9ca
# first bad commit: [d3b18ad31f93d0b6bae105c679018a1ba7daa9ca] tls: add 
bpf support to sk_msg handling

Powered by blists - more mailing lists