Message-ID: <0611984e-aea2-7eb5-af3e-e0635ca3b7ba@iogearbox.net>
Date: Mon, 16 Oct 2023 15:05:25 +0200
From: Daniel Borkmann <daniel@...earbox.net>
To: Kuniyuki Iwashima <kuniyu@...zon.com>,
 "David S. Miller" <davem@...emloft.net>, Eric Dumazet <edumazet@...gle.com>,
 Jakub Kicinski <kuba@...nel.org>, Paolo Abeni <pabeni@...hat.com>,
 David Ahern <dsahern@...nel.org>, Alexei Starovoitov <ast@...nel.org>,
 Andrii Nakryiko <andrii@...nel.org>, Martin KaFai Lau
 <martin.lau@...ux.dev>, Song Liu <song@...nel.org>,
 Yonghong Song <yonghong.song@...ux.dev>,
 John Fastabend <john.fastabend@...il.com>, KP Singh <kpsingh@...nel.org>,
 Stanislav Fomichev <sdf@...gle.com>, Hao Luo <haoluo@...gle.com>,
 Jiri Olsa <jolsa@...nel.org>, Mykola Lysenko <mykolal@...com>
Cc: Kuniyuki Iwashima <kuni1840@...il.com>, bpf@...r.kernel.org,
 netdev@...r.kernel.org
Subject: Re: [PATCH v1 bpf-next 00/11] bpf: tcp: Add SYN Cookie
 generation/validation SOCK_OPS hooks.

On 10/14/23 12:04 AM, Kuniyuki Iwashima wrote:
> Under SYN Flood, the TCP stack generates SYN Cookie to remain stateless
> for the connection request until a valid ACK is responded to the SYN+ACK.
> 
> The cookie contains two kinds of host-specific bits, a timestamp and
> secrets, so only can it be validated by the generator.  It means SYN
> Cookie consumes network resources between the client and the server;
> intermediate nodes must remember which nodes to route ACK for the cookie.
> 
> SYN Proxy reduces such unwanted resource allocation by handling 3WHS at
> the edge network.  After SYN Proxy completes 3WHS, it forwards SYN to the
> backend server and completes another 3WHS.  However, since the server's
> ISN differs from the cookie, the proxy must manage the ISN mappings and
> fix up SEQ/ACK numbers in every packet for each connection.  If a proxy
> node is down, all the connections through it are also down.  Keeping a
> state at proxy is painful from that perspective.
> 
> At AWS, we use a dirty hack to build truly stateless SYN Proxy at scale.
> Our SYN Proxy consists of the front proxy layer and the backend kernel
> module.  (See slides of netconf [0], p6 - p15)
> 
> The cookie that SYN Proxy generates differs from the kernel's cookie in
> that it contains a secret (called rolling salt) (i) shared by all the proxy
> nodes so that any node can validate ACK and (ii) updated periodically so
> that old cookies cannot be validated.  Also, ISN contains WScale, SACK, and
> ECN, not in TS val.  This is not to sacrifice any connection quality, where
> some customers turn off the timestamp option due to retro CVE.
> 
> After 3WHS, the proxy restores SYN and forwards it and ACK to the backend
> server.  Our kernel module works at Netfilter input/output hooks and first
> feeds SYN to the TCP stack to initiate 3WHS.  When the module is triggered
> for SYN+ACK, it looks up the corresponding request socket and overwrites
> tcp_rsk(req)->snt_isn with the proxy's cookie.  Then, the module can
> complete 3WHS with the original ACK as is.
> 
> This way, our SYN Proxy does not manage the ISN mappings and can stay
> stateless.  It's working very well for high-bandwidth services like
> multiple Tbps, but we are looking for a way to drop the dirty hack and
> further optimise the sequences.
> 
> If we could validate an arbitrary SYN Cookie on the backend server with
> BPF, the proxy would need not restore SYN nor pass it.  After validating
> ACK, the proxy node just needs to forward it, and then the server can do
> the lightweight validation (e.g. check if ACK came from proxy nodes, etc)
> and create a connection from the ACK.
> 
> This series adds two SOCK_OPS hooks to generate and validate arbitrary
> SYN Cookie.  Each hook is invoked if BPF_SOCK_OPS_SYNCOOKIE_CB_FLAG is
> set to the listening socket in advance by bpf_sock_ops_cb_flags_set().
> 
> The user interface looks like this:
> 
>    BPF_SOCK_OPS_GEN_SYNCOOKIE_CB
> 
>      input
>      |- bpf_sock_ops.sk           : 4-tuple
>      |- bpf_sock_ops.skb          : TCP header
>      |- bpf_sock_ops.args[0]      : MSS
>      `- bpf_sock_ops.args[1]      : BPF_SYNCOOKIE_XXX flags
> 
>      output
>      |- bpf_sock_ops.replylong[0] : ISN (SYN Cookie) ------.
>      `- bpf_sock_ops.replylong[1] : TS value -----------.  |
>                                                         |  |
>    BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB                      |  |
>                                                         |  |
>      input                                              |  |
>      |- bpf_sock_ops.sk           : 4-tuple             |  |
>      |- bpf_sock_ops.skb          : TCP header          |  |
>      |- bpf_sock_ops.args[0]      : ISN (SYN Cookie) <-----'
>      `- bpf_sock_ops.args[1]      : TS value <----------'
> 
>      output
>      |- bpf_sock_ops.replylong[0] : MSS
>      `- bpf_sock_ops.replylong[1] : BPF_SYNCOOKIE_XXX flags
> 
> To establish a connection from SYN Cookie, BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB
> hook must set a valid MSS to bpf_sock_ops.replylong[0], meaning that
> BPF_SOCK_OPS_GEN_SYNCOOKIE_CB hook must encode MSS to ISN or TS val to be
> restored in the validation hook.
> 
> If WScale, SACK, and ECN are detected to be available in SYN packet, the
> corresponding flags are passed to args[0] of BPF_SOCK_OPS_GEN_SYNCOOKIE_CB
> so that bpf prog need not parse the TCP header.  The same flags can be set
> to replylong[0] of BPF_SOCK_OPS_CHECK_SYNCOOKIE_CB to enable each feature
> on the connection.
> 
> For details, please see each patch.  Here's an overview:
> 
>    patch 1 - 4 : Misc cleanup
>    patch 5, 6  : Add SOCK_OPS hook (only ISN is available here)
>    patch 7, 8  : Make TS val available as the second cookie storage
>    patch 9, 10 : Make WScale, SACK, and ECN configurable from ACK
>    patch 11    : selftest, need some help from BPF experts...
> 
> [0]: https://netdev.bots.linux.dev/netconf/2023/kuniyuki.pdf

Fyi, just as quick feedback, this fails BPF CI selftests :

https://github.com/kernel-patches/bpf/actions/runs/6513838231/job/17694669376

Notice: Success: 427/3396, Skipped: 24, Failed: 1
Error: #274 tcpbpf_user
   Error: #274 tcpbpf_user
   test_tcpbpf_user:PASS:open and load skel 0 nsec
   test_tcpbpf_user:PASS:test__join_cgroup(/tcpbpf-user-test) 0 nsec
   test_tcpbpf_user:PASS:attach_cgroup(bpf_testcb) 0 nsec
   run_test:PASS:start_server 0 nsec
   run_test:PASS:connect_to_fd(listen_fd) 0 nsec
   run_test:PASS:accept(listen_fd) 0 nsec
   run_test:PASS:send(cli_fd) 0 nsec
   run_test:PASS:recv(accept_fd) 0 nsec
   run_test:PASS:send(accept_fd) 0 nsec
   run_test:PASS:recv(cli_fd) 0 nsec
   run_test:PASS:recv(cli_fd) for fin 0 nsec
   run_test:PASS:recv(accept_fd) for fin 0 nsec
   verify_result:PASS:event_map 0 nsec
   verify_result:PASS:bytes_received 0 nsec
   verify_result:PASS:bytes_acked 0 nsec
   verify_result:PASS:data_segs_in 0 nsec
   verify_result:PASS:data_segs_out 0 nsec
   verify_result:FAIL:bad_cb_test_rv unexpected bad_cb_test_rv: actual 0 != expected 128
   verify_result:PASS:good_cb_test_rv 0 nsec
   verify_result:PASS:num_listen 0 nsec
   verify_result:PASS:num_close_events 0 nsec
   verify_result:PASS:tcp_save_syn 0 nsec
   verify_result:PASS:tcp_saved_syn 0 nsec
   verify_result:PASS:window_clamp_client 0 nsec
   verify_result:PASS:window_clamp_server 0 nsec

Powered by blists - more mailing lists