netdev - Re: [PATCH net v3] tcp: check skb is non-NULL in tcp_rto_delta

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <CADVnQy=xv_qy77nZK2wVjxdKjsiBa+k5B4LhgF4mSR-v1R200A@mail.gmail.com>
Date: Tue, 10 Sep 2024 15:11:37 -0400
From: Neal Cardwell <ncardwell@...gle.com>
To: Josh Hunt <johunt@...mai.com>
Cc: edumazet@...gle.com, davem@...emloft.net, kuba@...nel.org, 
	pabeni@...hat.com, netdev@...r.kernel.org, linux-kernel@...r.kernel.org
Subject: Re: [PATCH net v3] tcp: check skb is non-NULL in tcp_rto_delta_us()

On Tue, Sep 10, 2024 at 3:08 PM Josh Hunt <johunt@...mai.com> wrote:
>
> We have some machines running stock Ubuntu 20.04.6 which is their 5.4.0-174-generic
> kernel that are running ceph and recently hit a null ptr dereference in
> tcp_rearm_rto(). Initially hitting it from the TLP path, but then later we also
> saw it getting hit from the RACK case as well. Here are examples of the oops
> messages we saw in each of those cases:
...
> The NULL ptr deref is coming from tcp_rto_delta_us() attempting to pull an skb
> off the head of the retransmit queue and then dereferencing that skb to get the
> skb_mstamp_ns value via tcp_skb_timestamp_us(skb).
>
> The crash is the same one that was reported a # of years ago here:
> https://lore.kernel.org/netdev/86c0f836-9a7c-438b-d81a-839be45f1f58@gmail.com/T/#t
>
> and the kernel we're running has the fix which was added to resolve this issue.
>
> Unfortunately we've been unsuccessful so far in reproducing this problem in the
> lab and do not have the luxury of pushing out a new kernel to try and test if
> newer kernels resolve this issue at the moment. I realize this is a report
> against both an Ubuntu kernel and also an older 5.4 kernel. I have reported this
> issue to Ubuntu here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/2077657
> however I feel like since this issue has possibly cropped up again it makes
> sense to build in some protection in this path (even on the latest kernel
> versions) since the code in question just blindly assumes there's a valid skb
> without testing if it's NULL b/f it looks at the timestamp.
>
> Given we have seen crashes in this path before and now this case it seems like
> we should protect ourselves for when packets_out accounting is incorrect.
> While we should fix that root cause we should also just make sure the skb
> is not NULL before dereferencing it. Also add a warn once here to capture
> some information if/when the problem case is hit again.
>
> Fixes: e1a10ef7fa87 ("tcp: introduce tcp_rto_delta_us() helper for xmit timer fix")
> Signed-off-by: Josh Hunt <johunt@...mai.com>
> ---
>
> v3: Added fixes tag and more packet accounting info as requested by Neal Cardwell.
>     Also updated rto calcs to reflect original code.
> v2: Removed cover letter and added context from original cover letter to
>     commit msg.
>
>  include/net/tcp.h | 21 +++++++++++++++++++--
>  1 file changed, 19 insertions(+), 2 deletions(-)
>
> diff --git a/include/net/tcp.h b/include/net/tcp.h
> index 2aac11e7e1cc..196c148fce8a 100644
> --- a/include/net/tcp.h
> +++ b/include/net/tcp.h
> @@ -2434,9 +2434,26 @@ static inline s64 tcp_rto_delta_us(const struct sock *sk)
>  {
>         const struct sk_buff *skb = tcp_rtx_queue_head(sk);
>         u32 rto = inet_csk(sk)->icsk_rto;
> -       u64 rto_time_stamp_us = tcp_skb_timestamp_us(skb) + jiffies_to_usecs(rto);
>
> -       return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
> +       if (likely(skb)) {
> +               u64 rto_time_stamp_us = tcp_skb_timestamp_us(skb) + jiffies_to_usecs(rto);
> +
> +               return rto_time_stamp_us - tcp_sk(sk)->tcp_mstamp;
> +       } else {
> +               WARN_ONCE(1,
> +                       "rtx queue emtpy: "
> +                       "out:%u sacked:%u lost:%u retrans:%u "
> +                       "tlp_high_seq:%u sk_state:%u ca_state:%u "
> +                       "advmss:%u mss_cache:%u pmtu:%u\n",
> +                       tcp_sk(sk)->packets_out, tcp_sk(sk)->sacked_out,
> +                       tcp_sk(sk)->lost_out, tcp_sk(sk)->retrans_out,
> +                       tcp_sk(sk)->tlp_high_seq, sk->sk_state,
> +                       inet_csk(sk)->icsk_ca_state,
> +                       tcp_sk(sk)->advmss, tcp_sk(sk)->mss_cache,
> +                       inet_csk(sk)->icsk_pmtu_cookie);
> +               return jiffies_to_usecs(rto);
> +       }
> +
>  }

Looks good to me. Thanks, Josh!

Acked-by: Neal Cardwell <ncardwell@...gle.com>

neal