lists.openwall.net   lists  /  announce  owl-users  owl-dev  john-users  john-dev  passwdqc-users  yescrypt  popa3d-users  /  oss-security  kernel-hardening  musl  sabotage  tlsify  passwords  /  crypt-dev  xvendor  /  Bugtraq  Full-Disclosure  linux-kernel  linux-netdev  linux-ext4  linux-hardening  PHC 
Open Source and information security mailing list archives
 
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Date:   Wed, 30 Sep 2020 09:29:22 +0200
From:   Eric Dumazet <eric.dumazet@...il.com>
To:     Stephen Hemminger <stephen@...workplumber.org>,
        netdev@...r.kernel.org, Julian Anastasov <ja@....bg>,
        "lvs-devel@...r.kernel.org" <lvs-devel@...r.kernel.org>,
        abt-admin@...l.ru
Subject: Re: Fw: [Bug 209427] New: Incorrect timestamp cause packet to be
 dropped



On 9/29/20 7:35 PM, Stephen Hemminger wrote:
> 
> 
> Begin forwarded message:
> 
> Date: Tue, 29 Sep 2020 17:15:23 +0000
> From: bugzilla-daemon@...zilla.kernel.org
> To: stephen@...workplumber.org
> Subject: [Bug 209427] New: Incorrect timestamp cause packet to be dropped
> 
> 
> https://bugzilla.kernel.org/show_bug.cgi?id=209427
> 
>             Bug ID: 209427
>            Summary: Incorrect timestamp cause packet to be dropped
>            Product: Networking
>            Version: 2.5
>     Kernel Version: 5.8.10
>           Hardware: All
>                 OS: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>           Assignee: stephen@...workplumber.org
>           Reporter: abt-admin@...l.ru
>         Regression: No
> 
> After upgrading from my 3.10 to 5.8.10 I found out some of my packets are
> getting dropped by ipvlan interface (I'm using qdisk fq). Debugging session led
> me to the place where this happens
> 
> net/sched/sch_fq.c:464
> ...
>         if (fq_packet_beyond_horizon(skb, q)) {
>                 if (q->horizon_drop) {
>                         q->stat_horizon_drops++;
>                         return qdisc_drop(skb, sch, to_free);
>                 }
>                 q->stat_horizon_caps++;
>                 skb->tstamp = q->ktime_cache + q->horizon;
>         }
> ...
> 
> then I noticed that in some cases skb->tstamp is equal to real ts whereas in
> the regular cases where a packet pass through it's time since kernel boot. This
> doesn't make any sense for me as this condition is satisfied constantly
> 
> net/sched/sch_fq.c:439
> static bool fq_packet_beyond_horizon(const struct sk_buff *skb,
>                                     const struct fq_sched_data *q)
> {
>         return unlikely((s64)skb->tstamp > (s64)(q->ktime_cache + q->horizon));
> }
> 
> Any ideas on what it can be?
> 
> some outputs:
>     [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1081131982. all:
> q->ktime_cache 328453964122, skb->tstamp 1601391546982793177 <<
>     [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1485308564. all:
> q->ktime_cache 334998110463, skb->tstamp 335012588783
>     ...
>     [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: -1010372082. all:
> q->ktime_cache 335873778729, skb->tstamp 335741726080
>     [Tue Sep 29 14:59:06 2020] DBG: TIME: trid: 192888327. all: q->ktime_cache
> 335860696387, skb->tstamp 335870531339
>     [Tue Sep 29 14:59:07 2020] DBG: TIME: trid: -1463571809. all:
> q->ktime_cache 335305774517, skb->tstamp 1601391548109319017 <<
> 
>   perf trace: 
>     curl 32613 [014]  1546.957467: skb:kfree_skb: skbaddr=0xffff888f57159ae0
> protocol=2048 location=0xffffffff817bfdad
>                   9bfd56 kfree_skb (/lib/modules/5.8.10/build/vmlinux)
>                   9bfd56 kfree_skb (/lib/modules/5.8.10/build/vmlinux)
>                   9bfdad kfree_skb_list (/lib/modules/5.8.10/build/vmlinux)
>                   9da2b1 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                     1f39 vlan_dev_open ([8021q])
>                   9d9851 dev_hard_start_xmit
> (/lib/modules/5.8.10/build/vmlinux)
>                   9d9d05 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   9d9d05 __dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   9da380 dev_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   9e32c3 neigh_connected_output
> (/lib/modules/5.8.10/build/vmlinux)
>                   a68cdb ip_finish_output2 (/lib/modules/5.8.10/build/vmlinux)
>                   a69b77 __ip_finish_output (/lib/modules/5.8.10/build/vmlinux)
>                   a69cad ip_finish_output (/lib/modules/5.8.10/build/vmlinux)
>                   a6b65a ip_output (/lib/modules/5.8.10/build/vmlinux)
>                     f2e4 ip_vs_tunnel_xmit ([ip_vs])
>                     536a ip_vs_nat_icmp ([ip_vs])
>                     5913 ip_vs_out_icmp ([ip_vs])
>                     59c6 ip_vs_out_icmp ([ip_vs])
>                   a58255 nf_hook_slow (/lib/modules/5.8.10/build/vmlinux)
>                   a656f1 ip_local_deliver (/lib/modules/5.8.10/build/vmlinux)
>                   a64e07 ip_rcv_finish (/lib/modules/5.8.10/build/vmlinux)
>                   b717bb ip_sabotage_in (/lib/modules/5.8.10/build/vmlinux)
>                   a58255 nf_hook_slow (/lib/modules/5.8.10/build/vmlinux)
>                   a657a0 ip_rcv (/lib/modules/5.8.10/build/vmlinux)
>                   9db608 __netif_receive_skb_one_core
> (/lib/modules/5.8.10/build/vmlinux)
>                   9db658 __netif_receive_skb
> (/lib/modules/5.8.10/build/vmlinux)
>                   9db6e9 netif_receive_skb (/lib/modules/5.8.10/build/vmlinux)
>                   b5c5b6 br_netif_receive_skb
> (/lib/modules/5.8.10/build/vmlinux)
>                   b5c7d3 br_pass_frame_up (/lib/modules/5.8.10/build/vmlinux)
>                   b5c94d br_handle_frame_finish
> (/lib/modules/5.8.10/build/vmlinux)
>                   b7253b br_nf_hook_thresh (/lib/modules/5.8.10/build/vmlinux)
>                   b72fb0 br_nf_pre_routing_finish
> (/lib/modules/5.8.10/build/vmlinux)
>                   b733bc br_nf_pre_routing (/lib/modules/5.8.10/build/vmlinux)
>                   b5cdf1 br_handle_frame (/lib/modules/5.8.10/build/vmlinux)
>                   9da8c8 __netif_receive_skb_core
> (/lib/modules/5.8.10/build/vmlinux)
>                   9db5bf __netif_receive_skb_one_core
> (/lib/modules/5.8.10/build/vmlinux)
>                   9db658 __netif_receive_skb
> (/lib/modules/5.8.10/build/vmlinux)
>                   9db8a9 process_backlog (/lib/modules/5.8.10/build/vmlinux)
>                   9dd337 net_rx_action (/lib/modules/5.8.10/build/vmlinux)
>                   e000e1 __do_softirq (/lib/modules/5.8.10/build/vmlinux)
>                   c010c2 asm_call_on_stack (/lib/modules/5.8.10/build/vmlinux)
>                   235bef do_softirq_own_stack
> (/lib/modules/5.8.10/build/vmlinux)
>                   297f86 do_softirq.part.0 (/lib/modules/5.8.10/build/vmlinux)
>                   297fe0 __local_bh_enable_ip
> (/lib/modules/5.8.10/build/vmlinux)
>                   a68cef ip_finish_output2 (/lib/modules/5.8.10/build/vmlinux)
>                   a69b77 __ip_finish_output (/lib/modules/5.8.10/build/vmlinux)
>                   a69cad ip_finish_output (/lib/modules/5.8.10/build/vmlinux)
>                   a6b65a ip_output (/lib/modules/5.8.10/build/vmlinux)
>                   a6ad4d ip_local_out (/lib/modules/5.8.10/build/vmlinux)
>                   a6b097 __ip_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   a8f740 ip_queue_xmit (/lib/modules/5.8.10/build/vmlinux)
>                   a89746 __tcp_transmit_skb (/lib/modules/5.8.10/build/vmlinux)
>                   a8a5de tcp_connect (/lib/modules/5.8.10/build/vmlinux)
>                   a90f73 tcp_v4_connect (/lib/modules/5.8.10/build/vmlinux)
>                   aad59a __inet_stream_connect
> (/lib/modules/5.8.10/build/vmlinux)
>                   aad88b inet_stream_connect
> (/lib/modules/5.8.10/build/vmlinux)
>                   9b3e3f __sys_connect_file (/lib/modules/5.8.10/build/vmlinux)
>                   9b3ef1 __sys_connect (/lib/modules/5.8.10/build/vmlinux)
>                   9b3f3a __x64_sys_connect (/lib/modules/5.8.10/build/vmlinux)
>                   b896b2 do_syscall_64 (/lib/modules/5.8.10/build/vmlinux)
>                   c0008c entry_SYSCALL_64 (/lib/modules/5.8.10/build/vmlinux)
>                    53878 [unknown] (/lib/ld-musl-x86_64.so.1)
>                        0 [unknown] ([unknown])
>                        0 [unknown] ([unknown])
> 

Thanks for the detailed report !

I suspect ipvs or bridge code needs something similar to the fixes done in 

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=de20900fbe1c4fd36de25a7a5a43223254ecf0d0

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=41d1c8839e5f8cb781cc635f12791decee8271b7

The reason for that is that skb->tstamp can get a timestamp in input path,
with a base which is not CLOCK_MONOTONIC, unfortunately.

Whenever a packet is forwarded, its tstamp must be cleared.

Can you try :

diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index b00866d777fe0e9ed8018087ebc664c56f29b5c9..11e8ccdae358a89067046efa62ed40308b9e06f9 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -952,6 +952,8 @@ ip_vs_prepare_tunneled_skb(struct sk_buff *skb, int skb_af,
 
        ip_vs_drop_early_demux_sk(skb);
 
+       skb->tstamp = 0;
+
        if (skb_headroom(skb) < max_headroom || skb_cloned(skb)) {
                new_skb = skb_realloc_headroom(skb, max_headroom);
                if (!new_skb)

Powered by blists - more mailing lists