netdev - Re: traceroute failure in kernel 6.1 and 6.2

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <168aa9cf-d80a-9c1b-887f-97015a0473dc@gmail.com>
Date:   Mon, 23 Jan 2023 23:45:46 +0200
From:   Mantas Mikulėnas <grawity@...il.com>
To:     Eric Dumazet <edumazet@...gle.com>
Cc:     netdev@...r.kernel.org
Subject: Re: traceroute failure in kernel 6.1 and 6.2

On 23/01/2023 22.56, Eric Dumazet wrote:
> On Mon, Jan 23, 2023 at 8:25 PM Mantas Mikulėnas <grawity@...il.com> wrote:
>>
>> On 2023-01-23 17:21, Eric Dumazet wrote:
>>> On Sat, Jan 21, 2023 at 7:09 PM Mantas Mikulėnas <grawity@...il.com> wrote:
>>>>
>>>> Hello,
>>>>
>>>> Not sure whether this has been reported, but:
>>>>
>>>> After upgrading from kernel 6.0.7 to 6.1.6 on Arch Linux, unprivileged
>>>> ICMP traceroute using the `traceroute -I` tool stopped working – it very
>>>> reliably fails with a "No route to host" at some point:
>>>>
>>>>           myth> traceroute -I 83.171.33.188
>>>>           traceroute to 83.171.33.188 (83.171.33.188), 30 hops max, 60
>>>>           byte packets
>>>>            1  _gateway (192.168.1.1)  0.819 ms
>>>>           send: No route to host
>>>>           [exited with 1]
>>>>
>>>> while it still works for root:
>>>>
>>>>           myth> sudo traceroute -I 83.171.33.188
>>>>           traceroute to 83.171.33.188 (83.171.33.188), 30 hops max, 60
>>>>           byte packets
>>>>            1  _gateway (192.168.1.1)  0.771 ms
>>>>            2  * * *
>>>>            3  10.69.21.145 (10.69.21.145)  47.194 ms
>>>>            4  82-135-179-168.static.zebra.lt (82.135.179.168)  49.124 ms
>>>>            5  213-190-41-3.static.telecom.lt (213.190.41.3)  44.211 ms
>>>>            6  193.219.153.25 (193.219.153.25)  77.171 ms
>>>>            7  83.171.33.188 (83.171.33.188)  78.198 ms
>>>>
>>>> According to `git bisect`, this started with:
>>>>
>>>>           commit 0d24148bd276ead5708ef56a4725580555bb48a3
>>>>           Author: Eric Dumazet <edumazet@...gle.com>
>>>>           Date:   Tue Oct 11 14:27:29 2022 -0700
>>>>
>>>>               inet: ping: fix recent breakage
>>>>
>>>>
>>>>
>>>>
>>>> It still happens with a fresh 6.2rc build, unless I revert that commit.
>>>>
>>>> The /bin/traceroute is the one that calls itself "Modern traceroute for
>>>> Linux, version 2.1.1", on Arch Linux. It seems to use socket(AF_INET,
>>>> SOCK_DGRAM, IPPROTO_ICMP), has neither setuid nor file capabilities.
>>>> (The problem does not occur if I run it as root.)
>>>>
>>>> This version of `traceroute` sends multiple probes at once (with TTLs
>>>> 1..16); according to strace, the first approx. 8-12 probes are sent
>>>> successfully, but eventually sendto() fails with EHOSTUNREACH. (Though
>>>> if I run it on local tty as opposed to SSH, it fails earlier.) If I use
>>>> -N1 to have it only send one probe at a time, the problem doesn't seem
>>>> to occur.
>>>
>>>
>>>
>>> I was not able to reproduce the issue (downloading
>>> https://sourceforge.net/projects/traceroute/files/latest/download)
>>>
>>> I suspect some kind of bug in this traceroute, when/if some ICMP error
>>> comes back.
>>>
>>> Double check by
>>>
>>> tcpdump -i ethXXXX icmp
>>>
>>> While you run traceroute -I ....
>>
>> Hmm, no, the only ICMP errors I see in tcpdump are "Time exceeded in
>> transit", which is expected for traceroute. Nothing else shows up.
>>
>> (But when I test against an address that causes *real* ICMP "Host
>> unreachable" errors, it seems to handle those correctly and prints "!H"
>> as usual -- that is, if it reaches that point without dying.)
>>
>> I was able to reproduce this on a fresh Linode 1G instance (starting
>> with their Arch image), where it also happens immediately:
>>
>>          # pacman -Sy archlinux-keyring
>>          # pacman -Syu
>>          # pacman -Sy traceroute strace
>>          # reboot
>>          # uname -r
>>          6.1.7-arch1-1
>>          # useradd foo
>>          # su -c "traceroute -I 8.8.8.8" foo
>>          traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
>>           1  10.210.1.195 (10.210.1.195)  0.209 ms
>>          send: No route to host
>>
>> So now I'm fairly sure it is not something caused by my own network, either.
>>
>> On one system, it seems to work properly about half the time, if I keep
>> re-running the same command.
>>
> 
> Here, running the latest  upstream tree and latest traceroute, I have no issue.
> 
> Send us :
> 
> 1) strace output
> 2) icmp packet capture.
> 
> Thanks.

Attached both.
Download attachment "arch_kconfig.gz" of type "application/gzip" (62237 bytes)

View attachment "as_root.strace" of type "text/plain" (48164 bytes)

Download attachment "traceroute.pcap" of type "application/vnd.tcpdump.pcap" (2326 bytes)

View attachment "traceroute.strace" of type "text/plain" (14195 bytes)