netdev - Re: PROBLEM: xfrm: XFRMINSTATEMODEERROR for transport mode IPsec SA when IP VTI is active

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives
Hash Suite: Windows password security audit tool. GUI, reports in PDF.
[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]
Message-ID: <9223b7dec124ef0089e0cc0ab04cd711@rmntn.net>
Date:   Sat, 22 Dec 2018 18:38:57 +0800
From:   Lemon Lam <almk@...tn.net>
To:     Steffen Klassert <steffen.klassert@...unet.com>
Cc:     herbert@...dor.apana.org.au, davem@...emloft.net,
        netdev@...r.kernel.org
Subject: Re: PROBLEM: xfrm: XFRMINSTATEMODEERROR for transport mode IPsec SA
 when IP VTI is active

Thanks Steffen, but I don't think it is the case.

I shut down VTI interface toward another VPS and GRE on top of it,
enabled the plain GRE for transport SA. It works on one end, but not for
the other end which has to leave VTI with `remote any` up. How can the
transport SA match this `remote any` VTI?

Attached relevant `ip tun`, `ip link`, `ip addr` and `ip xfrm pol` for
your investigation.

On the working end:

root@...ka:~# ip tun
test: gre/ip remote 139.162.122.174 local 149.248.36.252 dev ens3 ttl 
inherit nopmtudisc
goips-i0n2: gre/ip remote 172.16.44.7 local 172.16.44.6 dev ipsec0 ttl 
inherit
ipsec0: ip/ip remote 139.162.122.174 local 149.248.36.252 dev ens3 ttl 
inherit nopmtudisc key 410191106

root@...ka:~# ip link
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
mode DEFAULT group default qlen 1000
     link/ether 56:00:01:c6:b6:20 brd ff:ff:ff:ff:ff:ff
12: ipsec0@...3: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN 
mode DEFAULT group default qlen 1000
     link/ipip 149.248.36.252 peer 139.162.122.174
13: goips-i0n2@...ec0: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue 
state DOWN mode DEFAULT group default qlen 1000
     link/gre 172.16.44.6 peer 172.16.44.7
14: test@...3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue 
state UNKNOWN mode DEFAULT group default qlen 1000
     link/gre 149.248.36.252 peer 139.162.122.174

root@...ka:~# ip addr
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP 
group default qlen 1000
     link/ether 56:00:01:c6:b6:20 brd ff:ff:ff:ff:ff:ff
     inet 149.248.36.252/23 brd 149.248.37.255 scope global ens3
        valid_lft forever preferred_lft forever
12: ipsec0@...3: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN 
group default qlen 1000
     link/ipip 149.248.36.252 peer 139.162.122.174
     inet 172.16.44.6/31 brd 255.255.255.255 scope global ipsec0
        valid_lft forever preferred_lft forever
13: goips-i0n2@...ec0: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue 
state DOWN group default qlen 1000
     link/gre 172.16.44.6 peer 172.16.44.7
     inet 172.22.181.16/31 brd 255.255.255.255 scope global goips-i0n2
        valid_lft forever preferred_lft forever
14: test@...3: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue 
state UNKNOWN group default qlen 1000
     link/gre 149.248.36.252 peer 139.162.122.174
     inet 172.22.181.16/31 scope global test
        valid_lft forever preferred_lft forever

root@...ka:~# ip xfrm pol
src 149.248.36.252/32 dst 139.162.122.174/32 proto gre
         dir out priority 366975 ptype main
         tmpl src 0.0.0.0 dst 0.0.0.0
                 proto esp spi 0xc9d4a0c3 reqid 6 mode transport
src 139.162.122.174/32 dst 149.248.36.252/32 proto gre
         dir in priority 366975 ptype main
         tmpl src 0.0.0.0 dst 0.0.0.0
                 proto esp reqid 6 mode transport


On the not working end:

root@...one:~# ip tun
ipsec0: ip/ip remote any local 139.162.122.174 dev eth0 ttl inherit 
nopmtudisc key 410190082
goips-i0n1: gre/ip remote 172.16.44.1 local 172.16.44.0 dev ipsec0 ttl 
inherit
test: gre/ip remote 149.248.36.252 local 139.162.122.174 dev eth0 ttl 
inherit nopmtudisc
goips-i2n5: gre/ip remote 172.16.44.6 local 172.16.44.7 dev ipsec2 ttl 
inherit
ipsec2: ip/ip remote 149.248.36.252 local 139.162.122.174 dev eth0 ttl 
inherit nopmtudisc key 410191106

root@...one:~# ip link
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP mode DEFAULT group default qlen 1000
     link/ether f2:3c:91:70:9c:ad brd ff:ff:ff:ff:ff:ff
12: ipsec0@...0: <NOARP,UP,LOWER_UP> mtu 1446 qdisc noqueue state 
UNKNOWN mode DEFAULT group default qlen 1000
     link/ipip 139.162.122.174 brd 0.0.0.0
14: ipsec2@...0: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN 
mode DEFAULT group default qlen 1000
     link/ipip 139.162.122.174 peer 149.248.36.252
15: goips-i0n1@...ec0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1422 qdisc 
noqueue state UNKNOWN mode DEFAULT group default qlen 1000
     link/gre 172.16.44.0 peer 172.16.44.1
17: goips-i2n5@...ec2: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue 
state DOWN mode DEFAULT group default qlen 1000
     link/gre 172.16.44.7 peer 172.16.44.6
18: test@...0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue 
state UNKNOWN mode DEFAULT group default qlen 1000
     link/gre 139.162.122.174 peer 149.248.36.252

root@...one:~# ip addr
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP group default qlen 1000
     link/ether f2:3c:91:70:9c:ad brd ff:ff:ff:ff:ff:ff
     inet 139.162.122.174/24 brd 139.162.122.255 scope global eth0
        valid_lft forever preferred_lft forever
12: ipsec0@...0: <NOARP,UP,LOWER_UP> mtu 1446 qdisc noqueue state 
UNKNOWN group default qlen 1000
     link/ipip 139.162.122.174 brd 0.0.0.0
     inet 172.16.44.0/31 brd 255.255.255.255 scope global ipsec0
        valid_lft forever preferred_lft forever
14: ipsec2@...0: <POINTOPOINT,NOARP> mtu 1446 qdisc noqueue state DOWN 
group default qlen 1000
     link/ipip 139.162.122.174 peer 149.248.36.252
     inet 172.16.44.7/31 brd 255.255.255.255 scope global ipsec2
        valid_lft forever preferred_lft forever
15: goips-i0n1@...ec0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1422 qdisc 
noqueue state UNKNOWN group default qlen 1000
     link/gre 172.16.44.0 peer 172.16.44.1
     inet 172.22.181.10/31 brd 255.255.255.255 scope global goips-i0n1
        valid_lft forever preferred_lft forever
17: goips-i2n5@...ec2: <POINTOPOINT,NOARP,M-DOWN> mtu 1422 qdisc noqueue 
state DOWN group default qlen 1000
     link/gre 172.16.44.7 peer 172.16.44.6
     inet 172.22.181.17/31 brd 255.255.255.255 scope global goips-i2n5
        valid_lft forever preferred_lft forever
18: test@...0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1400 qdisc noqueue 
state UNKNOWN group default qlen 1000
     link/gre 139.162.122.174 peer 149.248.36.252
     inet 172.22.181.17/31 scope global test
        valid_lft forever preferred_lft forever

root@...one:~# ip xfrm pol
src 172.16.44.0/31 dst 172.16.44.0/31
         dir out priority 368255 ptype main
         mark 0x18730102/0xffffffff
         tmpl src 139.162.122.174 dst 122.100.194.181
                 proto esp spi 0xd5d1c8e9 reqid 24 mode tunnel
src 172.16.44.0/31 dst 172.16.44.0/31
         dir fwd priority 368255 ptype main
         mark 0x18730102/0xffffffff
         tmpl src 122.100.194.181 dst 139.162.122.174
                 proto esp reqid 24 mode tunnel
src 172.16.44.0/31 dst 172.16.44.0/31
         dir in priority 368255 ptype main
         mark 0x18730102/0xffffffff
         tmpl src 122.100.194.181 dst 139.162.122.174
                 proto esp reqid 24 mode tunnel
src 139.162.122.174/32 dst 149.248.36.252/32 proto gre
         dir out priority 366975 ptype main
         tmpl src 0.0.0.0 dst 0.0.0.0
                 proto esp spi 0xc815bb54 reqid 25 mode transport
src 149.248.36.252/32 dst 139.162.122.174/32 proto gre
         dir in priority 366975 ptype main
         tmpl src 0.0.0.0 dst 0.0.0.0
                 proto esp reqid 25 mode transport

On 2018/12/21 04:38 PM, Steffen Klassert wrote:
> On Sat, Dec 15, 2018 at 07:02:41PM +0800, Lemon Lam wrote:
>> Hello,
>> 
>> I recently joined DN42 with my virtual private servers, and decided to 
>> use
>> GRE
>> and IPsec to form interconnects between servers.
>> I can use GRE over IPsec VTI tunnel fine, but when I simplified some 
>> tunnels
>> down to GRE over IPsec transport, no incoming traffic is possible.
>> Please look into full description below.
>> 
>> [1.] One line summary of the problem:
>>      XFRMINSTATEMODEERROR for transport mode IPsec SA when IP VTI is 
>> active
>> 
>> [2.] Full description of the problem/report:
>>      I built tunnels according to StrongSwan's guide on VTI, i.e. 
>> using
>>      `ip tun add ipsecvti mode vti key <hex key>`, then I add GRE on 
>> top of
>>      it for MPLS. Everything works great at this stage.
>> 
>>      I want to strip it down to GRE over IPsec transport between my 
>> VPS but
>>      have to leave one as-is since there's endpoint with dynamic IP, 
>> need
>> this
>>      as workaround. After necessary configurations, I pinged between
>> transport
>>      mode tunnel, received no response. `swanctl -l` showed increased
>> outgoing
>>      traffic counter, but incoming counter stayed at zero. `tcpdump` 
>> showed
>>      incoming ESP packets on physical interfaces but no corresponding
>> packets
>>      on GRE tunnel.
>> 
>>      Hinted to look at `/proc/net/xfrm_stat` by developers from 
>> StrongSwan,
>>      found out that XFRMINSTATEMODEERROR increases by any traffic on
>>      transport tunnel. Later experiments discovered that merely
>>      `ip link set ipsecvti down` will let incoming traffic went 
>> through.
> 
> This looks like you have a transport mode SA that matches
> the src and dst endpoint of the vti tunnel. If this is
> the case, it is a conceptional problem. VTI behaves like
> an IP tunnel, it can not handle transport mode packets.