linux-kernel - ESP failure when PTB is received as non ESP packet and payload as ESP

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <TU4PR8401MB121629D61C3659EAD7C72441F6E10@TU4PR8401MB1216.NAMPRD84.PROD.OUTLOOK.COM>
Date:   Wed, 18 Nov 2020 03:49:34 +0000
From:   "Bhat, Jayalakshmi Manjunath" <jayalakshmi.bhat@...com>
To:     "linux-crypto@...r.kernel.org" <linux-crypto@...r.kernel.org>,
        "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>
Subject: ESP failure when PTB is received as non ESP packet and payload as ESP

Hi All,

There are 2 difference scenarios our device receives PTB.

In the first scenario PTB is received as ESP packet. First ESP gets decrypted and has payload of ICMP packet as PTB. PTB is 1280.
PMTU is set on the device via icmp6_err function in kernel.  

This is followed by device receiving fragmented PING requests.  As part of ESP and XFRM processing esp6_get_mtu gets called.
And esp6_get_mtu calculates MTU as MTU-ESP HEADER-IPHEADER etc. Here MTU remains as 1500 and calculated MTU is also fine.
Thus the execution works good.  

In the second scenario PTB is received as non ESP packet. However payload in this packet is an ESP. PTB is 1280.  PMTU is set via 
esp6_err function in kernel. This function also invokes xfrm_policy_destroy. 

This is followed by device receiving fragmented PING requests. Device creates new xfrm_state and sets the MTU as 1280.
esp6_get_mtu calculates MTU as 1280- ESP HEADER-IPHEADER etc which result MTU < IP6_MIN_MTU and test fails.

Has anyone encountered scenario 2. Any inputs are appreciated.

Thanks and Regards,
Jayalakshmi