netdev - Re: Invalid transport_offset with AF

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <79348d7b2068b5c8494a2a8dfbc642b82cb54600.camel@mellanox.com>
Date:   Wed, 28 Nov 2018 02:06:38 +0000
From:   Saeed Mahameed <saeedm@...lanox.com>
To:     Maxim Mikityanskiy <maximmi@...lanox.com>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "jasowang@...hat.com" <jasowang@...hat.com>,
        "edumazet@...gle.com" <edumazet@...gle.com>,
        "davem@...emloft.net" <davem@...emloft.net>
CC:     Eran Ben Elisha <eranbe@...lanox.com>,
        "willemb@...gle.com" <willemb@...gle.com>,
        Tariq Toukan <tariqt@...lanox.com>
Subject: Re: Invalid transport_offset with AF_PACKET socket

On Tue, 2018-11-27 at 10:00 +0000, Maxim Mikityanskiy wrote:
> Hi everyone,
> 
> We are experiencing an issue with Mellanox mlx5 driver, and I tracked
> it down to
> the packet_snd function in net/packet/af_packet.c.
> 
> Brief description: when a socket is created by calling
> `socket(AF_PACKET,
> SOCK_RAW, 0)`, the mlx5 driver receives an skb with wrong
> transport_offset,
> which can confuse the driver and cause the transmit to fail
> (depending on the
> configuration of the NIC).

Hi Max, 
Can you elaborate more, what NIC? what configuration ? what do you mean
by confusion, anyway please see below

> 
> The flow is the following:
> 
> 1. packet_snd is called.
> 
> 2. dev->hard_header_len (which is 14) is assigned to reserve.
> 
> 3. The value of the third parameter of the initial socket() call is
> assigned to
> skb->protocol. In our case, it's 0.
> 
> 4. skb_probe_transport_header is called with offset_hint == reserve
> (which is
> 14).
> 
> 5. __skb_flow_dissect fails, because skb->protocol is 0.
> 
> 6. skb_probe_transport_header happily sets transport_header to 14.
> 
> I find this behavior (defaulting to 14) strange, because
> network_header is also
> set to 14, and the transport_header value is just wrong. Moreover,
> there are two
> more calls to skb_probe_transport_header in this file with
> offset_hint == 0,
> which looks more reasonable (if we can't find the transport header,
> we indicate
> that there is none, instead of pointing to the network header).
> 
> Does anyone know why offset_hint is set to 14 in this single place?
> Can it be
> replaced by 0 safely, and what can be the consequences?
> 
> Also, what guarantees does kernel provide for the network and
> transport header
> offsets? Especially in raw sockets, where the headers are not
> generated by
> different stack layers.
> 

in mlx5 with ConnectX4 or Connext4-LX there is a requirement to copy at
least the ethernet header to the tx descriptor otherwise this might
cause the packet to be dropped, and for RAW sockets the skb headers
offsets are not set, but the latest mlx5 upstream driver would know how
to handle this, and copy the minmum amount required
please see:

static inline u16 mlx5e_calc_min_inline(enum mlx5_inline_modes mode,
					struct sk_buff *skb)


it should default to:


case MLX5_INLINE_MODE_L2:
	default:
		hlen = mlx5e_skb_l2_header_offset(skb);


static inline int mlx5e_skb_l2_header_offset(struct sk_buff *skb)
{
#define MLX5E_MIN_INLINE (ETH_HLEN + VLAN_HLEN)

	return max(skb_network_offset(skb), MLX5E_MIN_INLINE);
}

So it should return at least 18 and not 14.

We had some issues with this in old driver such as kernels 4.14/15, and
it depends in the use case so i need some information first:

1. What Cards do you have ? (lspci)

2. What kernel/driver version are you using ?

3. what is the current enum mlx5_inline_modes seen in
mlx5e_calc_min_inline or sq->min_inline_mode ?

4. Firmware version ? (ethtool -i)

can you share the packet format you are sending and seeing the bad
behavior with 

> Thanks,
> Max