Message-ID: <529ff560-8230-c799-2948-fae711de382e@kot-begemot.co.uk>
Date:   Thu, 12 Oct 2017 09:46:44 +0100
From:   Anton Ivanov <anton.ivanov@...-begemot.co.uk>
To:     Anton Ivanov <anton.ivanov@...bridgegreys.com>,
        Willem de Bruijn <willemdebruijn.kernel@...il.com>
Cc:     Network Development <netdev@...r.kernel.org>,
        David Miller <davem@...emloft.net>
Subject: Re: BUG:af_packet fails to TX TSO frames

Hi Willem,

Thanks for all your help so far.

I modified your test program to do sendmmsg as an option in addition to 
send. I will send you a github pull req with the changes later today. 
This allows to send let's say n frames (let's say 64) in one syscall - 
probably the highest TX rate you can get out of Linux userspace.

The test case works via that route too which is now getting into the 
realm of the surreal.

If I produce a real vnet frame out of a live kernel frame using 
virtio_net_hdr_from_skb() and try to send it it fails on the check in 
af_packet, while succeeding for tap. If I remove the af_packet check the 
frame is accepted by the hardware too.

If I produce it a synthetic frame + vnet header using the test program - 
it works. Go figure.

I am going to continue digging into it.

At the very least I now have a positive test case which uses the same 
semantics as my code so I have something to compare to.

A.

On 10/12/17 07:11, Anton Ivanov wrote:
> On 12/10/17 01:19, Willem de Bruijn wrote:
>> On Wed, Oct 11, 2017 at 6:01 PM, Anton Ivanov
>> <anton.ivanov@...bridgegreys.com> wrote:
>>> [snip]
>>>
>>>> This will be tomorrow though, it is late here.
>>>>
>>>> The only obvious difference I can see at this point is that I am using
>>>> iovs and sending the vnet header as iov[0] and the data in pieces after
>>>> that while your code is doing a send() for the whole frame. This should
>>>> not make any difference though - it all ends up as an iov internally in
>>>> the kernel.
>>> Spoke too soon. It is not reporting any errors, but there is nothing
>>> coming out on the actual Ethernet.
> Different issue - switch was blacklisting the fake dst MAC after the
> first packet as a suspected flood attack.
>
> It worked after I changed the mac in the source to a "real" one. In
> fact, an argument for that
> would be nice :)
>
> I will go through it step by step now and figure out exactly what and
> where is wrong with the framing on my side.
>
> Thanks for your help.
>
> A.
>
>> It works for me on various platforms. On the receiver, drop these fake
>> tcp packets in iptables and read them with tcpdump
>>
>>     iptables -A PREROUTING -t raw -p tcp --dport 9 -j DROP
>>     tcpdump src $src_ip
>>
>> Note that not all combinations of flags are supported by the kernel
>> and that some flags have non-obvious behavior (disable a feature, in
>> place of enable it).
>>
>> Specifically, mtu sized packets either must not pass a vnet_hdr or
>> must pass one with gso explicitly disabled ('-G').
>>
>>    psock_txring_vnet -s $src_ip $dst_ip -l 1400
>>    psock_txring_vnet -s $src_ip $dst_ip -l 1400 -v -G
>>    psock_txring_vnet -s $src_ip $dst_ip -l 1400 -N
>>    psock_txring_vnet -s $src_ip $dst_ip -l 1400 -N -v -G
>>
>> Conversely, packets that exceed mtu have to have the gso flags in the
>> virtio_net_hdr:
>>
>>    psock_txring_vnet -s $src_ip $dst_ip -l 4400 -v
>>    psock_txring_vnet -s $src_ip $dst_ip -l 4400 -N -v
>>
>> When sending a large packet, but not passing a virtio_net_hdr along
>> ('-v'), the test fails with
>>
>>    psock_txring_vnet: send: Message too long
>>
>> When passing a header along, but not disabling gso, the packet is
>> indeed dropped silently.
>>
>> I verified correct segmentation with three modes of ethtool
>>
>>    ethtool -K eth0 tso off gso off
>>    ethtool -K eth0 tso off gso on
>>    ethtool -K eth0 tso on gso on
>>
>> by reading tcpdump on the sender.
>>
>> The receive side results are the same with dev_queue_xmit and
>> packet_direct_xmit ('-q') mode. With direct_xmit, the packets are not
>> observed on the send side.
>>
>

Powered by blists - more mailing lists