Message-ID: <ae31b47c-b72b-a1b5-7d5f-54f77db8e282@bitbybitsoftwaregroup.com>
Date:   Wed, 17 Jan 2018 19:59:53 -0600
From:   Jamie Krueger <jamie@...bybitsoftwaregroup.com>
To:     Madalin-cristian Bucur <madalin.bucur@....com>,
        Darren Stevens <darren@...vens-zone.net>
Cc:     "linuxppc-dev@...ts.ozlabs.org" <linuxppc-dev@...ts.ozlabs.org>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>
Subject: Re: DPAA Ethernet problems with mainstream Linux kernels

Hi Madalin,

On 01/16/2018 11:33 AM, Madalin-cristian Bucur wrote:
>> -----Original Message-----
>> From: Darren Stevens [mailto:darren@...vens-zone.net]
>> Sent: Tuesday, January 16, 2018 12:40 AM
>> To: Madalin-cristian Bucur <madalin.bucur@....com>
>> Cc: Jamie Krueger <jamie@...bybitsoftwaregroup.com>; linuxppc-
>> dev@...ts.ozlabs.org; netdev@...r.kernel.org
>> Subject: Re: DPAA Ethernet problems with mainstream Linux kernels
>>
>> Hello Madalin-cristian
>>
>> On 15/01/2018, Madalin-cristian Bucur wrote:
>>>>> The device tree that you mention, cyrus_p5020.eth.dts is not found in
>>>>> the Linux kernel sources. The cyrus_p5020.dts file from the fsl ppc
>>>>> device tree folder does not include the PHY information for the DPAA
>>>>> interfaces. The problems that you experience may be caused by some
>>>>> issues with the PHY configuration (i.e. internal delay).
>>>> The cyrus_p5020.eth.dts is a modified version of the cyrus_p5020.dts,
>>>> which of course was based off the original p5020ds.dts file. As you
>>>> noted, the current cyrus_p5020.dts file is incomplete, and does not
>>>> map the Ethernet connections properly.
>> This is because the current linux kernel version of cyrus_p5020.dts
>> includes 'p5020si-pre.dtsi' and 'p5020si-post.dtsi' include files, which
>> orginally gave us working ethernet (when we used the SDK kernel) However
>> at some point you moved the ethernet aliases from the board dts file to
>> the p5020si-pre.dtsi file breaking the linkages for our board.
>>
>> cyrus-pre.dtsi is simply p5020si-pre.dtsi with only the correct aliases
>> in.
>>
>>>> ** I have attached both the cyrus_p5020.eth.dts and cyrus-pre.dtsi
>>>>        files with this email for comparison. Please let me know if you
>> see
>>>>        any corrections that should be made to either file.
>>> At a first glance they look fine to me.
>> That's good to know.
>>
>>>> I have started testing along that line, using Wireshark to view the
>>>> traffic on the X5000/20 itself, and from another machine connected
>>>> on the same subnet. So far (as indicated by some details of in my
>>>> initial email), I can see outgoing broadcast requests (for DHCP)
>>>> being sent out from the X5000/20, and these requests are correctly
>>>> constructed and visible outside the X5000/20.
>>>>
>>>> However, no responses to the DHCP broadcasts appear to reach
>>>> to X5000/20's DPAA Ethernet. I will need to setup some further
>>>> tests to determine if the DHCP server saw the requests and responded
>>>> to them. (I assume the DHCP server is getting them, and responding,
>>>> as I can always get a successful DHCP response to the X5000/20
>>>> when using an add-on Ethernet PICe card on the same subnet).
>> This matches what I see, the interface I have connected to the network
>> shows an increasing number of transmitted packets, but no received ones.
>>
>> Jamie also noticed the following error in dmesg (right after the ethernet
>> port becomes active)
>>
>> [    4.112165] fsl_dpa dpaa-ethernet.0 eth0: Probed interface eth0
>> [    4.116616] fsl_dpa dpaa-ethernet.1 eth1: Probed interface eth1
>> [ ... ]
>> [  106.501055] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
>> [  106.570944] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready
>> [  106.605044] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [  106.674918] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready
>> [  108.702771] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
>> [  109.032798] fsl-pamu: pamu_av_isr: access violation interrupt
>> [  109.032806] fsl-pamu: pamu_av_isr: POES1=00000000
>> [  109.032808] fsl-pamu: pamu_av_isr: POES2=00000000
>> [  109.032809] fsl-pamu: pamu_av_isr: AVS1=002d0081
>> [  109.032811] fsl-pamu: pamu_av_isr: AVS2=00000081
>> [  109.032813] fsl-pamu: pamu_av_isr: AVA=00000001f1328000
>> [  109.032815] fsl-pamu: pamu_av_isr: UDAD=002d0001
>> [  109.032817] fsl-pamu: pamu_av_isr: POEA=0000000000000000
>>
>> I haven't seen this anywhere else, and wondered if it is relevant.
>>
>> Regards
>> Darren
> The PAMU related errors may be relevant to the issue, if you have incorrect
> settings you may have no traffic passing through. The PAMU configuration
> should be made by the bootloader. Can you try to disable CONFIG_FSL_PAMU?
>
> Madalin
>

Here are some results from testing on the X5000/20 with 4.15.0-rc8
(w/CONFIG_FSL_PAMU disabled):

In my tests I can now get the interface to obtain an IP Address via DHCP,
at least *part* of the time that is.

Here is a link to a screenshot of a Wireshark capture which shows
a successful DHCP request/response for the X5000/20, followed by some
failed ping attempts.

(This traffic was captured on an external machine on the same subnet, 
and not from the X5000/20 itself)
http://bitbybitsoftwaregroup.com/share/X5000-DHCP-Answered-Wireshark.png

Additionally, here is a link to the export of the packet data shown in 
the above image:
http://bitbybitsoftwaregroup.com/share/Wireshark-X5000-DHCP-Success.html

As you can see by this example, even when the interface does obtain an 
IP address via
DHCP (about half the time), subsequent traffic still fails to receive 
responses.

In this example, I attempt to ping the gateway (192.168.1.1), and one or 
two other
local addresses, and nothing pings back (even when trying Skateman's 
suggestion
of expanding the buffers for the interface -

"A workaround to keep the nic alive a bit longer.... is the following 
command
ip link set eth0 qlen 10000"

For reference:

I have attached the Kernel config that the version I tested was built with.
Also, I have attached the dmesg output I am currently seeing.

-- 
Best Regards,

Jamie Krueger
BITbyBIT Software Group LLC


View attachment "proc_config-4.15.0-rc8-NO_PAMU.txt" of type "text/plain" (136814 bytes)

View attachment "fsl_dmesg_output.txt" of type "text/plain" (4761 bytes)

Powered by blists - more mailing lists