Message-ID: <1F86EBA5C8668B468CEE1C33B96E862B3689E278@IRSMSX104.ger.corp.intel.com>
Date:   Wed, 10 Jan 2018 09:33:07 +0000
From:   "Stachura, Mariusz" <mariusz.stachura@...el.com>
To:     Jakub Kicinski <kubakici@...pl>,
        "Kirsher, Jeffrey T" <jeffrey.t.kirsher@...el.com>
CC:     "davem@...emloft.net" <davem@...emloft.net>,
        "netdev@...r.kernel.org" <netdev@...r.kernel.org>,
        "nhorman@...hat.com" <nhorman@...hat.com>,
        "sassmann@...hat.com" <sassmann@...hat.com>,
        "jogreene@...hat.com" <jogreene@...hat.com>,
        "Williams, Mitch A" <mitch.a.williams@...el.com>
Subject: RE: [net-next v2 15/15] i40e: link_down_on_close private flag
 support

Hey Kuba,

Did you set private flag before doing ifconfig IFNAME down?

"ethtool --set-priv-flags enp5s0f0 link-down-on-close on"
"ifconfig enp5s0f0 down"

And the link partner should see this port as physically downed.

Setting phy_type to 0 is, as you said, a way of fooling FW. I'll try to address some of your comments in my next mail.

Many thanks and best regards,
Mariusz

-----Original Message-----
From: Jakub Kicinski [mailto:kubakici@...pl] 
Sent: Wednesday, January 10, 2018 7:41 AM
To: Kirsher, Jeffrey T <jeffrey.t.kirsher@...el.com>
Cc: davem@...emloft.net; Stachura, Mariusz <mariusz.stachura@...el.com>; netdev@...r.kernel.org; nhorman@...hat.com; sassmann@...hat.com; jogreene@...hat.com; Williams, Mitch A <mitch.a.williams@...el.com>
Subject: Re: [net-next v2 15/15] i40e: link_down_on_close private flag support

On Tue,  9 Jan 2018 08:19:30 -0800, Jeff Kirsher wrote:
> From: Mariusz Stachura <mariusz.stachura@...el.com>
> 
> This patch introduces new ethtool private flag used for forcing true 
> link state. Function i40e_force_link_state that implements this 
> functionality was added, it sets phy_type = 0 in order to work-around 
> firmware's LESM. False positive error messages were suppressed.

Hi, could you say a bit more about how this works?  Last week I've been actually trying to use a Fortville connected by a DAC to another card to test link state on that other.  ifconfig down does not bring the link down on the Fortville, apparently.  BMC is configured to only use LOM, no PXE etc.

With this patch I assume I will be able to force the link down?  What is keeping the link up otherwise?  You mention phy_type, is this phy-specific?  Or are you using the phy_type purely as a way of fooling FW to do what you want?

Few nits below..

> Signed-off-by: Mariusz Stachura <mariusz.stachura@...el.com>
> Signed-off-by: Mitch Williams <mitch.a.williams@...el.com>
> Tested-by: Andrew Bowers <andrewx.bowers@...el.com>
> Signed-off-by: Jeff Kirsher <jeffrey.t.kirsher@...el.com>
> ---
>  drivers/net/ethernet/intel/i40e/i40e_ethtool.c |  2 +
>  drivers/net/ethernet/intel/i40e/i40e_main.c    | 71 ++++++++++++++++++++++++++
>  2 files changed, 73 insertions(+)
> 
> diff --git a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c 
> b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
> index 34173f821fd9..50908f343221 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_ethtool.c
> @@ -230,6 +230,8 @@ static const struct i40e_priv_flags i40e_gstrings_priv_flags[] = {
>  	I40E_PRIV_FLAG("flow-director-atr", I40E_FLAG_FD_ATR_ENABLED, 0),
>  	I40E_PRIV_FLAG("veb-stats", I40E_FLAG_VEB_STATS_ENABLED, 0),
>  	I40E_PRIV_FLAG("hw-atr-eviction", I40E_FLAG_HW_ATR_EVICT_ENABLED, 
> 0),
> +	I40E_PRIV_FLAG("link-down-on-close",
> +		       I40E_FLAG_LINK_DOWN_ON_CLOSE_ENABLED, 0),
>  	I40E_PRIV_FLAG("legacy-rx", I40E_FLAG_LEGACY_RX, 0),
>  	I40E_PRIV_FLAG("disable-source-pruning",
>  		       I40E_FLAG_SOURCE_PRUNING_DISABLED, 0), diff --git 
> a/drivers/net/ethernet/intel/i40e/i40e_main.c 
> b/drivers/net/ethernet/intel/i40e/i40e_main.c
> index 666d2e7781e1..41fbccd41641 100644
> --- a/drivers/net/ethernet/intel/i40e/i40e_main.c
> +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c
> @@ -6600,6 +6600,72 @@ int i40e_up(struct i40e_vsi *vsi)
>  	return err;
>  }
>  
> +/**
> + * i40e_force_link_state - Force the link status
> + * @pf: board private structure
> + * @is_up: whether the link state should be forced up or down  **/ 
> +static void i40e_force_link_state(struct i40e_pf *pf, bool is_up) {
> +	struct i40e_aq_get_phy_abilities_resp abilities;
> +	struct i40e_aq_set_phy_config config = {0};
> +	struct i40e_hw *hw = &pf->hw;
> +	enum i40e_aq_phy_type cnt;
> +	u64 mask = 0;
> +	i40e_status err;

nit: reverse christmas tree

> +	/* Get the current phy config */
> +	err = i40e_aq_get_phy_capabilities(hw, false, false, &abilities,
> +					   NULL);

Are abilities guaranteed to be filled in by this function?  Otherwise you will use uninitialized stack memory.

> +	if (err)
> +		dev_dbg(&pf->pdev->dev,
> +			"failed to get phy cap., ret =  %s last_status =  %s\n",
> +			i40e_stat_str(hw, err),
> +			i40e_aq_str(hw, hw->aq.asq_last_status));

so getting 'abilities' can fail here...

> +	/* If link needs to go up, but was not forced to go down,
> +	 * no need for a flap
> +	 */
> +	if (is_up && abilities.phy_type != 0)
> +		return;
> +
> +	/* To force link we need to set bits for all supported PHY types,
> +	 * but there are now more than 32, so we need to split the bitmap
> +	 * across two fields.
> +	 */
> +	for (cnt = I40E_PHY_TYPE_SGMII; cnt < I40E_PHY_TYPE_MAX; cnt++)
> +		mask |= (1ULL << cnt);

nit: looks like a GENMASK_ULL() coded as a loop

> +	config.phy_type = is_up ? cpu_to_le32((u32)(mask & 0xffffffff)) : 0;
> +	config.phy_type_ext = is_up ? (u8)((mask >> 32) & 0xff) : 0;
> +	/* Copy the old settings, except of phy_type */
> +	config.abilities = abilities.abilities;
> +	config.link_speed = abilities.link_speed;
> +	config.eee_capability = abilities.eee_capability;
> +	config.eeer = abilities.eeer_val;
> +	config.low_power_ctrl = abilities.d3_lpan;

... but then here you use the values.  Can this not lead to trouble?

> +	err = i40e_aq_set_phy_config(hw, &config, NULL);
> +
> +	if (err)
> +		dev_dbg(&pf->pdev->dev,
> +			"set phy config ret =  %s last_status =  %s\n",
> +			i40e_stat_str(&pf->hw, err),
> +			i40e_aq_str(&pf->hw, pf->hw.aq.asq_last_status));

IMHO ignoring errors on close/down is fine in this case, but if the link is forced down and then subsequent open can't bring it up the open should fail.  dev_dbg() won't even show up in logs on 99.9% of machines.

Perhaps I'm misunderstanding what this patch does.

> +	/* Update the link info */
> +	err = i40e_update_link_info(hw);
> +	if (err) {
> +		/* Wait a little bit (on 40G cards it sometimes takes a really
> +		 * long time for link to come back from the atomic reset)
> +		 * and try once more
> +		 */
> +		msleep(1000);
> +		i40e_update_link_info(hw);
> +	}
> +
> +	i40e_aq_set_link_restart_an(hw, true, NULL); }
> +
>  /**
>   * i40e_down - Shutdown the connection processing
>   * @vsi: the VSI being stopped
> @@ -6617,6 +6683,9 @@ void i40e_down(struct i40e_vsi *vsi)
>  	}
>  	i40e_vsi_disable_irq(vsi);
>  	i40e_vsi_stop_rings(vsi);
> +	if (vsi->type == I40E_VSI_MAIN &&
> +	    vsi->back->flags & I40E_FLAG_LINK_DOWN_ON_CLOSE_ENABLED)
> +		i40e_force_link_state(vsi->back, false);
>  	i40e_napi_disable_all(vsi);
>  
>  	for (i = 0; i < vsi->num_queue_pairs; i++) { @@ -7578,6 +7647,8 @@ 
> int i40e_open(struct net_device *netdev)
>  
>  	netif_carrier_off(netdev);
>  
> +	i40e_force_link_state(pf, true);
> +
>  	err = i40e_vsi_open(vsi);
>  	if (err)
>  		return err;

--------------------------------------------------------------------

Intel Technology Poland sp. z o.o.
ul. Slowackiego 173 | 80-298 Gdansk | Sad Rejonowy Gdansk Polnoc | VII Wydzial Gospodarczy Krajowego Rejestru Sadowego - KRS 101882 | NIP 957-07-52-316 | Kapital zakladowy 200.000 PLN.

Ta wiadomosc wraz z zalacznikami jest przeznaczona dla okreslonego adresata i moze zawierac informacje poufne. W razie przypadkowego otrzymania tej wiadomosci, prosimy o powiadomienie nadawcy oraz trwale jej usuniecie; jakiekolwiek
przegladanie lub rozpowszechnianie jest zabronione.
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by
others is strictly prohibited.

Powered by blists - more mailing lists