netdev - Re: [PATCH v4 2/5] r8169: Coalesce RTL8411b PHY power-down recovery calls to reduce spinlock stalls

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <766f1321-ea01-40c1-832a-3989677e7d4b@intel.com>
Date: Mon, 30 Oct 2023 14:52:30 -0700
From: Jacob Keller <jacob.e.keller@...el.com>
To: Mirsad Goran Todorovac <mirsad.todorovac@....unizg.hr>, Jason Gunthorpe
	<jgg@...pe.ca>, Joerg Roedel <jroedel@...e.de>, Lu Baolu
	<baolu.lu@...ux.intel.com>, <iommu@...ts.linux.dev>,
	<linux-kernel@...r.kernel.org>, <netdev@...r.kernel.org>
CC: Joerg Roedel <joro@...tes.org>, Will Deacon <will@...nel.org>, "Robin
 Murphy" <robin.murphy@....com>, Heiner Kallweit <hkallweit1@...il.com>,
	<nic_swsd@...ltek.com>, "David S. Miller" <davem@...emloft.net>, Eric Dumazet
	<edumazet@...gle.com>, Jakub Kicinski <kuba@...nel.org>, Paolo Abeni
	<pabeni@...hat.com>, Marco Elver <elver@...gle.com>
Subject: Re: [PATCH v4 2/5] r8169: Coalesce RTL8411b PHY power-down recovery
 calls to reduce spinlock stalls



On 10/29/2023 4:04 AM, Mirsad Goran Todorovac wrote:
> On RTL8411b the RX unit gets confused if the PHY is powered-down.
> This was reported in [0] and confirmed by Realtek. Realtek provided
> a sequence to fix the RX unit after PHY wakeup.
> 
> A series of about 130 r8168_mac_ocp_write() calls is performed to
> program the RTL registers for recovery.
> 
> With about 130 of these sequential calls to r8168_mac_ocp_write() this looks like
> a lock storm that will stall all of the cores and CPUs on the same memory controller
> for certain time I/O takes to finish.
> 
> In a sequential case of RTL register programming, a sequence of writes to the RTL
> registers can be coalesced under a same raw spinlock. This can dramatically decrease
> the number of bus stalls in a multicore or multi-CPU system:
> 
>     static void rtl_hw_start_8411_2(struct rtl8169_private *tp)
>     {
> 
> 	...
> 
> 	/* The following Realtek-provided magic fixes an issue with the RX unit
> 	 * getting confused after the PHY having been powered-down.
> 	 */
> 
> 	static const struct recover_8411b_info init_zero_seq[] = {
> 	    { 0xFC28, 0x0000 }, { 0xFC2A, 0x0000 }, { 0xFC2C, 0x0000 },
> 	    ...
> 	};
> 
> 	static const struct recover_8411b_info recover_seq[] = {
> 	    { 0xF800, 0xE008 }, { 0xF802, 0xE00A }, { 0xF804, 0xE00C },
> 	    ...
> 	};
> 
> 	static const struct recover_8411b_info final_seq[] = {
> 	    { 0xFC2A, 0x0743 }, { 0xFC2C, 0x0801 }, { 0xFC2E, 0x0BE9 },
> 	    ...
> 	};
> 
> 	r8168_mac_ocp_write_seq(tp, init_zero_seq);
> 	mdelay(3);
> 	r8168_mac_ocp_write(tp, 0xFC26, 0x0000);
> 	r8168_mac_ocp_write_seq(tp, recover_seq);
> 	r8168_mac_ocp_write(tp, 0xFC26, 0x8000);
> 	r8168_mac_ocp_write_seq(tp, final_seq);
>     }
> 
> The hex data is preserved intact through s/r8168_mac_ocp_write[(]tp,/{ / and s/[)];/ },/
> functions that only changed the function names and the ending of the line, so the actual
> hex data is unchanged.
> 
> Note that the reason for the introduction of the original commit
> was to enable recovery of the RX unit on the RTL8411b which was confused by the
> powered-down PHY. This sequence of r8168_mac_ocp_write() calls amplifies the problem
> into a series of about 500+ memory bus locks, most waiting for the main MMIO memory
> read-modify-write under a LOCK. The memory barrier in RTL_W32 should suffice for
> the programming sequence to reach RTL NIC registers.
> 
> [0] https://bugzilla.redhat.com/show_bug.cgi?id=1692075
> 
> Fixes: fe4e8db0392a6 ("r8169: fix issue with confused RX unit after PHY power-down on RTL8411b")
> Cc: Heiner Kallweit <hkallweit1@...il.com>
> Cc: Marco Elver <elver@...gle.com>
> Cc: nic_swsd@...ltek.com
> Cc: "David S. Miller" <davem@...emloft.net>
> Cc: Eric Dumazet <edumazet@...gle.com>
> Cc: Jakub Kicinski <kuba@...nel.org>
> Cc: Paolo Abeni <pabeni@...hat.com>
> Cc: netdev@...r.kernel.org
> Cc: linux-kernel@...r.kernel.org
> Link: https://lore.kernel.org/lkml/20231028005153.2180411-1-mirsad.todorovac@alu.unizg.hr/
> Link: https://lore.kernel.org/lkml/20231028110459.2644926-1-mirsad.todorovac@alu.unizg.hr/
> Signed-off-by: Mirsad Goran Todorovac <mirsad.todorovac@....unizg.hr>
> ---

Reviewed-by: Jacob Keller <jacob.e.keller@...el.com>