netdev - Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on Meteor Lake

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <aB1JnJG_CH5vxAsw@mail-itl>
Date: Fri, 9 May 2025 02:17:32 +0200
From: Marek Marczykowski-Górecki <marmarek@...isiblethingslab.com>
To: Paul Menzel <pmenzel@...gen.mpg.de>
Cc: Vitaly Lifshits <vitaly.lifshits@...el.com>,
	Tony Nguyen <anthony.l.nguyen@...el.com>,
	Przemek Kitszel <przemyslaw.kitszel@...el.com>,
	netdev@...r.kernel.org, intel-wired-lan@...ts.osuosl.org,
	regressions@...ts.linux.dev, stable@...r.kernel.org,
	Sasha Levin <sashal@...nel.org>
Subject: Re: [Intel-wired-lan] [REGRESSION] e1000e heavy packet loss on
 Meteor Lake - 6.14.2

On Fri, May 09, 2025 at 01:28:36AM +0200, Marek Marczykowski-Górecki wrote:
> On Fri, May 09, 2025 at 01:13:28AM +0200, Paul Menzel wrote:
> > Dear Marek, dear Vitaly,
> > 
> > 
> > Am 09.05.25 um 00:41 schrieb Marek Marczykowski-Górecki:
> > > On Thu, May 08, 2025 at 09:26:18AM +0300, Lifshits, Vitaly
> > > > On 4/21/2025 4:28 PM, Marek Marczykowski-Górecki wrote:
> > > > > On Mon, Apr 21, 2025 at 03:19:12PM +0200, Marek Marczykowski-Górecki wrote:
> > > > > > On Mon, Apr 21, 2025 at 03:44:02PM +0300, Lifshits, Vitaly wrote:
> > > > > > > 
> > > > > > > 
> > > > > > > On 4/16/2025 3:43 PM, Marek Marczykowski-Górecki wrote:
> > > > > > > > On Wed, Apr 16, 2025 at 03:09:39PM +0300, Lifshits, Vitaly wrote:
> > > > > > > > > Can you please also share the output of ethtool -i? I would like to know the
> > > > > > > > > NVM version that you have on your device.
> > > > > > > > 
> > > > > > > > driver: e1000e
> > > > > > > > version: 6.14.1+
> > > > > > > > firmware-version: 1.1-4
> > > > > > > > expansion-rom-version:
> > > > > > > > bus-info: 0000:00:1f.6
> > > > > > > > supports-statistics: yes
> > > > > > > > supports-test: yes
> > > > > > > > supports-eeprom-access: yes
> > > > > > > > supports-register-dump: yes
> > > > > > > > supports-priv-flags: yes
> > > > > > > > 
> > > > > > > 
> > > > > > > Your firmware version is not the latest, can you check with the board
> > > > > > > manufacturer if there is a BIOS update to your system?
> > > > > > 
> > > > > > I can check, but still, it's a regression in the Linux driver - old
> > > > > > kernel did work perfectly well on this hw. Maybe new driver tries to use
> > > > > > some feature that is missing (or broken) in the old firmware?
> > > > > 
> > > > > A little bit of context: I'm maintaining the kernel package for a Qubes
> > > > > OS distribution. While I can try to update firmware on my test system, I
> > > > > have no influence on what hardware users will use this kernel, and
> > > > > which firmware version they will use (and whether all the vendors
> > > > > provide newer firmware at all). I cannot ship a kernel that is known
> > > > > to break network on some devices.
> > > > > 
> > > > > > > Also, you mentioned that on another system this issue doesn't reproduce, do
> > > > > > > they have the same firmware version?
> > > > > > 
> > > > > > The other one has also 1.1-4 firmware. And I re-checked, e1000e from
> > > > > > 6.14.2 works fine there.
> > 
> > > > Thank you for your detailed feedback and for providing the requested
> > > > information.
> > > > 
> > > > We have conducted extensive testing of this patch across multiple systems
> > > > and have not observed any packet loss issues. Upon comparing the mentioned
> > > > setups, we noted that while the LAN controller is similar, the CPU differs.
> > > > We believe that the issue may be related to transitions in the CPU's low
> > > > power states.
> > > > 
> > > > Consequently, we kindly request that you disable the CPU low power state
> > > > transitions in the S0 system state and verify if the issue persists. You can
> > > > disable this in the kernel parameters on the command line with idle=poll.
> > > > Please note that this command is intended for debugging purposes only, as it
> > > > may result in higher power consumption.
> > > 
> > > I tried with idle=poll, and it didn't help, I still see a lot of packet
> > > losses. But I can also confirm that idle=poll makes the system use
> > > significantly more power (previously at 25-30W, with this option stays
> > > at about 42W).
> > > 
> > > Is there any other info I can provide, enable some debug features or
> > > something?
> > > 
> > > I see the problem is with receiving packets - in my simple ping test,
> > > the ping target sees all the echo requests (and respond to them), but
> > > the responses aren't reaching ping back (and are not visible on tcpdump
> > > on the problematic system either).
> > 
> > As the cause is still unclear, can the commit please be reverted in the
> > master branch due adhere to Linux’ no-regression policy, so that it can be
> > reverted from the stable series?
> > 
> > Marek, did you also test 6.15 release candidates?
> 
> The last test I did was on 6.15-rc3. I can re-test on -rc5.

Same with 6.15-rc5.

-- 
Best Regards,
Marek Marczykowski-Górecki
Invisible Things Lab

Download attachment "signature.asc" of type "application/pgp-signature" (489 bytes)