Message-ID: <d0b0d0e2-dcb8-5f3b-7a0c-9eeba13952a6@intel.com>
Date: Thu, 22 Aug 2024 12:18:37 +0300
From: Vitaly Lifshits <vitaly.lifshits@...el.com>
To: Petr Valenta <petr@...klidu.cz>, Bjorn Helgaas <helgaas@...nel.org>, "Dima
 Ruinskiy" <dima.ruinskiy@...el.com>, Hui Wang <hui.wang@...onical.com>
CC: Jiri Slaby <jirislaby@...nel.org>, Bjorn Helgaas <bhelgaas@...gle.com>,
	Len Brown <lenb@...nel.org>, "linux-acpi@...r.kernel.org"
	<linux-acpi@...r.kernel.org>, Linux kernel mailing list
	<linux-kernel@...r.kernel.org>, Linux regressions mailing list
	<regressions@...ts.linux.dev>, Tony Nguyen <anthony.l.nguyen@...el.com>,
	<przemyslaw.kitszel@...el.com>, <intel-wired-lan@...ts.osuosl.org>, "Rafael
 J. Wysocki" <rafael@...nel.org>
Subject: Re: ACPI IRQ storm with 6.10


On 8/22/2024 11:33 AM, Petr Valenta wrote:
>
>
> Dne 22. 08. 24 v 9:44 Petr Valenta napsal(a):
>>
>>
>> Dne 21. 08. 24 v 17:17 Vitaly Lifshits napsal(a):
>>>
>>> On 8/21/2024 5:59 PM, Bjorn Helgaas wrote:
>>>> [+to Dima, Vitaly, Hui; beginning of thread at
>>>> https://lore.kernel.org/r/60ac8988-ace4-4cf0-8c44-028ca741c0a1@kernel.org] 
>>>>
>>>>
>>>> On Wed, Aug 21, 2024 at 01:39:11PM +0200, Petr Valenta wrote:
>>>>> Dne 20. 08. 24 v 23:30 Bjorn Helgaas napsal(a):
>>>>>> On Tue, Aug 20, 2024 at 11:13:54PM +0200, Petr Valenta wrote:
>>>>>>> Dne 20. 08. 24 v 20:09 Bjorn Helgaas napsal(a):
>>>>>>>> On Mon, Aug 19, 2024 at 07:23:42AM +0200, Jiri Slaby wrote:
>>>>>>>>> On 19. 08. 24, 6:50, Jiri Slaby wrote:
>>>>>>>>>> CC e1000e guys + Jesse (due to 75a3f93b5383) + Bjorn (due to 
>>>>>>>>>> b2c289415b2b)
>>>>> ...
>>>>>> I'm at a loss.  You could try reverting the entire b2c289415b2b 
>>>>>> commit
>>>>>> (patch for that is below).
>>>>> This patch didn't help, so I reverted it back.
>>>>>
>>>>>> If that doesn't help, I guess you could try reverting the other
>>>>>> commits Jiri mentioned:
>>>>>>
>>>>>>     76a0a3f9cc2f e1000e: fix force smbus during suspend flow
>>>>>>     c93a6f62cb1b e1000e: Fix S0ix residency on corporate systems
>>>>>>     bfd546a552e1 e1000e: move force SMBUS near the end of 
>>>>>> enable_ulp function
>>>>>>     6918107e2540 net: e1000e & ixgbe: Remove PCI_HEADER_TYPE_MFD 
>>>>>> duplicates
>>>>>>     1eb2cded45b3 net: annotate writes on dev->mtu from 
>>>>>> ndo_change_mtu()
>>>>>>     b2c289415b2b e1000e: Remove redundant runtime resume for 
>>>>>> ethtool_ops
>>>>>>     75a3f93b5383 net: intel: implement modern PM ops declarations
>>>>>>
>>>>>> If you do this, I would revert 76a0a3f9cc2f, test, then revert
>>>>>> c93a6f62cb1b in addition, test, then revert bfd546a552e1 in 
>>>>>> addition,
>>>>>> etc.
>>>>> I have created revert patches like this:
>>>>> git format-patch --stdout -1 76a0a3f9cc2f | interdiff -q /dev/stdin \
>>>>> /dev/null > revert_76a0a3f9cc2f.patch
>>>>>
>>>>> I have applied revert_76a0a3f9cc2f.patch (rebuild and tested), 
>>>>> then in
>>>>> addition revert_c93a6f62cb1b.patch and after applying
>>>>> revert_bfd546a552e1.patch irq storm didn't appear.
>>>>>
>>>>> I have tested it with 3 subsequent reboots and in all those cases 
>>>>> it was ok.
>>>> Thanks for all this testing.  It sounds like reverting all three of
>>>> 76a0a3f9cc2f, c93a6f62cb1b, and bfd546a552e1 fixed the IRQ storm, but
>>>> I'm not clear on the results of other situations.
>>>>
>>>> It looks like c93a6f62cb1b could be reverted by itself because it's
>>>> unrelated to 76a0a3f9cc2f and bfd546a552e1.  I added the authors of
>>>> all three in case they have any insights.
>>>>
>>>> Bjorn
>>>
>>>
>>> I doubt that it is related to c93a6f62cb1b, I believe that is more 
>>> probable to be related to the two other patches.
>>>
>>> Apart from what I suggested in the other mailing thread (enabling 
>>> e1000e debug and to test if it happens with a cable connected),
>>>
>>> I suggest to try to apply this patch and see if it fixes the issue:
>>>
>>> https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20240806132348.880744-1-vitaly.lifshits@intel.com/ 
>>>
>>
>> I have applied patch from link above and command bellow really 
>> doesn't start irq storm.
>>
>> echo 'auto' > /sys/bus/pci/devices/0000:00:1f.6/power/control
>>
>> Problem is that after executing this command and plugging cable to 
>> ethernet port, kernel is not able to detect link (LED indicate link 
>> is on) so network over cable is not working.
>>
>>>
>>>
>>
>>  From mboxrd@z Thu Jan  1 00:00:00 1970
>> Return-Path: <intel-wired-lan-bounces@...osl.org>
>> X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on
>>      aws-us-west-2-korg-lkml-1.web.codeaurora.org
>> Received: from smtp2.osuosl.org (smtp2.osuosl.org [140.211.166.133])
>>      (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 
>> bits))
>>      (No client certificate requested)
>>      by smtp.lore.kernel.org (Postfix) with ESMTPS id 7319CC531DF
>>      for <intel-wired-lan@...hiver.kernel.org>; Thu, 22 Aug 2024 
>> 07:44:59 +0000 (UTC)
>> Received: from localhost (localhost [127.0.0.1])
>>      by smtp2.osuosl.org (Postfix) with ESMTP id 2EE99404B8;
>>      Thu, 22 Aug 2024 07:44:59 +0000 (UTC)
>> X-Virus-Scanned: amavis at osuosl.org
>> Received: from smtp2.osuosl.org ([127.0.0.1])
>> by localhost (smtp2.osuosl.org [127.0.0.1]) (amavis, port 10024) with 
>> ESMTP
>> id VRgkrPDlq_WW; Thu, 22 Aug 2024 07:44:56 +0000 (UTC)
>> X-Comment: SPF check N/A for local connections - 
>> client-ip=140.211.166.34; helo=ash.osuosl.org; 
>> envelope-from=intel-wired-lan-bounces@...osl.org; receiver=<UNKNOWN> 
>> DKIM-Filter: OpenDKIM Filter v2.11.0 smtp2.osuosl.org 53F64405BA
>> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=osuosl.org;
>>      s=default; t=1724312696;
>>      bh=y3v3IIFARTszfLWu7n/j8Du29EOi4VTxMDP3GF4qp7E=;
>>      h=Date:To:References:From:In-Reply-To:Subject:List-Id:
>> List-Unsubscribe:List-Archive:List-Post:List-Help:List-Subscribe:
>>       Cc:From;
>>      b=ZIudOsHGSDoQvtekseiE4SUpOKofnvHlxj7aT3f7bLvqCDMOCfygsO6tctN23YgSh
>> xYqnq4yBSB4/JQ4v7Juyg0P/wqTcr+XFqhORTc2qBku9GCA+Y4wRKbRUeH4/AUNthL
>> cf/zG7uEOFEKz4YALwviQFqR5E+HW9gD+YnXahtGUVqYiTjB01HuESDZdYI5huiCLI
>> eHnQDw/SSwM1YmkjLzQgICjlxtIRVYjUL+shaltRg9f7t4otZa9bvrvLptzw5Mrfc0
>> GLvrNRmHckPFKEJOXgmIeQI40IOHckD3MX2dkQ2dQ0VCrkl9JIgtuSRuS3IpB1dr65
>>       TatTrq9Onm26w==
>> Received: from ash.osuosl.org (ash.osuosl.org [140.211.166.34])
>>      by smtp2.osuosl.org (Postfix) with ESMTP id 53F64405BA;
>>      Thu, 22 Aug 2024 07:44:56 +0000 (UTC)
>> Received: from smtp1.osuosl.org (smtp1.osuosl.org [140.211.166.138])
>> by ash.osuosl.org (Postfix) with ESMTP id 81E351BF322
>> for <intel-wired-lan@...ts.osuosl.org>; Thu, 22 Aug 2024 07:44:54 
>> +0000 (UTC)
>> Received: from localhost (localhost [127.0.0.1])
>> by smtp1.osuosl.org (Postfix) with ESMTP id 79A0C80A82
>> for <intel-wired-lan@...ts.osuosl.org>; Thu, 22 Aug 2024 07:44:54 
>> +0000 (UTC)
>> X-Virus-Scanned: amavis at osuosl.org
>> Received: from smtp1.osuosl.org ([127.0.0.1])
>> by localhost (smtp1.osuosl.org [127.0.0.1]) (amavis, port 10024) with 
>> ESMTP
>> id m9sJJpu9kR7y for <intel-wired-lan@...ts.osuosl.org>;
>> Thu, 22 Aug 2024 07:44:53 +0000 (UTC)
>> Received-SPF: Pass (mailfrom) identity=mailfrom; 
>> client-ip=136.143.188.52;
>> helo=sender4-of-o52.zoho.com; envelope-from=petr@...klidu.cz;
>> receiver=<UNKNOWN> DMARC-Filter: OpenDMARC Filter v1.4.2 
>> smtp1.osuosl.org 3674B80A59
>> DKIM-Filter: OpenDKIM Filter v2.11.0 smtp1.osuosl.org 3674B80A59
>> Received: from sender4-of-o52.zoho.com (sender4-of-o52.zoho.com
>> [136.143.188.52])
>> by smtp1.osuosl.org (Postfix) with ESMTPS id 3674B80A59
>> for <intel-wired-lan@...ts.osuosl.org>; Thu, 22 Aug 2024 07:44:51 
>> +0000 (UTC)
>> ARC-Seal: i=1; a=rsa-sha256; t=1724312671; cv=none; d=zohomail.com; 
>> s=zohoarc; 
>> b=B0wnUG3UHEcTRfbjC9HSfLJG+WBnpU18yag7r0240QuMQMnP/cHcj9e4oJU2FgxRPLpt6OGnlZOiPNE2GUFgnkBzKBPwzxb7eTHFwW4P8cW+1IrIOQ6jZWd2rhOIyWcRKYMydfCbMPM04Z+RwKVyRlrLTYL5UDBYYKKHOG08Ikc=
>> ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; 
>> d=zohomail.com;
>> s=zohoarc; t=1724312671;
>> h=Content-Type:Content-Transfer-Encoding:Cc:Cc:Date:Date:From:From:In-Reply-To:MIME-Version:Message-ID:References:Subject:Subject:To:To:Message-Id:Reply-To; 
>>
>> bh=y3v3IIFARTszfLWu7n/j8Du29EOi4VTxMDP3GF4qp7E=; 
>> b=VvZAc/xKVy85rZpNNCAwUwpxquk4r4Xw2QjZmePGlnINwOvJf6oilR9lqx2WDMezV20iKTW9f3dauO4jIjp363HOdh7P21UFfa66a0oK63RODo7IQMHSCqaCwAEoO1PKHfDfTMwz0/BShU1dt+nhtAeSeKwbG7G1qizCcoXTdjo=
>> ARC-Authentication-Results: i=1; mx.zohomail.com;
>> dkim=pass  header.i=jevklidu.cz;
>> spf=pass  smtp.mailfrom=petr@...klidu.cz;
>> dmarc=pass header.from=<petr@...klidu.cz>
>> Received: by mx.zohomail.com with SMTPS id 
>> 1724312669862808.3168476405893;
>> Thu, 22 Aug 2024 00:44:29 -0700 (PDT)
>> Message-ID: <5ba3c7c2-5695-421d-a747-2a23af48db26@...klidu.cz>
>> Date: Thu, 22 Aug 2024 09:44:22 +0200
>> MIME-Version: 1.0
>> User-Agent: Mozilla Thunderbird
>> To: Vitaly Lifshits <vitaly.lifshits@...el.com>,
>> Bjorn Helgaas <helgaas@...nel.org>, Dima Ruinskiy 
>> <dima.ruinskiy@...el.com>,
>> Hui Wang <hui.wang@...onical.com>
>> References: <20240821145959.GA248604@...lgaas>
>> <1041b9b5-cc78-13b1-459a-d1d3a313475a@...el.com>
>> Content-Language: cs-CZ, en-US
>> From: Petr Valenta <petr@...klidu.cz>
>> In-Reply-To: <1041b9b5-cc78-13b1-459a-d1d3a313475a@...el.com>
>> Content-Type: text/plain; charset=UTF-8; format=flowed
>> Content-Transfer-Encoding: 7bit
>> X-ZohoMailClient: External
>> X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt;
>> c=relaxed/relaxed; t=1724312671; s=zoho; d=jevklidu.cz; 
>> i=petr@...klidu.cz;
>> h=Message-ID:Date:Date:MIME-Version:Subject:Subject:To:To:Cc:Cc:References:From:From:In-Reply-To:Content-Type:Content-Transfer-Encoding:Message-Id:Reply-To; 
>>
>> bh=y3v3IIFARTszfLWu7n/j8Du29EOi4VTxMDP3GF4qp7E=;
>> b=RSnIQpqoQp2O3ExJNnw4fk9dlt8CX1T5sbtB6GflBDYejiRQJTcrU3zRHn3pRkFq
>> tm00/cgXr6pF6T5vJFttBkfrHtnRiPiE8cjqni5KsNxCyOXOwri6I5ARAmPcUj42eda
>> e/xHQX9E3ayXrWSBQDAsun3Ann63tcXQKwlT7ffI=
>> X-Mailman-Original-Authentication-Results: smtp1.osuosl.org;
>> dmarc=none (p=none dis=none)
>> header.from=jevklidu.cz
>> X-Mailman-Original-Authentication-Results: smtp1.osuosl.org;
>> dkim=pass (1024-bit key,
>> unprotected) header.d=jevklidu.cz header.i=petr@...klidu.cz
>> header.a=rsa-sha256 header.s=zoho header.b=RSnIQpqo
>> Subject: Re: [Intel-wired-lan] ACPI IRQ storm with 6.10
>> X-BeenThere: intel-wired-lan@...osl.org
>> X-Mailman-Version: 2.1.29
>> Precedence: list
>> List-Id: Intel Wired Ethernet Linux Kernel Driver Development
>> <intel-wired-lan.osuosl.org>
>> List-Unsubscribe: 
>> <https://lists.osuosl.org/mailman/options/intel-wired-lan>, 
>> <mailto:intel-wired-lan-request@...osl.org?subject=unsubscribe>
>> List-Archive: <http://lists.osuosl.org/pipermail/intel-wired-lan/>
>> List-Post: <mailto:intel-wired-lan@...osl.org>
>> List-Help: <mailto:intel-wired-lan-request@...osl.org?subject=help>
>> List-Subscribe: 
>> <https://lists.osuosl.org/mailman/listinfo/intel-wired-lan>,
>> <mailto:intel-wired-lan-request@...osl.org?subject=subscribe>
>> Cc: Linux regressions mailing list <regressions@...ts.linux.dev>,
>> "Rafael J. Wysocki" <rafael@...nel.org>, przemyslaw.kitszel@...el.com,
>> Linux kernel mailing list <linux-kernel@...r.kernel.org>,
>> "linux-acpi@...r.kernel.org" <linux-acpi@...r.kernel.org>,
>> Tony Nguyen <anthony.l.nguyen@...el.com>, Bjorn Helgaas 
>> <bhelgaas@...gle.com>,
>> intel-wired-lan@...ts.osuosl.org, Jiri Slaby <jirislaby@...nel.org>,
>> Len Brown <lenb@...nel.org>
>> Errors-To: intel-wired-lan-bounces@...osl.org
>> Sender: "Intel-wired-lan" <intel-wired-lan-bounces@...osl.org>
>>
>>
>>
>> Dne 21. 08. 24 v 17:17 Vitaly Lifshits napsal(a):
>>>
>>> On 8/21/2024 5:59 PM, Bjorn Helgaas wrote:
>>>> [+to Dima, Vitaly, Hui; beginning of thread at
>>>> https://lore.kernel.org/r/60ac8988-ace4-4cf0-8c44-028ca741c0a1@kernel.org] 
>>>>
>>>>
>>>> On Wed, Aug 21, 2024 at 01:39:11PM +0200, Petr Valenta wrote:
>>>>> Dne 20. 08. 24 v 23:30 Bjorn Helgaas napsal(a):
>>>>>> On Tue, Aug 20, 2024 at 11:13:54PM +0200, Petr Valenta wrote:
>>>>>>> Dne 20. 08. 24 v 20:09 Bjorn Helgaas napsal(a):
>>>>>>>> On Mon, Aug 19, 2024 at 07:23:42AM +0200, Jiri Slaby wrote:
>>>>>>>>> On 19. 08. 24, 6:50, Jiri Slaby wrote:
>>>>>>>>>> CC e1000e guys + Jesse (due to 75a3f93b5383) + Bjorn (due to 
>>>>>>>>>> b2c289415b2b)
>>>>> ...
>>>>>> I'm at a loss.  You could try reverting the entire b2c289415b2b 
>>>>>> commit
>>>>>> (patch for that is below).
>>>>> This patch didn't help, so I reverted it back.
>>>>>
>>>>>> If that doesn't help, I guess you could try reverting the other
>>>>>> commits Jiri mentioned:
>>>>>>
>>>>>>     76a0a3f9cc2f e1000e: fix force smbus during suspend flow
>>>>>>     c93a6f62cb1b e1000e: Fix S0ix residency on corporate systems
>>>>>>     bfd546a552e1 e1000e: move force SMBUS near the end of 
>>>>>> enable_ulp function
>>>>>>     6918107e2540 net: e1000e & ixgbe: Remove PCI_HEADER_TYPE_MFD 
>>>>>> duplicates
>>>>>>     1eb2cded45b3 net: annotate writes on dev->mtu from 
>>>>>> ndo_change_mtu()
>>>>>>     b2c289415b2b e1000e: Remove redundant runtime resume for 
>>>>>> ethtool_ops
>>>>>>     75a3f93b5383 net: intel: implement modern PM ops declarations
>>>>>>
>>>>>> If you do this, I would revert 76a0a3f9cc2f, test, then revert
>>>>>> c93a6f62cb1b in addition, test, then revert bfd546a552e1 in 
>>>>>> addition,
>>>>>> etc.
>>>>> I have created revert patches like this:
>>>>> git format-patch --stdout -1 76a0a3f9cc2f | interdiff -q /dev/stdin \
>>>>> /dev/null > revert_76a0a3f9cc2f.patch
>>>>>
>>>>> I have applied revert_76a0a3f9cc2f.patch (rebuild and tested), 
>>>>> then in
>>>>> addition revert_c93a6f62cb1b.patch and after applying
>>>>> revert_bfd546a552e1.patch irq storm didn't appear.
>>>>>
>>>>> I have tested it with 3 subsequent reboots and in all those cases 
>>>>> it was ok.
>>>> Thanks for all this testing.  It sounds like reverting all three of
>>>> 76a0a3f9cc2f, c93a6f62cb1b, and bfd546a552e1 fixed the IRQ storm, but
>>>> I'm not clear on the results of other situations.
>>>>
>>>> It looks like c93a6f62cb1b could be reverted by itself because it's
>>>> unrelated to 76a0a3f9cc2f and bfd546a552e1.  I added the authors of
>>>> all three in case they have any insights.
>>>>
>>>> Bjorn
>>>
>>>
>>> I doubt that it is related to c93a6f62cb1b, I believe that is more 
>>> probable to be related to the two other patches.
>>>
>>> Apart from what I suggested in the other mailing thread (enabling 
>>> e1000e debug and to test if it happens with a cable connected),
>>>
>>> I suggest to try to apply this patch and see if it fixes the issue:
>>>
>>> https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20240806132348.880744-1-vitaly.lifshits@intel.com/ 
>>>
>>
>> I have applied patch from link above and command bellow really 
>> doesn't start irq storm.
>>
>> echo 'auto' > /sys/bus/pci/devices/0000:00:1f.6/power/control
>>
>> Problem is that after executing this command and plugging cable to 
>> ethernet port, kernel is not able to detect link (LED indicate link 
>> is on) so network over cable is not working.
>>
>
> I have tested now how it behaves with kernel 6.9.9. There is a new 
> finding. After running "echo 'auto' > 
> /sys/bus/pci/devices/0000:00:1f.6/power/control" network over cable 
> works but irq storm arrises. I have never tested this before because I 
> don't use cable with this laptop at all. After unplugging cable irq 
> storm is gone.
>
> A possible workaround would be to turn off power control for the 
> e1000e at the kernel level (if is it possible) so that utilities like 
> powertop don't cause irq storm or broken network.
>
I would like to suggest the following for now as a triage:

1. revert the following patches:

    i.  e1000e: fix force smbus during suspend flow (76a0a3f9cc2f)

    ii.   e1000e: move force SMBUS near the end of enable_ulp function 
(bfd546a552e1)

    iii. "e1000e: move force SMBUS from enable ulp function to avoid PHY 
loss issue ( 861e8086029e)


2. apply this patch:
https://patchwork.ozlabs.org/project/intel-wired-lan/patch/20240806132348.880744-1-vitaly.lifshits@intel.com/ 



I expect this configuration not to have neither the IRQ storm nor the 
link indication issue (>"Problem is that after executing this command 
and plugging cable to ethernet port, kernel is not able to detect link 
(LED indicate link is on) so network over cable is not working.")

>>>
>>>
>>

Powered by blists - more mailing lists