linux-kernel - Re: Interrupts are being dropped under heavy IO on -rt .

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [day] [month] [year] [list]

Message-ID: <49F53F07.8020203@linux.vnet.ibm.com>
Date:	Sun, 26 Apr 2009 22:13:43 -0700
From:	jvrao <jvrao@...ux.vnet.ibm.com>
To:	Andrew Vasquez <andrew.vasquez@...gic.com>
CC:	LKML <linux-kernel@...r.kernel.org>,
	rt-users <linux-rt-users@...r.kernel.org>,
	Mike Anderson <andmike@...ux.vnet.ibm.com>,
	"Luis Claudio R. Goncalves" <lgoncalv@...hat.com>,
	Linux Driver <Linux-Driver@...gic.com>
Subject: Re: Interrupts are being dropped under heavy IO on -rt .

Hi Andrew, I ran the IO intensive tests with MSI interrupts enabled.


[root@...3c29 ~]# grep MSI /proc/interrupts 
2300:          0          2          0         36   PCI-MSI-edge      eth
2301:          0          0          0         27   PCI-MSI-edge      eth1
2302:         38      13965      15586   15402995   PCI-MSI-edge      qla2xxx
2303:         24      12116      13071   14253323   PCI-MSI-edge      qla2xxx


As per your request I enabled ql2xextended_error_logging, and also set the dev.scsi.logging_level to 4100 for additional info.

Please find the messages file attached.

Thanks,
JV

jvrao wrote:
> Andrew Vasquez wrote:
>> On Fri, 10 Apr 2009, jvrao wrote:
>>
>>> Qlogic driver uses MSI-EDGE interrupts by default.
>>>  
>>> On -rt kernel under heavy IO load we observed that the driver missed 
>>> interrupts occasionally.
>>>
>>> Missing interrupts caused spurious mailbox timeouts and path failovers.
>>>
>>> Suspecting MSI interrupts, we disabled MSI and forcing the driver to use 
>>> APIC interrupts (Patch attached).
>>>
>>> With this change, our IO tests ran for extended period of time without 
>>> any issues.
>>>
>>> It would be interesting to see if the MSI infrastructure itself has 
>>> issues on realtime, or it is the combination of qla2xxx/MSI.
>>>
>>> Right now we don't have any setup to test out MSI with other adapters. 
>>> Hence, wondering if anyone faced similar issues.
>> Could you perhaps provide some details on hardware used within your
>> testing -- HBA type, storage, etc?  Also, can you share any logs of
>> the failures with extended-error-logging enabled
>> (ql2xextended_error_logging)?
> 
> Hi, the following link provides more information on the logs and our analysis.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=466903
> 
> [root@...3c29 R2-SR1]# lspci | grep -i qlogic
> 03:05.0 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
> 03:05.1 Fibre Channel: QLogic Corp. ISP2422-based 4Gb Fibre Channel to PCI-X HBA (rev 02)
> 
> 
> Storage is DS4700 (Product ID: 1814)
> 
> If you need more information, please let me know, I can generate more information/logs.
> I will also find time to reproduce this with ql2xextended_error_logging enabled.
> 
> Thanks,
> JV
> 
>> Regards,
>> Andrew Vasquez
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
>> the body of a message to majordomo@...r.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
> the body of a message to majordomo@...r.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


View attachment "messages.withMSI" of type "text/plain" (198440 bytes)