linux-kernel - Re: [PATCH 2.6.30-rc4] r8169: avoid losing MSI interrupts

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <m1eir2djjl.fsf@fess.ebiederm.org>
Date:	Sun, 23 Aug 2009 19:37:02 -0700
From:	ebiederm@...ssion.com (Eric W. Biederman)
To:	David Dillow <dave@...dillows.org>
Cc:	Michael Riepe <michael.riepe@...glemail.com>,
	Michael Buesch <mb@...sch.de>,
	Francois Romieu <romieu@...zoreil.com>,
	Rui Santos <rsantos@...popie.com>,
	Michael Büker <m.bueker@...lin.de>,
	linux-kernel@...r.kernel.org, netdev@...r.kernel.org
Subject: Re: [PATCH 2.6.30-rc4] r8169: avoid losing MSI interrupts

David Dillow <dave@...dillows.org> writes:

> On Sat, 2009-08-22 at 05:07 -0700, Eric W. Biederman wrote:
>> ebiederm@...ssion.com (Eric W. Biederman) writes:
>> 
>> > David Dillow <dave@...dillows.org> writes:
>> >
>> >>
>> >> Re-looking at the code, I'd guess that some IRQ status line is getting
>> >> stuck high, but I don't see why -- we should acknowledge all outstanding
>> >> interrupts each time through the loop, whether we care about them or
>> >> not.
>> >>
>> >> Could reproduce a problem with the following patch applied, and send the
>> >> full dmesg, please?
>> >
>> > Here is what I get.
>> >
>> > r8169 screaming irq status 00000085 mask 0000ffff event 0000803f napi 0000001d
>> 
>> And now that the machine has come out of it, that was followed by:
>> Looks like the soft lockup did not manage to trigger in this case.
>
> I need some more context, please. What is the network load through this
> NIC when you have the issues? Light, heavy? Can you give me more details
> about the machine? A full dmesg from boot until this happens would help
> quite a bit. At a minimum it would help answer which version of the chip
> we're dealing with and what the machine it is in looks like.

dmesg attached.

What seems to reproduce the problem is a download of about a gigabyte.
Which this machine does every hour or two.  The switch and the upstream
server are all 10Gig. So at least in bursts I expect I am saturating
the network adapter with traffic coming in as fast as it can come.

Last night I reverted your patch and the machine seems to be happy
and not having problems since then.

> Can you reproduce this with pci=nomsi? I'm assuming it the chip running
> in MSI mode.

It is.

> Also, can you reproduce it when booting UP (or maxcpus=1)? I'm thinking
> about a race between rtl8169_interrupt() and rtl8169_poll(), but it
> isn't jumping out at me.
>
> Also, I'm having connectivity troubles this weekend, so my response may
> be spotty. :(

No problem.

I haven't wrapped my head around the device specific bits but I suspect
we are simply receiving more packets while the interrupt handler is
running.

Eric


View attachment "bs5-dmesg1.txt" of type "text/plain" (49419 bytes)