| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Tue, 12 Jan 2010 04:04:26 -0500 From: Mark Hounschell <dmarkh@....rr.com> To: "Pallipadi, Venkatesh" <venkatesh.pallipadi@...el.com> CC: "markh@...pro.net" <markh@...pro.net>, Andi Kleen <andi@...stfloor.org>, Linus Torvalds <torvalds@...ux-foundation.org>, Alain Knaff <alain@...ff.lu>, Linux Kernel Mailing List <linux-kernel@...r.kernel.org>, "fdutils@...tils.linux.lu" <fdutils@...tils.linux.lu>, "Li, Shaohua" <shaohua.li@...el.com>, Ingo Molnar <mingo@...e.hu> Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28 On 01/11/2010 07:19 PM, Pallipadi, Venkatesh wrote: > On Fri, 2010-01-08 at 09:42 -0800, Mark Hounschell wrote: >> On 12/23/2009 03:30 PM, Pallipadi, Venkatesh wrote: >> >>>>> Can you try this one line patch either on .28 or .32 (with /proc/interrupts >>>>> output). >>>>> This disables hpet2 and lapic timer should then be used on CPU 0. If things >>>>> work with this test patch, we will know that the failure is somehow related >>>>> to HPET usage in MSI mode. >>>>> >>>>> Thanks, >>>>> Venki >>>>> >>>>> Reduce the rating of percpu hpet timer >>>>> >>>>> Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@...el.com> >>>>> --- >>>>> arch/x86/kernel/hpet.c | 2 +- >>>>> 1 files changed, 1 insertions(+), 1 deletions(-) >>>>> >>>>> diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c >>>>> index cafb1c6..f89d17a 100644 >>>>> --- a/arch/x86/kernel/hpet.c >>>>> +++ b/arch/x86/kernel/hpet.c >>>>> @@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) >>>>> hpet_setup_irq(hdev); >>>>> evt->irq = hdev->irq; >>>>> >>>>> - evt->rating = 110; >>>>> + evt->rating = 40; >>>>> evt->features = CLOCK_EVT_FEAT_ONESHOT; >>>>> if (hdev->flags & HPET_DEV_PERI_CAP) >>>>> evt->features |= CLOCK_EVT_FEAT_PERIODIC; >>>> >>>> That made it work. Used 2.6.32.2 >>>> >>>> cat /proc/interrupts >>>> CPU0 CPU1 CPU2 CPU3 >>>> 0: 82 0 0 1 IO-APIC-edge timer >>>> 1: 0 0 0 67 IO-APIC-edge i8042 >>>> 3: 0 0 0 6 IO-APIC-edge >>>> 4: 0 0 0 4 IO-APIC-edge >>>> 6: 0 0 0 4 IO-APIC-edge floppy >>>> 8: 0 0 0 8 IO-APIC-edge rtc0 >>>> 9: 0 0 0 0 IO-APIC-fasteoi acpi >>>> 12: 0 0 10 1519 IO-APIC-edge i8042 >>>> 14: 0 0 39 10995 IO-APIC-edge >>>> pata_atiixp >>>> 15: 0 0 3 391 IO-APIC-edge >>>> pata_atiixp >>>> 16: 0 0 2 606 IO-APIC-fasteoi >>>> aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib >>>> 17: 0 0 0 3 IO-APIC-fasteoi >>>> ehci_hcd:usb1, parport0, ni-pci-gpib >>>> 18: 0 0 10 2168 IO-APIC-fasteoi >>>> ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia >>>> 19: 0 0 0 130 IO-APIC-fasteoi >>>> aic7xxx, ehci_hcd:usb2, ttySLG0, eth1 >>>> 22: 0 0 8 1151 IO-APIC-fasteoi ahci >>>> 24: 0 0 0 0 HPET_MSI-edge hpet2 >>>> 29: 0 0 0 48 PCI-MSI-edge >>>> sky2@pci:0000:04:00.0 >>>> NMI: 0 0 0 0 Non-maskable interrupts >>>> LOC: 34842 30177 29672 29632 Local timer interrupts >>>> SPU: 0 0 0 0 Spurious interrupts >>>> PMI: 0 0 0 0 Performance monitoring >>>> interrupts >>>> PND: 0 0 0 0 Performance pending work >>>> RES: 17501 20449 16670 11224 Rescheduling interrupts >>>> CAL: 10554 2336 1102 1071 Function call interrupts >>>> TLB: 364 562 753 468 TLB shootdowns >>>> ERR: 0 >>>> MIS: 0 >>>> >>>> >>>> # fdformat /dev/fd0u1440 >>>> Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. >>>> Formatting ... done >>>> Verifying ... done >>> >>> Hmmm.. Thats very interesting indeed. >>> >>> That clearly says that HPET MSI interrupts somehow is causing some >>> caching side effect in the chipset that results in this floppy dma >>> failure. >>> >>> Here's is what we have until now. >>> IRQ 0 is based on HPET legacy interrupt and HPET device is also capable >>> of MSI on this platform. So we also have a percpu hpet (hpet2 tied to >>> CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast >>> in cases where LAPIC timer will stop working in deep C-state. As we have >>> only one HPET channel free for percpu HPET, we only have hpet2 tied to >>> CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with >>> deep C-state. >>> >>> One problem here is that percpu hpet should only get used when LAPIC >>> cannot be used (that is when CPU enters deep C-state). Using hpet2 in >>> place of LAPIC timer even when deep C-state is not supported is not >>> right in terms of performance. We need some changes here to fix that >>> [Problem 1]. >>> >>> But, that still does not explain why we are seeing this problem in the >>> first place. I mean, using hpet2 is not optimal, but should not have >>> functionality issues like this. Even fixing [Problem 1] above, we may >>> see this problem on some other platform that supports deep C-state and >>> so has hpet2 enabled for a valid reason. >>> >>> Also, I am not sure whether the problem also happens if legacy HPET >>> interrupts are used during run time in place of LAPIC timer (May be >>> worth to try this with a simple test patch, let me think about it). In >>> this case, legacy HPET interrupt rightly goes quiet after boot, giving >>> priority to LAPIC timer. >>> >>> With hpet MSI interrupts, we do a write followed by read of HPET >>> memmapped register to set a HPET channel timeout + read of global HPET >>> timer. This happens on every timer interrupt on CPU 0. And we also have >>> MSI interrupt being delivered to CPU 0. I cannot think of any reason why >>> this can break dma. We can probably try adding some dummy HPET read >>> after dma write, to see if that flushes things properly. >>> >> >> Haven't seen any activity on this thread in a while. Just curious, are we >> still working this? >> Is there anything else I can do to help? > > Sorry for not following up on this. We have narrowed this down to HPET > MSI and floppy DMA. I still don't know how HPET MSI interrupts are > breaking floppy DMA. > > You are seeing the problem on two different systems. Correct? Do you > have any system where this works with HPET MSI enabled? > I see the problem on every system in which the HPET2 shows up in /proc/interrupts. The machines that work with HPET enabled don't show HPET at all in /proc/interrupts. I have some of each. All the boxes that fail here use the (AMD) 790x series chip sets. > Couple of options on how we can go about this one: > 1) Change the HPET-MSI change to not get activated when there are no > C-states with LAPIC stoppage involved. This will resolve the problem on > the systems you reported as there are no deep C-states. But, I fear that > with the actual problem unresolved, we may hit it in future with this or > some other platform having same issue with CPUs that support deep > C-state. > 2) Try this testcase on few other platforms that support HPET-MSI and > deep C-states and check how widespread the problem is and then add a > whitelist-blacklist for HPET MSI usage. > > I think, for 2.6.33 option 1 is better. Will work on that and send in > patches for you test. > OK, thanks Mark -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists