| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Date: Wed, 23 Dec 2009 12:30:43 -0800 From: "Pallipadi, Venkatesh" <venkatesh.pallipadi@...el.com> To: "markh@...pro.net" <markh@...pro.net> Cc: Andi Kleen <andi@...stfloor.org>, Linus Torvalds <torvalds@...ux-foundation.org>, "dmarkh@....rr.com" <dmarkh@....rr.com>, Alain Knaff <alain@...ff.lu>, Linux Kernel Mailing List <linux-kernel@...r.kernel.org>, "fdutils@...tils.linux.lu" <fdutils@...tils.linux.lu>, "Li, Shaohua" <shaohua.li@...el.com>, Ingo Molnar <mingo@...e.hu> Subject: Re: [Fdutils] DMA cache consistency bug introduced in 2.6.28 On Wed, 2009-12-23 at 11:35 -0800, Mark Hounschell wrote: > On 12/23/2009 02:18 PM, Pallipadi, Venkatesh wrote: > > On Wed, Dec 23, 2009 at 09:41:50AM -0800, Mark Hounschell wrote: > >> On 12/23/2009 11:38 AM, Andi Kleen wrote: > >>> Linus Torvalds <torvalds@...ux-foundation.org> writes: > >>> > >>>> It's not using the lapic for CPU0. > >>>> > >>>> Using the HPET as a per-cpu timer is some crazy sh*t, since it's pretty > >>>> expensive to reprogram (compared to the local apic). And having different > >>>> timers for different CPU's is just odd. > >>>> > >>>> The fact that the timer subsystem can do this and it all (mostly) works at > >>>> all is nice and impressive, but doesn't make it any less crazy ;) > >>> > >>> I suspect it's a system where the APIC timer stops in deeper idle > >>> states and it supports them. In this case CPU #0 does timer broadcasts > >>> when needed to wake the other CPUs up from deep C, but for that it has > >>> to run with HPET. At least the other ones can still enjoy the LAPIC > >>> timer. > >>> > >>> This might suggest that Mark's floppy controller doesn't like > >>> deep C? Mark, did you try booting with processor.max_cstate=1 > >>> and HPET enabled? > >> > >> I just did and /proc/interrupts looks the same and the floppy still does > >> not format. > >> > > > > Can you try this one line patch either on .28 or .32 (with /proc/interrupts > > output). > > This disables hpet2 and lapic timer should then be used on CPU 0. If things > > work with this test patch, we will know that the failure is somehow related > > to HPET usage in MSI mode. > > > > Thanks, > > Venki > > > > Reduce the rating of percpu hpet timer > > > > Signed-off-by: Venkatesh Pallipadi <venkatesh.pallipadi@...el.com> > > --- > > arch/x86/kernel/hpet.c | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > diff --git a/arch/x86/kernel/hpet.c b/arch/x86/kernel/hpet.c > > index cafb1c6..f89d17a 100644 > > --- a/arch/x86/kernel/hpet.c > > +++ b/arch/x86/kernel/hpet.c > > @@ -480,7 +480,7 @@ static void init_one_hpet_msi_clockevent(struct hpet_dev *hdev, int cpu) > > hpet_setup_irq(hdev); > > evt->irq = hdev->irq; > > > > - evt->rating = 110; > > + evt->rating = 40; > > evt->features = CLOCK_EVT_FEAT_ONESHOT; > > if (hdev->flags & HPET_DEV_PERI_CAP) > > evt->features |= CLOCK_EVT_FEAT_PERIODIC; > > That made it work. Used 2.6.32.2 > > cat /proc/interrupts > CPU0 CPU1 CPU2 CPU3 > 0: 82 0 0 1 IO-APIC-edge timer > 1: 0 0 0 67 IO-APIC-edge i8042 > 3: 0 0 0 6 IO-APIC-edge > 4: 0 0 0 4 IO-APIC-edge > 6: 0 0 0 4 IO-APIC-edge floppy > 8: 0 0 0 8 IO-APIC-edge rtc0 > 9: 0 0 0 0 IO-APIC-fasteoi acpi > 12: 0 0 10 1519 IO-APIC-edge i8042 > 14: 0 0 39 10995 IO-APIC-edge > pata_atiixp > 15: 0 0 3 391 IO-APIC-edge > pata_atiixp > 16: 0 0 2 606 IO-APIC-fasteoi > aic79xx, ohci_hcd:usb3, ohci_hcd:usb4, HDA Intel, Digi DBX2, ni-pci-gpib > 17: 0 0 0 3 IO-APIC-fasteoi > ehci_hcd:usb1, parport0, ni-pci-gpib > 18: 0 0 10 2168 IO-APIC-fasteoi > ohci_hcd:usb5, ohci_hcd:usb6, ohci_hcd:usb7, Digi DBX2, nvidia > 19: 0 0 0 130 IO-APIC-fasteoi > aic7xxx, ehci_hcd:usb2, ttySLG0, eth1 > 22: 0 0 8 1151 IO-APIC-fasteoi ahci > 24: 0 0 0 0 HPET_MSI-edge hpet2 > 29: 0 0 0 48 PCI-MSI-edge > sky2@pci:0000:04:00.0 > NMI: 0 0 0 0 Non-maskable interrupts > LOC: 34842 30177 29672 29632 Local timer interrupts > SPU: 0 0 0 0 Spurious interrupts > PMI: 0 0 0 0 Performance monitoring > interrupts > PND: 0 0 0 0 Performance pending work > RES: 17501 20449 16670 11224 Rescheduling interrupts > CAL: 10554 2336 1102 1071 Function call interrupts > TLB: 364 562 753 468 TLB shootdowns > ERR: 0 > MIS: 0 > > > # fdformat /dev/fd0u1440 > Double-sided, 80 tracks, 18 sec/track. Total capacity 1440 kB. > Formatting ... done > Verifying ... done Hmmm.. Thats very interesting indeed. That clearly says that HPET MSI interrupts somehow is causing some caching side effect in the chipset that results in this floppy dma failure. Here's is what we have until now. IRQ 0 is based on HPET legacy interrupt and HPET device is also capable of MSI on this platform. So we also have a percpu hpet (hpet2 tied to CPU0). percpu hpet was added to avoid the usage of IRQ0+LAPIC broadcast in cases where LAPIC timer will stop working in deep C-state. As we have only one HPET channel free for percpu HPET, we only have hpet2 tied to CPU 0 and other CPUs still have to go through IRQ0+LAPIC broadcast with deep C-state. One problem here is that percpu hpet should only get used when LAPIC cannot be used (that is when CPU enters deep C-state). Using hpet2 in place of LAPIC timer even when deep C-state is not supported is not right in terms of performance. We need some changes here to fix that [Problem 1]. But, that still does not explain why we are seeing this problem in the first place. I mean, using hpet2 is not optimal, but should not have functionality issues like this. Even fixing [Problem 1] above, we may see this problem on some other platform that supports deep C-state and so has hpet2 enabled for a valid reason. Also, I am not sure whether the problem also happens if legacy HPET interrupts are used during run time in place of LAPIC timer (May be worth to try this with a simple test patch, let me think about it). In this case, legacy HPET interrupt rightly goes quiet after boot, giving priority to LAPIC timer. With hpet MSI interrupts, we do a write followed by read of HPET memmapped register to set a HPET channel timeout + read of global HPET timer. This happens on every timer interrupt on CPU 0. And we also have MSI interrupt being delivered to CPU 0. I cannot think of any reason why this can break dma. We can probably try adding some dummy HPET read after dma write, to see if that flushes things properly. Thanks, Venki -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo@...r.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
Powered by blists - more mailing lists