| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <4AAA5B1F.3020103@compro.net>
Date: Fri, 11 Sep 2009 10:13:51 -0400
From: Mark Hounschell <markh@...pro.net>
To: linux-pci@...r.kernel.org
CC: Mark Hounschell <dmarkh@....rr.com>, linux-kernel@...r.kernel.org
Subject: problems doing direct dma from a pci device to pci-e device
I know this is really just a pci issue but have CC'd LKML just in case. Sorry LKML for the noise.
I have a situation where a card on a regular PCI bus (Compro gpiohsd) does dma's
directly into another PCI cards memory (VMIC-5565 reflective memory) living on
another PCI bus. These 2 cards are sometimes seperated by many bridges.
Expansion racks etc. We've been doing this forever. No problem (mostly).
I now have an AM3 based DFI DK 790FXB-M3H5 motherboard. This board has 3 regular
PCI slots and 3 PCI-E (16x) slots. I also have a PCI-E (x4) version of the VMIC-5565
reflective memory card in one of the PCI-E slots and our gpiohsd card in one of the regular
PCI slots. All on the motherboard. No expansion slots being used. However I cannot get
data from our gpiohsd into the PCI-E VMIC-5565 cards memory. I can certainly get the data there
from a userland buffer, no problem. Just not from one card to the other directly. Oh and when
I put the regular PCI version of the VMIC into one of the regular PCI slots everything works
as expected. They are then both on the same PCI bus and no bridges are involved though.
Here is some information about my system configuration. I would very much appreciate any
help or pointers as to why this doesn't work. I'm no PCI expert but do have a basic understanding
of it and I don't see anything wrong myself. I'm using a vanilla 2.6.26.8 kernel BTW and all
drivers involved are GPL.
Attached is a complete lspci -vvv
Short version:
# lspci -tv
-[0000:00]-+-00.0 ATI Technologies Inc RD790 Northbridge only dual slot PCI-e_GFX and HT3 K8 part
------> +-02.0-[0000:01-02]----00.0-[0000:02]----0c.0 VMIC Device 5565
| +-04.0-[0000:03-04]----00.0-[0000:04]--
| +-09.0-[0000:05]----00.0 Marvell Technology Group Ltd. 88E8056 PCI-E Gigabit Ethernet Controller
| +-0b.0-[0000:06]----00.0 nVidia Corporation G70 [GeForce 7800 GT]
| +-11.0 ATI Technologies Inc SB700/SB800 SATA Controller [IDE mode]
| +-14.0 ATI Technologies Inc SBx00 SMBus Controller
| +-14.1 ATI Technologies Inc SB700/SB800 IDE Controller
| +-14.2 ATI Technologies Inc SBx00 Azalia
| +-14.3 ATI Technologies Inc SB700/SB800 LPC host controller
| +-14.4-[0000:07]--+-05.0 Compro Computer Services, Inc. Device 4610
--------< | \-07.0 PLX Technology, Inc. Device 0480
+-18.0 Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] HyperTransport Configuration
+-18.1 Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Address Map
+-18.2 Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] DRAM Controller
+-18.3 Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Miscellaneous Control
\-18.4 Advanced Micro Devices [AMD] Family 10h [Opteron, Athlon64, Sempron] Link Control
And also the lspci -vvv info on just the cards and bridges involved.
Motherboard bridge that the VMIC card with it's internal bridge is attached to:
00:02.0 PCI bridge: ATI Technologies Inc RD790 PCI to PCI bridge (external gfx0 port A) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Bus: primary=00, secondary=01, subordinate=02, sec-latency=0
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: b0000000-cfffffff
Prefetchable memory behind bridge: 00000000fde00000-00000000fdefffff
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [50] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [58] Express (v2) Root Port (Slot-), MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag+ RBE+ FLReset-
DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported+
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
ClockPM- Suprise- LLActRep+ BwNot+
LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive+ BWMgmt+ ABWMgmt-
RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
RootCap: CRSVisible-
RootSta: PME ReqID 0000, PMEStatus- PMEPending-
Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable+
Address: fee0300c Data: 41c9
Capabilities: [b0] Subsystem: ATI Technologies Inc Device 5956
Capabilities: [b8] HyperTransport: MSI Mapping Enable+ Fixed+
Capabilities: [100] Vendor Specific Information <?>
Capabilities: [110] Virtual Channel <?>
Kernel driver in use: pcieport-driver
Kernel modules: shpchp
VMIC card with internal bridge:
01:00.0 PCI bridge: Intel Corporation Device 032d (rev 09) (prog-if 00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 32 bytes
Bus: primary=01, secondary=02, subordinate=02, sec-latency=64
I/O behind bridge: 0000c000-0000cfff
Memory behind bridge: b0000000-cfffffff
Prefetchable memory behind bridge: 00000000fde00000-00000000fdefffff
Secondary status: 66MHz+ FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: [44] Express (v1) PCI/PCI-X Bridge, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE- FLReset-
DevCtl: Report errors: Correctable- Non-Fatal- Fatal+ Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- BrConfRtry-
MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 2.5GT/s, Width x8, ASPM L0s, Latency L0 <256ns, L1 unlimited
ClockPM- Suprise- LLActRep- BwNot-
LnkCtl: ASPM Disabled; Disabled- Retrain- CommClk+
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s, Width x4, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
Capabilities: [5c] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0 Enable-
Address: 0000000000000000 Data: 0000
Capabilities: [6c] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [d8] PCI-X bridge device
Secondary Status: 64bit+ 133MHz+ SCD- USC- SCO- SRD- Freq=conv
Status: Dev=01:00.0 64bit- 133MHz- SCD- USC- SCO- SRD-
Upstream: Capacity=65535 CommitmentLimit=65535
Downstream: Capacity=65535 CommitmentLimit=65535
Capabilities: [100] Advanced Error Reporting <?>
Capabilities: [300] Power Budgeting <?>
Kernel modules: shpchp
02:0c.0 Network controller: VMIC Device 5565
Subsystem: PLD APPLICATIONS Device 0080
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz+ UDF- FastB2B- ParErr- DEVSEL=slow >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Interrupt: pin A routed to IRQ 18
Region 0: Memory at cffff000 (32-bit, non-prefetchable) [size=512]
Region 1: I/O ports at ce00 [size=256]
Region 2: Memory at cfffe000 (32-bit, non-prefetchable) [size=64]
Region 3: Memory at b0000000 (32-bit, non-prefetchable) [size=256M]
Capabilities: [78] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
Here is the bridge on the MB and gpiohsd card info:
00:14.4 PCI bridge: ATI Technologies Inc SBx00 PCI to PCI Bridge (prog-if 01 [Subtractive decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop+ ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64
Bus: primary=00, secondary=07, subordinate=07, sec-latency=64
I/O behind bridge: 0000a000-0000afff
Memory behind bridge: fc800000-fd7fffff
Prefetchable memory behind bridge: fdb00000-fdbfffff
Secondary status: 66MHz- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
07:07.0 Intelligent controller [0e80]: PLX Technology, Inc. Device 0480 (rev 52)
Subsystem: Compro Computer Services, Inc. Device 4620
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 64, Cache Line Size: 32 bytes
Interrupt: pin A routed to IRQ 22
BIST result: 00
Region 0: Memory at fd7fe000 (32-bit, non-prefetchable) [size=1K]
Region 1: Memory at fc800000 (32-bit, non-prefetchable) [size=8M]
Region 2: Memory at fd7fd000 (32-bit, non-prefetchable) [size=16]
Capabilities: [40] #00 [0000]
Capabilities: [54] #00 [0080]
Capabilities: [58] Vital Product Data <?>
Again, any help as to why this doesn't work would be appreciated.
Thanks in advance
Mark
View attachment "d.lspci-vvv" of type "text/plain" (26525 bytes)
Powered by blists - more mailing lists