linux-kernel - Re: Can Linux control PCIe Transaction Layer Packet creation, when writing to a region pointed to by Base Address Register

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-id: <484E9267.8090701@shaw.ca>
Date:	Tue, 10 Jun 2008 08:40:39 -0600
From:	Robert Hancock <hancockr@...w.ca>
To:	"Meyers, Jordan" <Jordan_Meyers@...le.comcast.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: Can Linux control PCIe Transaction Layer Packet creation,
 when writing to a region pointed to by Base Address Register

Meyers, Jordan wrote:
>      During the process of writing a PCIe device driver, I observed an interesting phenomenon with respect to Transaction Layer Packet (TLP) generation.  The device driver needs to transfer upwards of 1/3 KB in the payload of a single TLP.  I used the pci_iomap function combined with memcpy_toio for data transfer to the address space pointed to by a Base Address Register (BAR).  When I used the memcpy_toio, the buffer content I tried to write was split into dword (4B) sized pieces and each dword was given its own TLP.  According to PCI-SIG and some of the research I've done, the PCIe Bus Controller is responsible for this behavior.  The PCIe Bus Controller turns the driver accesses into TLPs.  This presents an interesting problem.  Since the generation of these TLPs (from the BAR region) are out of the hands of the Operating System, what procedure can a driver go through to ensure that all the data makes it into a single TLP payload.  One thought I had was the use of DMA b
uffers, an
> d having the PCIe card pull the data from the DMA buffer.  I observed the DMA pushes from the card to the buffer, were not subject to the same limiting factors.  
>      All the observations I made were done by using a Lecroy PCIe analyzer.  The machine I ran this on was a Dell Precision 490 (which had 2 AMD quad core, x86_64 architecture).  I've tested the behavior on SUSE Linux running kernel 2.6.16 and 2.6.22.  And the chipset was Intel 5000x ( the relevant PCIe component of the chipset is the 6321 ESB I/O Hub Controller).
> 
>      Any thoughts or suggestions of how to ensure a buffer's data is transferred from the driver to the PCIe card in a single TLP (where the intended payload size is less than the max payload value in Device Control register of the PCI register space)?  Thank you to everyone who has taken the time to read this post. 

In general you have no control over what the chipset decides to do with 
CPU writes over the PCI or PCI Express bus. In newer kernels you might 
be able to use iomap_wc, etc. to map the device BAR as write combining 
(if your device can handle the effects of this) which would likely at 
least result in sending more than one dword per TLP. However, in 
general, if you want full efficiency in bus utilization, you typically 
have to make the device perform the reads from memory rather than 
pushing data from the CPU.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/