linux-kernel - Writing to file with O

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [day] [month] [year] [list]

Message-ID: <0e54ad1c-5008-480b-844a-a53e64c8ea7d@destevez.net>
Date: Fri, 26 Jul 2024 17:29:05 +0200
From: Daniel Estévez <daniel@...tevez.net>
To: linux-kernel@...r.kernel.org
Subject: Writing to file with O_DIRECT from DMA buffer

Hello,

I'm working on a Zynq MPSoC (aarch64) system. I'm implementing a 
userspace application that writes data from a DMA buffer (contiguous in 
physical memory) into a file in an ext4 filesystem on a SSD. I want to 
use O_DIRECT for this, because there is a larger throughput difference 
when benchmarking writes from a buffer located in regular userspace 
memory (250 MB/s without O_DIRECT vs 1200 MB/s with O_DIRECT).

The DMA buffer is declared with a reserved-memory entry in the device 
tree by using a fixed physical address. There is a simple kernel module 
that lets the userspace application mmap() this buffer. The kernel 
module simply does remap_pfn_range() to create the userspace pages 
corresponding to the physical address range declared in the device tree. 
A DMA in the MPSoC FPGA has already written data into the physical 
memory corresponding to this DMA buffer before the application attempts 
to write the data to a file.

When the userspace application attempts to write data from the buffer 
into the file, which has been opened with O_DIRECT, the writes fail with 
EFAULT. Writes succeed if the file is not opened with O_DIRECT, or if 
instead of the DMA buffer, regular memory from the application (obtained 
with malloc) is used to write into the file (in this case even with 
O_DIRECT enabled).

Considering the difference in how the page tables are set up for regular 
application memory (which supports O_DIRECT) versus my DMA buffer (which 
doesn't work with O_DIRECT), it seems to me that I'm missing something. 
I have tried both with the "no-map" attribute in the reserved-memory 
entry that defines the DMA buffer and without it.

As reported by other users, it seems that this also fails if the DMA 
buffer is allocated with dmam_alloc_coherent():

https://support.xilinx.com/s/question/0D54U00005U9xUhSAJ/odirect-access-to-mmaped-dmamalloccoherent-problems?language=en_US

https://www.linuxquestions.org/questions/linux-software-2/direct-io-fails-when-writting-from-cma-allocated-buffer-4175696145/

I have tried to search what requirements the page tables need to have 
for O_DIRECT to work, but didn't found any information.

Best,
Daniel.

Download attachment "OpenPGP_signature.asc" of type "application/pgp-signature" (834 bytes)