linux-kernel - Re: O_DIRECT reads appear to be cached on block device partition file?

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <20100914073643.GB15695@dastard>
Date:	Tue, 14 Sep 2010 17:36:43 +1000
From:	Dave Chinner <david@...morbit.com>
To:	Brett Russ <icycle+lkml@...il.com>
Cc:	linux-kernel@...r.kernel.org
Subject: Re: O_DIRECT reads appear to be cached on block device partition
 file?

On Mon, Sep 13, 2010 at 11:49:32PM -0400, Brett Russ wrote:
> Running a 2.6.31 kernel on a blade chassis system with multiple
> blades sharing common JBOD storage.  The application intelligently
> divides the drives up among the blades, but one blade in particular
> is charged with monitoring.  As part of this, this one monitoring
> blade can perform reads of a certain 512B sector of all disks in the
> system.  This sector is often written by other blades, these writes
> are sync'd to disk.  To work around the lack of cache coherency
> between the distinct blades, I'm using O_DIRECT on the monitoring
> blade such that it always reads from the media to get the latest
> copy of this sector.  The basic steps are:
> 
> # grab a 512B aligned buffer (use 4KB to be safe)
> posix_memalign(&ptr, getpagesize(), 512B)
> open(/dev/sdX3, O_RDONLY|O_DIRECT)
> lseek(fd, offset, SEEK_SET)
> read(fd, ptr, 512B)
> 
> If I run the above on the monitoring blade, then sync an update to
> the sector in question from another blade, then re-reun the above
> code on the monitoring blade, believe it or not I appear to be
> reading stale data.  If I use dd with iflag=direct, reading the same
> sector offset at the /dev/sdX3 partition file, I see the same stale
> data as seen from the code above.  If, however, I instead access
> this sector offset from the /dev/sdX device file using the (offset
> of partition 3 + offset of the sector) I see the intended data,
> which makes me believe some caching occurred locally for /dev/sdX3.

What does blktrace tell you?

Cheers,

Dave.
-- 
Dave Chinner
david@...morbit.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/