linux-kernel - Multiple copy-on-write branches of an anonymous memory segment

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite: Windows password security audit tool. GUI, reports in PDF.

[<prev] [next>] [day] [month] [year] [list]

Message-Id: <9492A0F4-990D-44F0-B10A-1B55D25995B6@gmail.com>
Date:	Tue, 5 Nov 2013 10:41:43 -0800
From:	Steven Schlansker <stevenschlansker@...il.com>
To:	linux-kernel@...r.kernel.org
Subject: Multiple copy-on-write branches of an anonymous memory segment

Hi,

I am developing a data structure that resides in a large memory buffer.  It is important that I be able to
mutate the data structure without interfering with concurrent reads, and then swap it in at a convenient time.

There is a set of initial data stored in a file.  I use mmap(2) to map the file into memory.  At some later point, I would like to run incremental updates.  This involves copying the data structure, mutating it, then publishing it.  Only a very small percentage of the data structure is changed, but it can change at random.

Currently, I accomplish this by mapping a new anonymous segment, copying the data, and modifying it.  However this is wasteful both in time (copying all that data takes a fair amount of time) and memory (most of the data did not actually change, but now we have two copies).

I would like to utilize kernel support for copy-on-write to eliminate both of these bottlenecks.  Ideally, I would be able to:

1) create a new COW mapping of an existing anonymous mapping that shares pages initially and I can write to without affecting the original
2) At the time of creating that new mapping, extend the buffer with initially-zero pages (in case I need more space to append to the end of the data structure in the new version)

The mmap and mremap documentation get me tantalizingly close — I can resize the segment I already have, I can allocate file-backed copy on write segments.  Also, forking while holding a private mapping seems to provide other bits of the functionality (COW of anonymous segments).  But I can’t seem to figure out how to fit the puzzle pieces together.  Also, I’m running in the JVM, so forking is not a possibility.

There’s a couple of unanswered StackOverflow questions in a similar vein, so I hope I’m not just missing something obvious.  The most relevant is:

http://stackoverflow.com/questions/16965505/allocating-copy-on-write-memory-within-a-process

And it looks like this dance is even possible on Mach, with some clever use of ‘vm_remap'.

I’m hoping that I’m just missing the magic incantation to pass to mmap to achieve this behavior.  I wasn’t brave enough to subscribe to LKML just yet, so please CC me on replies!

TIA,
Steven Schlansker

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/