Message-Id: <1415047353-29160-1-git-send-email-j.glisse@gmail.com>
Date:	Mon,  3 Nov 2014 15:42:28 -0500
From:	j.glisse@...il.com
To:	akpm@...ux-foundation.org
Cc:	<linux-kernel@...r.kernel.org>, <linux-mm@...ck.org>,
	Linus Torvalds <torvalds@...ux-foundation.org>,
	<joro@...tes.org>, Mel Gorman <mgorman@...e.de>,
	"H. Peter Anvin" <hpa@...or.com>,
	Peter Zijlstra <peterz@...radead.org>,
	Andrea Arcangeli <aarcange@...hat.com>,
	Johannes Weiner <jweiner@...hat.com>,
	Larry Woodman <lwoodman@...hat.com>,
	Rik van Riel <riel@...hat.com>,
	Dave Airlie <airlied@...hat.com>,
	Brendan Conoboy <blc@...hat.com>,
	Joe Donohue <jdonohue@...hat.com>,
	Duncan Poole <dpoole@...dia.com>,
	Sherry Cheung <SCheung@...dia.com>,
	Subhash Gutti <sgutti@...dia.com>,
	John Hubbard <jhubbard@...dia.com>,
	Mark Hairgrove <mhairgrove@...dia.com>,
	Lucien Dunning <ldunning@...dia.com>,
	Cameron Buschardt <cabuschardt@...dia.com>,
	Arvind Gopalakrishnan <arvindg@...dia.com>,
	Shachar Raindel <raindel@...lanox.com>,
	Liran Liss <liranl@...lanox.com>,
	Roland Dreier <roland@...estorage.com>,
	Ben Sander <ben.sander@....com>,
	Greg Stoner <Greg.Stoner@....com>,
	John Bridgman <John.Bridgman@....com>,
	Michael Mantor <Michael.Mantor@....com>,
	Paul Blinzer <Paul.Blinzer@....com>,
	Laurent Morichetti <Laurent.Morichetti@....com>,
	Alexander Deucher <Alexander.Deucher@....com>,
	Oded Gabbay <Oded.Gabbay@....com>,
	<linux-fsdevel@...r.kernel.org>, "Linda Wang" <lwang@...hat.com>,
	"Kevin E Martin" <kem@...hat.com>,
	"Jerome Glisse" <jglisse@...hat.com>, "Jeff Law" <law@...hat.com>,
	"Haggai Eran" <haggaie@...lanox.com>,
	"Or Gerlitz" <ogerlitz@...lanox.com>,
	"Sagi Grimberg" <sagig@...lanox.com>
Subject: HMM (heterogeneous memory management) v5

Andrew i received no feedback since last time i sent this patchset, so i
would really like to have it merge for next kernel. While right now there
is no kernel driver that leverage this code, the hardware is coming and we
still have a long way to go before we have all the features needed. Right
now i am blocking any further work on the merge of this core code.

(Note that patch 5 the dummy driver is included as reference and should not
be merge unless you want me to grow it into some testing infrastructure. I
only include it here so people can have a look on how HMM is suppose to be
use).


What it is ?

In a nutshell HMM is a subsystem that provide an easy to use api to mirror a
process address on a device with minimal hardware requirement (mainly device
page fault and read only page mapping). This does not rely on ATS and PASID
PCIE extensions. It intends to supersede those extensions by allowing to move
system memory to device memory in a transparent fashion for core kernel mm
code (ie cpu page fault on page residing in device memory will trigger
migration back to system memory).


Why doing this ?

We want to be able to mirror a process address space so that compute api such
as OpenCL or other similar api can start using the exact same address space on
the GPU as on the CPU. This will greatly simplify usages of those api. Moreover
we believe that we will see more and more specialize unit functions that will
want to mirror process address using their own mmu.

The migration side is simply because GPU memory bandwidth is far beyond than
system memory bandwith and there is no sign that this gap is closing (quite the
opposite).


Current status and future features :

None of this core code change in any major way core kernel mm code. This
is simple ground work with no impact on existing code path. Features that
will be implemented on top of this are :
  1 - Tansparently handle page mapping on behalf of device driver (DMA).
  2 - Improve DMA api to better match new usage pattern of HMM.
  3 - Migration of anonymous memory to device memory.
  4 - Locking memory to remote memory (CPU access triger SIGBUS).
  5 - Access exclusion btw CPU and device for atomic operations.
  6 - Migration of file backed memory to device memory.


How future features will be implemented :
1 - Simply use existing DMA api to map page on behalf of a device.
2 - Introduce new DMA api to match new semantic of HMM. It is no longer page
    we map but address range and managing which page is effectively backing
    an address should be easy to update. I gave a presentation about that
    during this LPC.
3 - Requires change to cpu page fault code path to handle migration back to
    system memory on cpu access. An implementation of this was already sent
    as part of v1. This will be low impact and only add a new special swap
    type handling to existing fault code.
4 - Require a new syscall as i can not see which current syscall would be
    appropriate for this. My first feeling was to use mbind as it has the
    right semantic (binding a range of address to a device) but mbind is
    too numa centric.

    Second one was madvise, but semantic does not match, madvise does allow
    kernel to ignore them while we do want to block cpu access for as long
    as the range is bind to a device.

    So i do not think any of existing syscall can be extended with new flags
    but maybe i am wrong.
5 - Allowing to map a page as read only on the CPU while a device perform
    some atomic operation on it (this is mainly to work around system bus
    that do not support atomic memory access and sadly there is a large
    base of hardware without that feature).

    Easiest implementation would be using some page flags but there is none
    left. So it must be some flags in vma to know if there is a need to query
    HMM for write protection.

6 - This is the trickiest one to implement and while i showed a proof of
    concept with v1, i am still have a lot of conflictual feeling about how
    to achieve this.


As usual comments are more then welcome. Thanks in advance to anyone that
take a look at this code.

Previous patchset posting :
  v1 http://lwn.net/Articles/597289/
  v2 https://lkml.org/lkml/2014/6/12/559 (cover letter did not make it to ml)
  v3 https://lkml.org/lkml/2014/6/13/633
  v4 https://lkml.org/lkml/2014/8/29/423

Cheers,
Jérôme

To: "Andrew Morton" <akpm@...ux-foundation.org>,
Cc: <linux-kernel@...r.kernel.org>,
Cc: linux-mm <linux-mm@...ck.org>,
Cc: <linux-fsdevel@...r.kernel.org>,
Cc: "Linus Torvalds" <torvalds@...ux-foundation.org>,
Cc: "Mel Gorman" <mgorman@...e.de>,
Cc: "H. Peter Anvin" <hpa@...or.com>,
Cc: "Peter Zijlstra" <peterz@...radead.org>,
Cc: "Linda Wang" <lwang@...hat.com>,
Cc: "Kevin E Martin" <kem@...hat.com>,
Cc: "Jerome Glisse" <jglisse@...hat.com>,
Cc: "Andrea Arcangeli" <aarcange@...hat.com>,
Cc: "Johannes Weiner" <jweiner@...hat.com>,
Cc: "Larry Woodman" <lwoodman@...hat.com>,
Cc: "Rik van Riel" <riel@...hat.com>,
Cc: "Dave Airlie" <airlied@...hat.com>,
Cc: "Jeff Law" <law@...hat.com>,
Cc: "Brendan Conoboy" <blc@...hat.com>,
Cc: "Joe Donohue" <jdonohue@...hat.com>,
Cc: "Duncan Poole" <dpoole@...dia.com>,
Cc: "Sherry Cheung" <SCheung@...dia.com>,
Cc: "Subhash Gutti" <sgutti@...dia.com>,
Cc: "John Hubbard" <jhubbard@...dia.com>,
Cc: "Mark Hairgrove" <mhairgrove@...dia.com>,
Cc: "Lucien Dunning" <ldunning@...dia.com>,
Cc: "Cameron Buschardt" <cabuschardt@...dia.com>,
Cc: "Arvind Gopalakrishnan" <arvindg@...dia.com>,
Cc: "Haggai Eran" <haggaie@...lanox.com>,
Cc: "Or Gerlitz" <ogerlitz@...lanox.com>,
Cc: "Sagi Grimberg" <sagig@...lanox.com>
Cc: "Shachar Raindel" <raindel@...lanox.com>,
Cc: "Liran Liss" <liranl@...lanox.com>,
Cc: "Roland Dreier" <roland@...estorage.com>,
Cc: "Sander, Ben" <ben.sander@....com>,
Cc: "Stoner, Greg" <Greg.Stoner@....com>,
Cc: "Bridgman, John" <John.Bridgman@....com>,
Cc: "Mantor, Michael" <Michael.Mantor@....com>,
Cc: "Blinzer, Paul" <Paul.Blinzer@....com>,
Cc: "Morichetti, Laurent" <Laurent.Morichetti@....com>,
Cc: "Deucher, Alexander" <Alexander.Deucher@....com>,
Cc: "Gabbay, Oded" <Oded.Gabbay@....com>,

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@...r.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Powered by blists - more mailing lists