Message-ID: <Y72YMk21sSJLfUiz@dhcp22.suse.cz>
Date:   Tue, 10 Jan 2023 17:54:10 +0100
From:   Michal Hocko <mhocko@...e.com>
To:     Mike Rapoport <rppt@...nel.org>
Cc:     Jonathan Corbet <corbet@....net>,
        Andrew Morton <akpm@...ux-foundation.org>,
        Bagas Sanjaya <bagasdotme@...il.com>,
        David Hildenbrand <david@...hat.com>,
        Johannes Weiner <hannes@...xchg.org>,
        Lorenzo Stoakes <lstoakes@...il.com>,
        "Matthew Wilcox (Oracle)" <willy@...radead.org>,
        Mel Gorman <mgorman@...e.de>, Vlastimil Babka <vbabka@...e.cz>,
        linux-doc@...r.kernel.org, linux-kernel@...r.kernel.org,
        linux-mm@...ck.org
Subject: Re: [PATCH v2 2/2] docs/mm: Physical Memory: add structure,
 introduction and nodes description

On Tue 10-01-23 17:23:58, Mike Rapoport wrote:
[...]
> +* ``ZONE_DMA`` and ``ZONE_DMA32`` represent memory suitable for DMA by
> +  peripheral devices that cannot access all of the addressable memory.

I think it would be better to not keep the historical DMA based menaning
and teach that future developers. You can say something like

ZONE_DMA and ZONE_DMA32 have historically been used for memory suitable
for DMA. For many years there are better more robust interfaces to
get memory with DMA specific requirements (Documentation/core-api/dma-api.rst).

> +  Depending on the architecture, either of these zone types or even they both
> +  can be disabled at build time using ``CONFIG_ZONE_DMA`` and
> +  ``CONFIG_ZONE_DMA32`` configuration options. Some 64-bit platforms may need
> +  both zones as they support peripherals with different DMA addressing
> +  limitations.
> +
> +* ``ZONE_NORMAL`` is for normal memory that can be accessed by the kernel all
> +  the time. DMA operations can be performed on pages in this zone if the DMA
> +  devices support transfers to all addressable memory. ``ZONE_NORMAL`` is
> +  always enabled.
> +
> +* ``ZONE_HIGHMEM`` is the part of the physical memory that is not covered by a
> +  permanent mapping in the kernel page tables. The memory in this zone is only
> +  accessible to the kernel using temporary mappings. This zone is available
> +  only on some 32-bit architectures and is enabled with ``CONFIG_HIGHMEM``.
> +
> +* ``ZONE_MOVABLE`` is for normal accessible memory, just like ``ZONE_NORMAL``.
> +  The difference is that most pages in ``ZONE_MOVABLE`` are movable.

This is really confusing because those pages are not really movable. You
cannot move a page itself. I guess you meant to say something like

The difference is that there are means to migrate memory via
migrate_pages interface. A typical example would be a memory mapped to
userspace which can be rellocate the underlying memory content and
update page tables so that userspace doesn't notice the physical data
placement has changed.

> That means
> +  that while virtual addresses of these pages do not change, their content may
> +  move between different physical pages. ``ZONE_MOVABLE`` is only enabled when
> +  one of ``kernelcore``, ``movablecore`` and ``movable_node`` parameters is
> +  present in the kernel command line. See :ref:`Page migration
> +  <page_migration>` for additional details.

This is not really true. The movable zone can be also enabled by memory
hotplug. In fact it is one of the more common usecases for the zone
because memory hot remove largerly depends on memory to be migrated for
offlining to succeed in most cases.

> +
> +* ``ZONE_DEVICE`` represents memory residing on devices such as PMEM and GPU.
> +  It has different characteristics than RAM zone types and it exists to provide
> +  :ref:`struct page <Pages>` and memory map services for device driver
> +  identified physical address ranges. ``ZONE_DEVICE`` is enabled with
> +  configuration option ``CONFIG_ZONE_DEVICE``.
> +
> +It is important to note that many kernel operations can only take place using
> +``ZONE_NORMAL`` so it is the most performance critical zone. Zones are
> +discussed further in Section :ref:`Zones <zones>`.
> +
> +The relation between node and zone extents is determined by the physical memory
> +map reported by the firmware, architectural constraints for memory addressing
> +and certain parameters in the kernel command line.
> +
> +For example, with 32-bit kernel on an x86 UMA machine with 2 Gbytes of RAM the
> +entire memory will be on node 0 and there will be three zones: ``ZONE_DMA``,
> +``ZONE_NORMAL`` and ``ZONE_HIGHMEM``::
> +
> +  0                                                            2G
> +  +-------------------------------------------------------------+
> +  |                            node 0                           |
> +  +-------------------------------------------------------------+
> +
> +  0         16M                    896M                        2G
> +  +----------+-----------------------+--------------------------+
> +  | ZONE_DMA |      ZONE_NORMAL      |       ZONE_HIGHMEM       |
> +  +----------+-----------------------+--------------------------+
> +
> +
> +With a kernel built with ``ZONE_DMA`` disabled and ``ZONE_DMA32`` enabled and
> +booted with ``movablecore=80%`` parameter on an arm64 machine with 16 Gbytes of
> +RAM equally split between two nodes, there will be ``ZONE_DMA32``,
> +``ZONE_NORMAL`` and ``ZONE_MOVABLE`` on node 0, and ``ZONE_NORMAL`` and
> +``ZONE_MOVABLE`` on node 1::
> +
> +
> +  1G                                9G                         17G
> +  +--------------------------------+ +--------------------------+
> +  |              node 0            | |          node 1          |
> +  +--------------------------------+ +--------------------------+
> +
> +  1G       4G        4200M          9G          9320M          17G
> +  +---------+----------+-----------+ +------------+-------------+
> +  |  DMA32  |  NORMAL  |  MOVABLE  | |   NORMAL   |   MOVABLE   |
> +  +---------+----------+-----------+ +------------+-------------+

I think it is useful to note that nodes and zones can overlap in the
physical address range. It is not uncommong to interleave two nodes and
it is also possible that memory holes are memory hotplugged into MOVABLE
zone arbitrarily in the physical address range.

Other than that looks good to me and thanks for taking care of filling
up these gaps! This is highly appreciated.
-- 
Michal Hocko
SUSE Labs

Powered by blists - more mailing lists