linux-kernel - Re: [PATCH v3 0/5] reduce system memory requirement for hibernation

lists.openwall.net		lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC
Open Source and information security mailing list archives

Hash Suite for Android: free password hash cracker in your pocket

[<prev] [next>] [<thread-prev] [thread-next>] [day] [month] [year] [list]

Message-ID: <6758cecc-d324-4ed5-b38e-2a4384a34d60@amd.com>
Date: Tue, 8 Jul 2025 10:31:01 -0400
From: Mario Limonciello <mario.limonciello@....com>
To: "Rafael J. Wysocki" <rafael@...nel.org>
Cc: lijo.lazar@....com, victor.zhao@....com, haijun.chang@....com,
 Qing.Ma@....com, Owen.Zhang2@....com, linux-pm@...r.kernel.org,
 linux-kernel@...r.kernel.org, amd-gfx@...ts.freedesktop.org,
 dri-devel@...ts.freedesktop.org, Samuel Zhang <guoqing.zhang@....com>,
 alexander.deucher@....com, christian.koenig@....com, len.brown@...el.com,
 pavel@...nel.org, gregkh@...uxfoundation.org, dakr@...nel.org,
 airlied@...il.com, simona@...ll.ch, ray.huang@....com,
 matthew.auld@...el.com, matthew.brost@...el.com,
 maarten.lankhorst@...ux.intel.com, mripard@...nel.org, tzimmermann@...e.de
Subject: Re: [PATCH v3 0/5] reduce system memory requirement for hibernation

On 7/8/2025 3:42 AM, Samuel Zhang wrote:
> Modern data center dGPUs are usually equipped with very large VRAM. On
> server with such dGPUs(192GB VRAM * 8) and 2TB system memory, hibernate
> will fail due to no enough free memory.
> 
> The root cause is that during hibernation all VRAM memory get evicted to
> GTT or shmem. In both case, it is in system memory and kernel will try to
> copy the pages to hibernation image. In the worst case, this causes 2
> copies of VRAM memory in system memory, 2TB is not enough for the
> hibernation image. 192GB * 8 * 2 = 3TB > 2TB.
> 
> The fix includes following changes. With these changes, there's much less
> pages needed to be copied to hibernate image and hibernation can succeed.
> * patch 1 and 2: move GTT to shmem after evicting VRAM. so that the GTT
>    pages can be freed.
> * patch 3: force write shmem pages to swap disk and free shmem pages.
> 
> After swapout GTT to shmem in hibernation prepare stage, the GPU will be
> resumed again in thaw stage. The swapin and restore BOs of resume takes
> lots of time (50 mintues observed for 8 dGPUs). And it's unnecessary since
> writing hibernation image do not need GPU for hibernate successful case.
> * patch 4 and 5: skip resume of device in thaw stage for successful
>    hibernation case to reduce the hibernation time.
> 
> v2:
> * split first patch to 2 patches, 1 for ttm, 1 for amdgpu
> * refined the new ttm api
> * add more comments for shrink_shmem_memory() and its callsite
> * export variable pm_transition in kernel
> * skip resume in thaw() for successful hibernation case
> v3:
> * refined ttm_device_prepare_hibernation() to accept device argument
> * use guard(mutex) to replace mutex_lock and mutex_unlock
> * move ttm_device_prepare_hibernation call to amdgpu_device_evict_resources()
> * add pm_transition_event(), instead of exporting pm_transition variable
> * refined amdgpu_pmops_thaw(), use switch-case for more clarity
> 
> Samuel Zhang (5):
> 1. drm/ttm: add ttm_device_prepare_hibernation() api
> 2. drm/amdgpu: move GTT to shmem after eviction for hibernation
> 3. PM: hibernate: shrink shmem pages after dev_pm_ops.prepare()
> 4. PM: hibernate: add new api pm_transition_event()
> 5. drm/amdgpu: do not resume device in thaw for normal hibernation
> 
>   drivers/base/power/main.c                  |  5 +++++
>   drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 10 ++++++++-
>   drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c    | 15 ++++++++++++-
>   drivers/gpu/drm/ttm/ttm_device.c           | 23 +++++++++++++++++++
>   include/drm/ttm/ttm_device.h               |  1 +
>   include/linux/pm.h                         | 16 +++++++++++++
>   kernel/power/hibernate.c                   | 26 ++++++++++++++++++++++
>   7 files changed, 94 insertions(+), 2 deletions(-)
> 

As there isn't a mention of intent on how to merge when this is fully 
reviewed, I wanted to ask Rafael what he thinks.

The options are either:
* the linux-pm changes go to linux-pm and an immutable branch for drm to 
merge
* everything goes through amd-staging-drm-next (and an amdgpu PR to drm 
later)
* everything goes through drm-misc-next

I think everything through drm-misc-next makes most sense if everyone is 
amenable.