[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20250506192641.7872cbd0@inno-thin-client>
Date: Tue, 6 May 2025 19:26:41 +0300
From: Zhi Wang <zhiw@...dia.com>
To: Joel Fernandes <joelagnelf@...dia.com>
CC: <linux-kernel@...r.kernel.org>, Danilo Krummrich <dakr@...nel.org>, "David
Airlie" <airlied@...il.com>, Simona Vetter <simona@...ll.ch>, "Maarten
Lankhorst" <maarten.lankhorst@...ux.intel.com>, Maxime Ripard
<mripard@...nel.org>, Thomas Zimmermann <tzimmermann@...e.de>, "Jonathan
Corbet" <corbet@....net>, <nouveau@...ts.freedesktop.org>,
<dri-devel@...ts.freedesktop.org>, Alexandre Courbot <acourbot@...dia.com>,
John Hubbard <jhubbard@...dia.com>, Shirish Baskaran <sbaskaran@...dia.com>,
Alistair Popple <apopple@...dia.com>, Timur Tabi <ttabi@...dia.com>, "Ben
Skeggs" <bskeggs@...dia.com>, <rust-for-linux@...r.kernel.org>,
<linux-doc@...r.kernel.org>
Subject: Re: [PATCH v2 4/7] nova-core: docs: Document fwsec operation and
layout
On Sat, 3 May 2025 00:07:56 -0400
Joel Fernandes <joelagnelf@...dia.com> wrote:
> Add explanation of fwsec with diagrams. This helps clarify how the
> nova-core falcon boot works.
>
> Signed-off-by: Joel Fernandes <joelagnelf@...dia.com>
> ---
> Documentation/gpu/nova/core/fwsec.rst | 180
> ++++++++++++++++++++++++++ Documentation/gpu/nova/index.rst |
> 1 + 2 files changed, 181 insertions(+)
> create mode 100644 Documentation/gpu/nova/core/fwsec.rst
>
> diff --git a/Documentation/gpu/nova/core/fwsec.rst
> b/Documentation/gpu/nova/core/fwsec.rst new file mode 100644
> index 000000000000..bed941ac3f2b
> --- /dev/null
> +++ b/Documentation/gpu/nova/core/fwsec.rst
> @@ -0,0 +1,180 @@
> +.. SPDX-License-Identifier: (GPL-2.0+ OR MIT)
> +=========================
> +FWSEC (Firmware Security)
> +=========================
> +This document briefly/conceptually describes the FWSEC (Firmware
> Security) image +and its role in the GPU boot sequence. As such, this
> information is subject to +change in the future and is only current
> as of the Ampere GPU family. However, +hopefully the concepts
> described will be useful for understanding the kernel code +that
> deals with it. All the information is derived from publicly available
> +sources such as public drivers and documentation. +
> +The role of FWSEC to provide secure boot, it is running in
> Heavy-secure mode. It does +firmware verification after GPU reset and
> load various ucode images on to the other +microcontrollers on the
> GPU such as the PMU and GSP. +
> +FWSEC itself is an application stored in the VBIOS ROM in the FWSEC
> partition of +ROM (see vbios.rst for more details). It contains
> different commands like FRTS +(Firmware Runtime Services) and SB
> (Secure Booting other microcontrollers after +reset and loading them
> with other non-FWSEC ucode). The kernel driver only needs to +to do
> FRTS, since SB is already already after reset by the time the kernel
> driver +is loaded. +
> +The FRTS command carves out the WPR2 region (Write protected region)
> which contains +data data required for power management. Once setup,
repetitive word "data" ^
> only HS mode ucode can +access it (see falcon.rst for privilege
> levels). +
> +The FWSEC image is located in the VBIOS ROM in the partition of the
> ROM that contains +various ucode images (also known as applications)
> -- one of them being FWSEC. For how +it is extracted, see vbios.rst
> and the vbios.rs source code. +
> +The Falcon data for each ucode images (including the FWSEC image) is
> a combination +of headers, data sections (DMEM) and instruction code
> sections (IMEM). All these +ucode images are stored in the same ROM
> partition and the PMU table is used to look +up the application to
> load it based on its application ID (see vbios.rs). +
> +For the purposes of nova-core driver, the FWSEC contains an
> 'application interface' +called DMEMMAPPER which is used to the
> "FWSEC-FRTS" command (among other commands it +is capable of
> executing). For Ampere, FWSEC is running on the GSP in Heavy-secure
> +mode and runs FRTS. +
> +FWSEC Memory Layout
> +-------------------
> +The memory layout of the FWSEC image is as follows (this is using an
> GA-102 +Ampere GPU as an example and could vary for future GPUs and
> is subject to change +completely, it is just provided as an example):
> +
> +Here is a block diagram of the FWSEC memory layout::
> + ┌───────────────────────────────────────────────────────────────┐
> + │ FWSEC ROM image (type 0xE0) │
> + │ │
> + │ ┌─────────────────────────────────┐ │
> + │ │ PMU Falcon Ucode Table │ │
> + │ │ (PmuLookupTable) │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Table Header │ │ │
> + │ │ │ - version: 0x01 │ │ │
> + │ │ │ - header_size: 6 │ │ │
> + │ │ │ - entry_size: 6 │ │ │
> + │ │ │ - entry_count: N │ │ │
> + │ │ │ - desc_version:3(unused)│ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ │ ... │ │
> + │ │ ┌─────────────────────────┐ │ │
> + │ │ │ Entry for FWSEC (0x85) │ │ │
> + │ │ │ (PmuLookupTableEntry) │ │ │
> + │ │ │ - app_id: 0x85 (FWSEC) │ ───┼────┐ │
> + │ │ │ - target_id: 0x01 (PMU) │ │ │ │
> + │ │ │ - data: offset ─────────┼────┼────┼───┐ look up FWSEC │
> + │ │ └─────────────────────────┘ │ │ │ application. │
> + │ └─────────────────────────────────┘ │ │ │
> + │ │ │ │
> + │ │ │ │
> + │ ┌─────────────────────────────────┐ │ │ │
> + │ │ FWSEC Ucode Component │<───┘ │ │
> + │ │ (aka Falcon data) │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ FalconUCodeDescV3 │<───┼────────┘ │
> + │ │ │ - hdr │ │ │
> + │ │ │ - stored_size │ │ │
> + │ │ │ - pkc_data_offset │ │ │
> + │ │ │ - interface_offset ─────┼────┼────────────────┐ │
> + │ │ │ - imem_phys_base │ │ │ │
> + │ │ │ - imem_load_size │ │ │ │
> + │ │ │ - imem_virt_base │ │ │ │
> + │ │ │ - dmem_phys_base │ │ │ │
> + │ │ │ - dmem_load_size │ │ │ │
> + │ │ │ - engine_id_mask │ │ │ │
> + │ │ │ - ucode_id │ │ │ │
> + │ │ │ - signature_count │ │ look up sig │ │
> + │ │ │ - signature_versions --------------+ │ │
> + │ │ └─────────────────────────┘ │ │ │ │
> + │ │ (no gap) │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │ │
> + │ │ │ Signatures Section │<───┼─────┘ │ │
> + │ │ │ (384 bytes per sig) │ │ │ │
> + │ │ │ - RSA-3K Signature 1 │ │ │ │
> + │ │ │ - RSA-3K Signature 2 │ │ │ │
> + │ │ │ ... │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ IMEM Section (Code) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ Contains instruction │ │ │ │
> + │ │ │ code etc. │ │ │ │
> + │ │ └─────────────────────────┘ │ │ │
> + │ │ │ │ │
> + │ │ ┌─────────────────────────┐ │ │ │
> + │ │ │ DMEM Section (Data) │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Application │ │<───┼────────────────┘ │
> + │ │ │ │ Interface Table │ │ │ │
> + │ │ │ │ (FalconAppifHdrV1) │ │ │ │
> + │ │ │ │ Header: │ │ │ │
> + │ │ │ │ - version: 0x01 │ │ │ │
> + │ │ │ │ - header_size: 4 │ │ │ │
> + │ │ │ │ - entry_size: 8 │ │ │ │
> + │ │ │ │ - entry_count: N │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ │ Entries: │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DEVINIT (ID 1) │ │ │ │ │
> + │ │ │ │ │ - id: 0x01 │ │ │ │ │
> + │ │ │ │ │ - dmemOffset X ─┼─┼─┼────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │
> + │ │ │ │ ┌─────────────────┐ │ │ │ │
> + │ │ │ │ │ DMEMMAPPER(ID 4)│ │ │ │ │
> + │ │ │ │ │ - id: 0x04 │ │ │ │ Used only for DevInit │
> + │ │ │ │ │ (NVFW_FALCON_ │ │ │ │ application (not FWSEC) │
> + │ │ │ │ │ APPIF_ID_DMEMMAPPER) │ │
> + │ │ │ │ │ - dmemOffset Y ─┼─┼─┼────┼─────┐ │
> + │ │ │ │ └─────────────────┘ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DEVINIT Engine │<┼────┘ │ Used by FWSEC │
> + │ │ │ │ Interface │ │ │ │ app. │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ DMEM Mapper (ID 4) │<┼────+─────┘ │
> + │ │ │ │ (FalconAppifDmemmapperV3) │ │
> + │ │ │ │ - signature: "DMAP" │ │ │ │
> + │ │ │ │ - version: 0x0003 │ │ │ │
> + │ │ │ │ - Size: 64 bytes │ │ │ │
> + │ │ │ │ - cmd_in_buffer_off │ │────┼────────────┐ │
> + │ │ │ │ - cmd_in_buffer_size│ │ │ │ │
> + │ │ │ │ - cmd_out_buffer_off│ │────┼────────────┼─────┐ │
> + │ │ │ │ - cmd_out_buffer_sz │ │ │ │ │ │
> + │ │ │ │ - init_cmd │ │ │ │ │ │
> + │ │ │ │ - features │ │ │ │ │ │
> + │ │ │ │ - cmd_mask0/1 │ │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │ │
> + │ │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │ │
> + │ │ │ │ Command Input Buffer│<┼────┼────────────┘ │ │
> + │ │ │ │ - Command data │ │ │ │ │
> + │ │ │ │ - Arguments │ │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │ │
> + │ │ │ │ │ │ │
> + │ │ │ ┌─────────────────────┐ │ │ │ │
> + │ │ │ │ Command Output │<┼────┼──────────────────┘ │
> + │ │ │ │ Buffer │ │ │ │
> + │ │ │ │ - Results │ │ │ │
> + │ │ │ │ - Status │ │ │ │
> + │ │ │ └─────────────────────┘ │ │ │
> + │ │ └─────────────────────────┘ │ │
> + │ └─────────────────────────────────┘ │
> + │ │
> + └───────────────────────────────────────────────────────────────┘
> +
> +.. note::
> + The FWSEC image also plays a role in memory scrubbing (ECC
> initialization) and VPR
> + (Video Protected Region) initialization as well. Before the
> nova-core driver is even
> + loaded, the FWSEC image is running on the GSP in heavy-secure
> mode. After the devinit
> + sequence completes, it does VRAM memory scrubbing (ECC
> initialization). On consumer
> + GPUs, it scrubs only part of memory and then initiates 'async
> scrubbing'. Before this
> + async scrubbing completes, the unscrubbed VRAM cannot be used for
> allocation (thus DRM
> + memory allocators need to wait for this scrubbing to complete).
> \ No newline at end of file
> diff --git a/Documentation/gpu/nova/index.rst
> b/Documentation/gpu/nova/index.rst index 91cc802ed94f..22e5712ac6b0
> 100644 --- a/Documentation/gpu/nova/index.rst
> +++ b/Documentation/gpu/nova/index.rst
> @@ -28,4 +28,5 @@ vGPU manager VFIO driver and the nova-drm driver.
>
> core/guidelines
> core/vbios
> + core/fwsec
> core/todo
Powered by blists - more mailing lists