| lists.openwall.net | lists / announce owl-users owl-dev john-users john-dev passwdqc-users yescrypt popa3d-users / oss-security kernel-hardening musl sabotage tlsify passwords / crypt-dev xvendor / Bugtraq Full-Disclosure linux-kernel linux-netdev linux-ext4 linux-hardening linux-cve-announce PHC | |
|
Open Source and information security mailing list archives
| ||
|
Message-ID: <04eef4d1-b1d0-40f4-a633-1bedff779604@habana.ai>
Date: Wed, 19 Jun 2024 11:40:12 +0000
From: Omer Shpigelman <oshpigelman@...ana.ai>
To: Alexander Lobakin <aleksander.lobakin@...el.com>
CC: "linux-kernel@...r.kernel.org" <linux-kernel@...r.kernel.org>,
"linux-rdma@...r.kernel.org" <linux-rdma@...r.kernel.org>,
"netdev@...r.kernel.org" <netdev@...r.kernel.org>,
"dri-devel@...ts.freedesktop.org" <dri-devel@...ts.freedesktop.org>,
"ogabbay@...nel.org" <ogabbay@...nel.org>,
Zvika Yehudai <zyehudai@...ana.ai>
Subject: Re: [PATCH 00/15] Introduce HabanaLabs network drivers
On 6/17/24 15:34, Alexander Lobakin wrote:
> [Some people who received this message don't often get email from aleksander.lobakin@...el.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> From: Omer Shpigelman <oshpigelman@...ana.ai>
> Date: Thu, 13 Jun 2024 11:21:53 +0300
>
>> This patch set implements the HabanaLabs network drivers for Gaudi2 ASIC
>> which is designed for scaling of AI neural networks training.
>> The patch set includes the common code which is shared by all Gaudi ASICs
>> and the Gaudi2 ASIC specific code. Newer ASICs code will be followed.
>> All of these network drivers are modeled as an auxiliary devices to the
>> parent driver.
>>
>> The newly added drivers are Core Network (CN), Ethernet and InfiniBand.
>> All of these drivers are based on the existing habanalabs driver which
>> serves as the compute driver and the entire platform.
>> The habanalabs driver probes the network drivers which configure the
>> relevant NIC HW of the device. In addition, it continuously communicates
>> with the CN driver for providing some services which are not NIC specific
>> e.g. PCI, MMU, FW communication etc.
>>
>> See the drivers scheme at:
>> Documentation/networking/device_drivers/ethernet/intel/hbl.rst
>>
>> The CN driver is both a parent and a son driver. It serves as the common
>> layer of many shared operations that are required by both EN and IB
>> drivers.
>>
>> The Gaudi2 NIC HW is composed of 48 physical lanes, 56Gbps each. Each pair
>> of lanes represent a 100Gbps logical port.
>>
>> The NIC HW was designed specifically for scaling AI training.
>> Hence it basically functions as a regular NIC device but it is tuned for
>> its dedicated purpose. As a result, the NIC HW supports Ethernet traffic
>> and RDMA over modified ROCEv2 protocol.
>> For example, with respect to the IB driver, the HW supports a single
>> context and a single PD. The reason for this is that the operational use
>> case of AI training for Gaudi2 consists of a single user
>> application/process.
>> Another example related to the IB driver is the lack of MR since a single
>> application/process can share the entire MMU with the compute device.
>> Moreover, the memory allocation of user data buffers which are used for
>> RDMA communication is done via the habanalabs compute driver uAPI.
>> With respect to the Ethernet driver, since the Ethernet flow is used
>> mainly for control, the HW is not performance tuned e.g. it assumes a
>> contiguous memory for the Rx buffers. Thus the EN driver needs to copy the
>> Rx packets from the Rx buffer into the skb memory.
>>
>> The first 8 patches implement the CN driver.
>> The next 2 patches implement the EN driver.
>> The next 2 patches implement the IB driver.
>> The last 3 patches modify the compute driver to support the CN driver.
>>
>> The patches are rebased on v6.10-rc3 tag:
>> https://github.com/torvalds/linux/releases/tag/v6.10-rc3
>>
>> The patches are also available at:
>> https://github.com/HabanaAI/drivers.gpu.linux-nic.kernel/tree/hbl_next
>>
>> The user-mode of the driver is being reviewed at:
>> https://github.com/linux-rdma/rdma-core/pull/1472
>>
>> Any feedback, comment or question is welcome.
>>
>> Thanks,
>> Omer
>>
>> Omer Shpigelman (15):
>> net: hbl_cn: add habanalabs Core Network driver
>> net: hbl_cn: memory manager component
>> net: hbl_cn: physical layer support
>> net: hbl_cn: QP state machine
>> net: hbl_cn: memory trace events
>> net: hbl_cn: debugfs support
>> net: hbl_cn: gaudi2: ASIC register header files
>> net: hbl_cn: gaudi2: ASIC specific support
>> net: hbl_en: add habanalabs Ethernet driver
>> net: hbl_en: gaudi2: ASIC specific support
>> RDMA/hbl: add habanalabs RDMA driver
>> RDMA/hbl: direct verbs support
>> accel/habanalabs: network scaling support
>> accel/habanalabs/gaudi2: CN registers header files
>> accel/habanalabs/gaudi2: network scaling support
>>
>> .../ABI/testing/debugfs-driver-habanalabs_cn | 195 +
>> .../device_drivers/ethernet/index.rst | 1 +
>> .../device_drivers/ethernet/intel/hbl.rst | 82 +
>> MAINTAINERS | 33 +
>> drivers/accel/habanalabs/Kconfig | 1 +
>> drivers/accel/habanalabs/Makefile | 3 +
>> drivers/accel/habanalabs/cn/Makefile | 2 +
>> drivers/accel/habanalabs/cn/cn.c | 815 +
>> drivers/accel/habanalabs/cn/cn.h | 133 +
>> .../habanalabs/common/command_submission.c | 2 +-
>> drivers/accel/habanalabs/common/device.c | 23 +
>> drivers/accel/habanalabs/common/firmware_if.c | 20 +
>> drivers/accel/habanalabs/common/habanalabs.h | 43 +-
>> .../accel/habanalabs/common/habanalabs_drv.c | 37 +-
>> .../habanalabs/common/habanalabs_ioctl.c | 2 +
>> drivers/accel/habanalabs/common/memory.c | 123 +
>> drivers/accel/habanalabs/gaudi/gaudi.c | 14 +-
>> drivers/accel/habanalabs/gaudi2/Makefile | 2 +-
>> drivers/accel/habanalabs/gaudi2/gaudi2.c | 440 +-
>> drivers/accel/habanalabs/gaudi2/gaudi2P.h | 41 +-
>> drivers/accel/habanalabs/gaudi2/gaudi2_cn.c | 424 +
>> drivers/accel/habanalabs/gaudi2/gaudi2_cn.h | 42 +
>> .../habanalabs/gaudi2/gaudi2_coresight.c | 145 +-
>> .../accel/habanalabs/gaudi2/gaudi2_security.c | 16 +-
>> drivers/accel/habanalabs/goya/goya.c | 6 +
>> .../include/gaudi2/asic_reg/gaudi2_regs.h | 10 +-
>> .../include/gaudi2/asic_reg/nic0_phy_regs.h | 59 +
>> .../nic0_qm0_axuser_nonsecured_regs.h | 61 +
>> .../include/gaudi2/asic_reg/nic0_qpc1_regs.h | 905 +
>> .../include/gaudi2/asic_reg/nic0_rxe0_regs.h | 725 +
>> .../include/gaudi2/asic_reg/nic0_rxe1_regs.h | 725 +
>> .../include/gaudi2/asic_reg/nic0_txe0_regs.h | 529 +
>> .../include/gaudi2/asic_reg/nic0_txs0_regs.h | 289 +
>> .../include/hw_ip/nic/nic_general.h | 15 +
>> drivers/infiniband/Kconfig | 1 +
>> drivers/infiniband/hw/Makefile | 1 +
>> drivers/infiniband/hw/hbl/Kconfig | 18 +
>> drivers/infiniband/hw/hbl/Makefile | 12 +
>> drivers/infiniband/hw/hbl/hbl.h | 326 +
>> drivers/infiniband/hw/hbl/hbl_encap.c | 216 +
>> drivers/infiniband/hw/hbl/hbl_main.c | 493 +
>> drivers/infiniband/hw/hbl/hbl_query_port.c | 96 +
>> drivers/infiniband/hw/hbl/hbl_set_port_ex.c | 96 +
>> drivers/infiniband/hw/hbl/hbl_usr_fifo.c | 252 +
>> drivers/infiniband/hw/hbl/hbl_verbs.c | 2686 +
>> drivers/net/ethernet/intel/Kconfig | 38 +
>> drivers/net/ethernet/intel/Makefile | 2 +
>> drivers/net/ethernet/intel/hbl_cn/Makefile | 14 +
>> .../net/ethernet/intel/hbl_cn/common/Makefile | 3 +
>> .../net/ethernet/intel/hbl_cn/common/hbl_cn.c | 5984 ++
>> .../net/ethernet/intel/hbl_cn/common/hbl_cn.h | 1666 +
>> .../intel/hbl_cn/common/hbl_cn_debugfs.c | 1457 +
>> .../ethernet/intel/hbl_cn/common/hbl_cn_drv.c | 240 +
>> .../intel/hbl_cn/common/hbl_cn_memory.c | 368 +
>> .../ethernet/intel/hbl_cn/common/hbl_cn_phy.c | 234 +
>> .../ethernet/intel/hbl_cn/common/hbl_cn_qp.c | 491 +
>> .../net/ethernet/intel/hbl_cn/gaudi2/Makefile | 3 +
>> .../asic_reg/arc_farm_kdma_ctx_axuser_masks.h | 135 +
>> .../asic_reg/dcore0_sync_mngr_objs_regs.h | 43543 +++++++++++++++
>> .../asic_reg/gaudi2_blocks_linux_driver.h | 45068 ++++++++++++++++
>
> I don't think adding generated register defs etc. is a good idea.
> You just bloat the kernel code while most of the values are not used.
>
> When I work with HW and need to use some register defs, I add them
> manually one-by-one only when needed. I know it takes more time than to
> just add a whole generated reg file, but we don't need tens of thousand
> unused locs in the kernel.
> Please add only the actually used definitions. This applies to every
> file from the series.
>
Ok, I'll check which register definitions are actually needed.
>> .../hbl_cn/gaudi2/asic_reg/gaudi2_regs.h | 77 +
>> .../asic_reg/nic0_mac_ch0_mac_128_masks.h | 339 +
>> .../asic_reg/nic0_mac_ch0_mac_128_regs.h | 101 +
>> .../asic_reg/nic0_mac_ch0_mac_pcs_masks.h | 713 +
>> .../asic_reg/nic0_mac_ch0_mac_pcs_regs.h | 271 +
>> .../asic_reg/nic0_mac_ch1_mac_pcs_regs.h | 271 +
>> .../asic_reg/nic0_mac_ch2_mac_pcs_regs.h | 271 +
>> .../asic_reg/nic0_mac_ch3_mac_pcs_regs.h | 271 +
>> .../nic0_mac_glob_stat_control_reg_masks.h | 67 +
>> .../nic0_mac_glob_stat_control_reg_regs.h | 37 +
>> .../asic_reg/nic0_mac_glob_stat_rx0_regs.h | 93 +
>> .../asic_reg/nic0_mac_glob_stat_rx2_regs.h | 93 +
>> .../asic_reg/nic0_mac_glob_stat_tx0_regs.h | 75 +
>> .../asic_reg/nic0_mac_glob_stat_tx2_regs.h | 75 +
>> .../gaudi2/asic_reg/nic0_mac_rs_fec_regs.h | 157 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_phy_masks.h | 77 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_phy_regs.h | 59 +
>> .../nic0_qm0_axuser_nonsecured_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_cong_que_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_db_fifo_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_err_fifo_regs.h | 61 +
>> .../nic0_qpc0_axuser_ev_que_lbw_intr_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_qpc_req_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_qpc_resp_regs.h | 61 +
>> .../asic_reg/nic0_qpc0_axuser_rxwqe_regs.h | 61 +
>> .../nic0_qpc0_axuser_txwqe_lbw_qman_bp_regs.h | 61 +
>> .../nic0_qpc0_dbfifo0_ci_upd_addr_regs.h | 27 +
>> .../nic0_qpc0_dbfifosecur_ci_upd_addr_regs.h | 27 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_qpc0_masks.h | 963 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_qpc0_regs.h | 905 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_qpc1_regs.h | 905 +
>> .../gaudi2/asic_reg/nic0_rxb_core_masks.h | 459 +
>> .../gaudi2/asic_reg/nic0_rxb_core_regs.h | 665 +
>> .../nic0_rxe0_axuser_axuser_cq0_regs.h | 61 +
>> .../nic0_rxe0_axuser_axuser_cq1_regs.h | 61 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_rxe0_masks.h | 705 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_rxe0_regs.h | 725 +
>> .../asic_reg/nic0_rxe0_wqe_aruser_regs.h | 61 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_rxe1_regs.h | 725 +
>> .../gaudi2/asic_reg/nic0_serdes0_masks.h | 7163 +++
>> .../gaudi2/asic_reg/nic0_serdes0_regs.h | 1679 +
>> .../gaudi2/asic_reg/nic0_serdes1_regs.h | 1679 +
>> .../asic_reg/nic0_tmr_axuser_tmr_fifo_regs.h | 61 +
>> .../nic0_tmr_axuser_tmr_free_list_regs.h | 61 +
>> .../asic_reg/nic0_tmr_axuser_tmr_fsm_regs.h | 61 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_tmr_masks.h | 361 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_tmr_regs.h | 183 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_txb_regs.h | 167 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_txe0_masks.h | 759 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_txe0_regs.h | 529 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_txs0_masks.h | 555 +
>> .../hbl_cn/gaudi2/asic_reg/nic0_txs0_regs.h | 289 +
>> .../nic0_umr0_0_completion_queue_ci_1_regs.h | 27 +
>> .../nic0_umr0_0_unsecure_doorbell0_regs.h | 31 +
>> .../nic0_umr0_0_unsecure_doorbell1_regs.h | 31 +
>> .../gaudi2/asic_reg/prt0_mac_core_masks.h | 137 +
>> .../gaudi2/asic_reg/prt0_mac_core_regs.h | 67 +
>> .../ethernet/intel/hbl_cn/gaudi2/gaudi2_cn.c | 5689 ++
>> .../ethernet/intel/hbl_cn/gaudi2/gaudi2_cn.h | 427 +
>> .../intel/hbl_cn/gaudi2/gaudi2_cn_debugfs.c | 319 +
>> .../intel/hbl_cn/gaudi2/gaudi2_cn_eq.c | 732 +
>> .../intel/hbl_cn/gaudi2/gaudi2_cn_phy.c | 2743 +
>> drivers/net/ethernet/intel/hbl_en/Makefile | 12 +
>> .../net/ethernet/intel/hbl_en/common/Makefile | 3 +
>> .../net/ethernet/intel/hbl_en/common/hbl_en.c | 1170 +
>> .../net/ethernet/intel/hbl_en/common/hbl_en.h | 208 +
>> .../intel/hbl_en/common/hbl_en_dcbnl.c | 101 +
>> .../ethernet/intel/hbl_en/common/hbl_en_drv.c | 211 +
>> .../intel/hbl_en/common/hbl_en_ethtool.c | 452 +
>> .../net/ethernet/intel/hbl_en/gaudi2/Makefile | 2 +
>> .../ethernet/intel/hbl_en/gaudi2/gaudi2_en.c | 728 +
>> .../ethernet/intel/hbl_en/gaudi2/gaudi2_en.h | 53 +
>> .../intel/hbl_en/gaudi2/gaudi2_en_dcbnl.c | 32 +
>> include/linux/habanalabs/cpucp_if.h | 125 +-
>> include/linux/habanalabs/hl_boot_if.h | 9 +-
>> include/linux/net/intel/cn.h | 474 +
>> include/linux/net/intel/cn_aux.h | 298 +
>> include/linux/net/intel/cni.h | 636 +
>> include/linux/net/intel/gaudi2.h | 432 +
>> include/linux/net/intel/gaudi2_aux.h | 94 +
>> include/trace/events/habanalabs_cn.h | 116 +
>> include/uapi/drm/habanalabs_accel.h | 10 +-
>> include/uapi/rdma/hbl-abi.h | 204 +
>> include/uapi/rdma/hbl_user_ioctl_cmds.h | 66 +
>> include/uapi/rdma/hbl_user_ioctl_verbs.h | 106 +
>> include/uapi/rdma/ib_user_ioctl_verbs.h | 1 +
>> 146 files changed, 148514 insertions(+), 70 deletions(-)
>
> So most of these new lines are generated register definitions. The
> series can be several times smaller if you follow my advice.
>
> Thanks,
> Olek
Powered by blists - more mailing lists