第3章 GPU环境的操作系统、Docker和Kubernetes调优¶
即使拥有高度优化的GPU代码和库，系统级瓶颈也可能限制大规模AI训练的性能。最快的GPU只有在环境能够有效提供数据和指令时才能发挥最大作用。在本章中，我们将探讨如何调优操作系统和容器运行时，让GPU发挥其全部潜力。
Even with highly optimized GPU code and libraries, system-level bottlenecks can limit performance in large-scale AI training. The fastest GPU is only as good as the environment feeding it data and instructions. In this chapter, we explore how to tune the operating system and container runtime to let GPUs reach their full potential.
我们首先探索基础的GPU软件栈。然后深入探讨关键的CPU和内存优化，如NUMA亲和性和大页内存（hugepages）。这些确保数据能够高效地从存储通过CPU流向GPU。同时，我们讨论关键的GPU驱动设置，如持久化模式、多进程服务（MPS）和多实例GPU（MIG）分区。这些设置通过减少开销和有效同步资源来帮助保持最大的GPU利用率。
We begin by exploring the foundational GPU software stack. We then dive into key CPU and memory optimizations such as NUMA affinity and hugepages. These ensure that data flows efficiently from storage through the CPU to the GPU. In parallel, we discuss critical GPU driver settings like persistence mode, Multi-Process Service (MPS), and Multi-Instance GPU (MIG) partitions. These help maintain maximum GPU utilization by reducing overhead and synchronizing resources effectively.
使用NVIDIA Container Toolkit、Container Runtime、Kubernetes Topology Manager和Kubernetes GPU Operator等解决方案，你可以为GPU环境创建统一且高度优化的软件栈。这些解决方案支持单节点和多节点GPU环境中的高效资源分配和工作负载调度--并确保GPU能力得到充分利用。
Using solutions like the NVIDIA Container Toolkit, Container Runtime, Kubernetes Topology Manager, and Kubernetes GPU Operator, you can create a unified and highly optimized software stack for GPU environments. These solutions enable efficient resource allocation and workload scheduling across single-node and multinode GPU environments-and ensure GPU capabilities are fully utilized.
在这个过程中，你将建立对这些优化为何重要的直觉。本质上，它们最小化延迟，最大化吞吐量，并确保你的GPU不断获得数据并在峰值性能下运行。结果是一个健壮、可扩展的系统，为训练和推理工作负载提供显著的性能提升和高goodput百分比。
Along the way, you'll build intuition for why these optimizations matter. In essence, they minimize latency, maximize throughput, and ensure your GPUs are constantly fed with data and operating at their peak performance. The result is a robust, scalable system that delivers significant performance gains-and a high goodput percentage-for both training and inference workloads.
3.1 操作系统 (Operating System)¶
操作系统（OS）是一切运行的基础。GPU服务器通常运行Linux发行版，如Ubuntu Server LTS或Red Hat，并使用支持最新GPU硬件的更新内核。NVIDIA驱动安装内核模块，创建设备文件，如/dev/nvidia0、/dev/nvidia1和/dev/nvidia2--每个GPU一个。
The operating system (OS) is the foundation that everything runs on. GPU servers typically run a Linux distribution such as Ubuntu Server LTS or Red Hat with an updated kernel that supports the latest GPU hardware. The NVIDIA driver installs kernel modules that create device files like /dev/nvidia0, /dev/nvidia1, and /dev/nvidia2-one for each GPU.
驱动程序还创建/dev/nvidiactl用于驱动程序控制操作，/dev/nvidia-uvm用于统一虚拟内存，/dev/nvidia-modeset用于模式设置和缓冲区管理。
The driver also creates /dev/nvidiactl for driver control operations, /dev/nvidia-uvm for unified virtual memory, and /dev/nvidia-modeset for mode-setting and buffer management.
OS管理CPU调度、内存、网络和存储--所有这些都应该针对高GPU吞吐量进行调优。因此，OS应该配置为避免干扰GPU任务。例如，GPU节点应该禁用交换或将vm.swappiness设置为0，以避免任何OS发起的内存交换干扰GPU工作负载。
The OS manages CPU scheduling, memory, networking, and storage-all of which should be tuned for high GPU throughput. As such, the OS should be configured to avoid interfering with GPU tasks. For example, GPU nodes should disable swapping or set vm.swappiness to 0 to avoid any OS-initiated memory swapping that could interfere with GPU workloads.
作为性能工程师，我们工作的一部分是调整这些OS设置，使GPU达到最大性能。
Part of our job as performance engineers is to adjust these OS settings to set the GPUs up for maximum performance.
面向GPU的服务器可能需要运行额外的守护进程或后台进程，如NVIDIA Persistence Daemon以保持GPU驱动和硬件上下文加载并准备就绪--即使没有GPU作业运行。此外，Fabric Manager管理GPU互连拓扑。NVIDIA数据中心GPU管理器（DCGM）监控GPU系统健康指标。
A GPU-focused server might want to run additional daemons, or background processes, such as the NVIDIA Persistence Daemon to keep the GPU driver and hardware context loaded and ready-even when no GPU jobs are running. In addition, the Fabric Manager manages GPU interconnect topology. And the NVIDIA Data Center GPU Manager (DCGM) monitors GPU system health metrics.
3.2 NVIDIA软件栈 (NVIDIA Software Stack)¶
运行多petaFLOP GPU集群不仅仅涉及编写高级PyTorch、TensorFlow或JAX代码。支撑GPU操作的是整个软件栈，每一层都可能影响性能。图3-1显示了用于开发和生产化现代LLM工作负载的常见框架、库、编译器、运行时和工具集。
Running a multi-petaFLOP GPU cluster involves more than just writing high-level PyTorch, TensorFlow, or JAX code. There is a whole software stack underpinning GPU operations, and each layer can affect performance. Figure 3-1 shows a common set of frameworks, libraries, compilers, runtimes, and tools used to develop and productionize modern LLM workloads.
Figure 3-1. Common set of frameworks, libraries, compilers, runtimes, and tools used to develop and productionize modern LLM workloads
此外，NVIDIA GPU和CUDA生态系统拥抱Python库，允许你使用OpenAI的Triton领域特定语言（DSL）和NVIDIA的Warp框架等框架在Python中创建CUDA内核--以及NVIDIA的CUDA Python、cuTile和CUTLASS库。
In addition, the NVIDIA GPU and CUDA ecosystem embraces Python libraries and allows you to create CUDA kernels in Python using frameworks like OpenAI's Triton domain-specific language (DSL) and NVIDIA's Warp framework-as well as NVIDIA's CUDA Python, cuTile, and CUTLASS libraries.
3.2.1 GPU驱动 (GPU Driver)¶
基础是NVIDIA GPU驱动，它在Linux OS和GPU硬件之间建立接口。驱动管理低级GPU操作，包括设备上的内存分配、GPU核心上的任务调度，以及为多租户使用分区GPU。
At the base is the NVIDIA GPU driver, which interfaces between the Linux OS and the GPU hardware. The driver manages low-level GPU operations, including memory allocation on the device, task scheduling on GPU cores, and partitioning the GPU for multitenant usage.
GPU驱动开启GPU的功能并保持硬件有工作可做。保持NVIDIA驱动更新很重要。新驱动版本通常会解锁性能改进并支持最新的GPU架构和CUDA功能。nvidia-smi等工具随驱动提供，允许你监控温度、测量利用率、查询纠错码（ECC）内存状态，并启用不同的GPU模式。
The GPU driver turns on the GPUs' features and keeps the hardware fed with work. It's important to keep the NVIDIA driver up-to-date. New driver releases often unlock performance improvements and support the latest GPU architectures and CUDA features. Tools such as nvidia-smi come with the driver and allow you to monitor temperatures, measure utilization, query error-correcting code (ECC) memory status, and enable different GPU modes like persistence mode.
3.2.2 CUDA Toolkit和运行时 (CUDA Toolkit and Runtime)¶
在驱动之上是CUDA Runtime和称为CUDA Toolkit的库。工具包包括CUDA编译器nvcc，用于编译CUDA C++内核。编译后，CUDA程序链接到CUDA运行时（cudart）。CUDA运行时直接与NVIDIA驱动通信，在GPU上启动工作并分配内存。
On top of the driver sits the CUDA Runtime and libraries called the CUDA Toolkit. The toolkit includes the CUDA compiler, nvcc, used to compile CUDA C++ kernels. When compiled, CUDA programs link against the CUDA runtime (cudart). The CUDA runtime communicates directly with the NVIDIA driver to launch work and allocate memory on the GPU.
此外，CUDA Toolkit提供许多优化库：用于神经网络原语的cuDNN、用于线性代数的cuBLAS、用于多GPU通信的NCCL等。因此，使用支持GPU计算能力（CC）的最新CUDA Toolkit版本至关重要，因为最新的工具包具有针对你GPU的最新编译器优化和库。
Additionally, the CUDA Toolkit provides many optimized libraries: cuDNN for neural network primitives, cuBLAS for linear algebra, NCCL for multi-GPU communication, etc. As such, it's critical to use the latest CUDA Toolkit version that supports your GPU's compute capability (CC) since an up-to-date toolkit has the latest compiler optimizations and libraries specific to your GPU.
我们将在接下来的章节中更详细地介绍CUDA编译器和编程模型，以及CUDA（和PyTorch）优化。
We will cover the CUDA compiler and programming model-as well as CUDA (and PyTorch) optimizations-in more detail in the upcoming chapters.
3.2.3 CUDA跨GPU硬件代的向前和向后兼容性 (CUDA Forward and Backward Compatibility Across GPU Hardware Generations)¶
NVIDIA GPU编程模型的一个重要特性是其跨硬件代的兼容性。当你编译CUDA代码时，生成的二进制文件包含虚拟或中间PTX代码以及物理设备代码（例如，ARM、x86、GPU指令），如图3-2所示。
An important feature of NVIDIA's GPU programming model is its compatibility across hardware generations. When you compile CUDA code, the resulting binary includes virtual, or intermediate, PTX code as well as physical device code (e.g., ARM, x86, GPU instructions), as shown in Figure 3-2.
Figure 3-2. Using nvcc to compile a CUDA program into PTX-and ultimately the low-level instructions for the GPU target device
这允许较新的GPU即时（JIT）编译PTX，使你的程序能在未来架构上运行--并允许较新的GPU执行针对先前架构的旧二进制代码。这种兼容性通过NVIDIA的fatbinary模型实现，该模型包含用于面向未来的PTX和用于已知架构的CUBIN（或架构特定的CUDA设备代码二进制文件）。
This allows newer GPUs to just-in-time (JIT) compile the PTX so your program runs on future architectures-and allows newer GPUs to execute older binary code for prior architectures. This compatibility is achieved through NVIDIA's fatbinary model, which contains PTX for future-proofing and CUBIN, or architecture-specific CUDA device code binaries, for known architectures.
CUBIN是使用nvcc的-cubin选项生成的二进制文件。它包含为给定NVIDIA架构编译的GPU流汇编器（SASS）指令。它被打包成fatbinary，供CUDA驱动在运行时加载。与PTX（一种中间的、向前兼容的表示）不同，CUBIN二进制文件允许在已知GPU架构上直接执行。当与PTX一起包含在fatbinary中时，CUBIN支持为新GPU JIT编译PTX和在较新硬件上运行旧CUBIN代码。
CUBIN is the binary produced by nvcc using the -cubin option. It contains compiled GPU streaming assembler (SASS) instructions for a given NVIDIA architecture. It's packaged into a fatbinary for loading by the CUDA driver at runtime. Unlike PTX, which is an intermediate, forward-compatible representation, CUBIN binary files allow direct execution on known GPU architectures. When included alongside PTX in a fatbinary, CUBIN supports both JIT-compiling PTX for future GPUs and running older CUBIN code on newer hardware.
简而言之，当嵌入PTX时，CUDA提供向前兼容性，因为驱动可以在运行时为新架构JIT编译PTX。CUBIN对象是架构特定的，不能向前兼容未来的GPU架构，因此你应该包含PTX或发布包含当前架构SASS和用于向前兼容性的PTX的fat binary（也称为"fatbinaries"或简称"fatbins"）。
In short, CUDA provides forward compatibility when PTX is embedded because the driver can JIT-compile PTX for newer architectures at runtime. CUBIN objects are architecture-specific and are not forward-compatible to future GPU architectures, so you should include PTX or ship fat binaries (aka "fatbinaries" or just "fatbins") that contain both SASS for the current architectures and PTX for forward compatibility.
3.2.4 C++和Python CUDA库 (C++ and Python CUDA Libraries)¶
虽然大多数CUDA工具包库是C++，但NVIDIA当前面向Python的选项包括CUDA Python（例如，低级驱动和运行时访问）；cuPyNumeric、CuTe DSL、cuTile和CuPy用于数组编程；以及NVIDIA Warp用于在Python中编写GPU内核。CUTLASS是C++模板库，在cuBLAS等库的底层使用，而不是Python库。
While most CUDA toolkit libraries are C++, NVIDIA's current Python-facing options include CUDA Python (e.g., low-level driver and runtime access); cuPyNumeric, CuTe DSL, cuTile, and CuPy for array programming; and NVIDIA Warp for authoring GPU kernels in Python. CUTLASS is a C++ templated library used under the hood by libraries such as cuBLAS rather than a Python library.
虽然大多数CUDA Toolkit库是基于C++的，但越来越多基于Python的库从NVIDIA涌现，它们以"Cu"为前缀并基于C++工具包构建。例如，cuTile和cuPyNumeric是2025年初推出的Python库。它们的目标是降低Python开发者使用CUDA为NVIDIA GPU构建应用程序的门槛。
While most of the CUDA Toolkit libraries are C++ based, more and more Python-based libraries are emerging from NVIDIA that are prefixed with "Cu" and built upon the C++ toolkit. For instance, cuTile and cuPyNumeric are Python libraries launched in early 2025. They are targeted at lowering the barrier to entry for Python developers to build applications for NVIDIA GPUs using CUDA.
cuTile是一个Python库，旨在通过将大型矩阵分解为称为tile的更小、更易管理的子矩阵来简化GPU上的大型矩阵操作。它提供了高级的、基于tile的抽象，使执行块级计算、优化内存访问模式和高效调度GPU内核变得更加容易。通过将大型矩阵分成tile，cuTile帮助开发者充分利用GPU的并行性，而无需手动管理低级细节。这种方法可以改善缓存使用，并在需要密集矩阵计算的应用程序中提高整体性能。
cuTile is a Python library designed to simplify working with large matrices on GPUs by breaking them into smaller, more manageable submatrices called tiles. It provides a high-level, tile-based abstraction that makes it easier to perform block-wise computations, optimize memory access patterns, and efficiently schedule GPU kernels. By dividing a large matrix into tiles, cuTile helps developers take full advantage of the GPU's parallelism without needing to manage low-level details manually. This approach can lead to improved cache usage and overall better performance in applications that require intensive matrix computations.
cuPyNumeric是流行的numpy Python库的即插即用替代品，它利用GPU。它提供与NumPy几乎相同的函数、方法和行为，因此开发者通常只需对代码进行最小更改即可切换到它。在底层，cuPyNumeric利用CUDA在GPU上并行执行操作。这为大规模数值计算、矩阵操作和数据分析等计算密集型任务带来显著的性能提升。通过将工作卸载到GPU，cuPyNumeric加速计算并提高处理海量数据集的应用程序的效率。其目标是为Python开发者降低利用GPU能力的门槛，而无需学习全新的接口，使其成为高性能计算中NumPy的强大替代品。
cuPyNumeric is a drop-in replacement for the popular numpy Python library that utilizes the GPU. It provides nearly the same functions, methods, and behaviors as NumPy, so developers can often switch to it with minimal changes to their code. Under the hood, cuPyNumeric leverages CUDA to perform operations in parallel on the GPU. This leads to significant performance gains for compute-intensive tasks such as large-scale numerical computations, matrix operations, and data analysis. By offloading work to the GPU, cuPyNumeric accelerates computation and improves efficiency for applications handling massive datasets. Its goal is to lower the barrier for Python developers to harness GPU power without having to learn a completely new interface, making it a powerful drop-in alternative to NumPy for high-performance computing.
另一个值得注意的基于Python的编程模型是OpenAI的开源Triton语言和编译器。Triton是一个Python DSL，允许在Python中编写自定义GPU内核。虽然不是NVIDIA库，但Triton通过允许开发者直接在Python中编写高性能内核来补充CUDA。我们将在后面的章节中介绍Triton和各种基于Triton的优化，但只需知道Triton在许多情况下减少了对手工编写CUDA C++的需求。它已集成到PyTorch的编译器后端中，可自动优化和融合GPU操作以获得更好的性能。现在让我们转向PyTorch的讨论。
Another notable Python-based programming model is OpenAI's open source Triton language and compiler. Triton is a Python DSL that allows writing custom GPU kernels in Python. While not an NVIDIA library, Triton complements CUDA by allowing developers to write high-performance kernels directly in Python. We cover Triton and various Triton-based optimizations in a later chapter, but just know that Triton reduces the need for handwritten CUDA C++ in many cases. And it's integrated into PyTorch's compiler backend to automatically optimize and fuse GPU operations for better performance. Let's now turn the discussion to PyTorch.
3.2.5 PyTorch和高级AI框架 (PyTorch and Higher-Level AI Frameworks)¶
一些基于CUDA构建的流行Python框架包括PyTorch、TensorFlow、JAX和Keras。这些框架为深度学习提供高级接口，同时利用NVIDIA GPU的能力。本书主要关注PyTorch的编译和图优化功能，包括torch.compile栈。
Some popular Python-based frameworks built on CUDA are PyTorch, TensorFlow, JAX, and Keras. These frameworks provide high-level interfaces for deep learning while leveraging the power of NVIDIA GPUs. This book primarily focuses on PyTorch's compilation and graph optimization features, including the torch.compile stack.
PyTorch编译器栈由TorchDynamo、AOT Autograd和TorchInductor或加速线性代数（XLA）等后端组成，它们自动捕获和优化你的模型。TorchInductor是最常见的后端，它在底层使用OpenAI的Triton。Triton融合内核并为你的特定GPU和系统环境执行内核自动调优，我们将在第14章介绍。
The PyTorch compiler stack consists of TorchDynamo, AOT Autograd, and a backend like TorchInductor or Accelerated Linear Algebra (XLA), which automatically capture and optimize your models. TorchInductor is the most common backend, and it uses OpenAI's Triton under the hood. Triton fuses kernels and performs kernel autotuning for your specific GPU and system environment, as we'll cover in Chapter 14.
当你使用GPU对PyTorch张量执行操作时，它们从CPU移动到GPU，看起来像是单个Python调用。然而，这个单个调用实际上被转换为一系列对CUDA运行时的调用，利用各种CUDA库，如图3-3所示。
When you perform operations on PyTorch tensors using GPUs, they are moved from the CPU to the GPU in what appears to be a single Python call. However, this single call is actually translated into a series of calls to the CUDA runtime utilizing various CUDA libraries, as shown in Figure 3-3.
Figure 3-3. Flow from PyTorch code to GPU device
例如，当你执行矩阵乘法时，PyTorch将这些任务委托给cuBLAS等库。cuBLAS是CUDA Toolkit的一部分，针对GPU执行进行了优化。在幕后，PyTorch确保前向和后向传递等操作使用低级、优化的CUDA函数和库执行。
When you perform matrix multiplications, for example, PyTorch delegates these tasks to libraries such as cuBLAS. cuBLAS is part of the CUDA Toolkit and optimized for GPU execution. Behind the scenes, PyTorch ensures that operations like forward and backward passes are executed using low-level, optimized CUDA functions and libraries.
简而言之，PyTorch抽象了直接CUDA编程的复杂性，允许你编写直观的Python代码，最终调用高度优化的CUDA例程，既提供开发便捷性又提供高性能。我们将在第4章和第5章讨论CUDA编程和优化--以及第9章的PyTorch优化。
In short, PyTorch abstracts away the complexity of direct CUDA programming, allowing you to write intuitive Python code that ultimately calls highly optimized CUDA routines, delivering both ease of development and high performance. We will discuss CUDA programming and optimizations in Chapters 4 and 5-as well as PyTorch optimizations in Chapter 9.
所有这些组件--OS、GPU驱动、CUDA Toolkit、CUDA库和PyTorch--必须协同工作，才能创建理想的基于GPU的开发环境。当研究人员提交训练作业时，调度器保留节点，OS使用NVIDIA驱动提供GPU设备和内存分配，容器提供正确的软件环境（包括优化的、硬件感知的CUDA库）。用户代码（如PyTorch、TensorFlow、JAX）使用这些CUDA库，这些库最终与驱动和硬件通信。本章描述的优化旨在使这个栈的每一层尽可能高效。它们将帮助GPU保持忙碌于实际有用的训练和推理工作--而不是GPU等待CPU、等待内存或磁盘I/O，或等待其他GPU同步。
All of these components-OS, GPU Driver, CUDA Toolkit, CUDA libraries, and PyTorch-must work together to create the ideal GPU-based development environment. When a researcher submits a training job, the scheduler reserves nodes, the OS provides the GPU devices and memory allocations using the NVIDIA driver, and the container provides the correct software environment (including the optimized, hardware-aware CUDA libraries). The user code (e.g., PyTorch, TensorFlow, JAX) uses these CUDA libraries, which ultimately communicate with the driver and hardware. The optimizations described in this chapter are designed to make each layer of this stack as efficient as possible. They will help the GPUs stay busy with actual useful training and inference work-instead of the GPU waiting on the CPU, waiting for memory or disk I/O, or waiting on other GPUs to synchronize.
一个调优良好的系统确保跨数十个GPU分割的模型不会因I/O或OS开销而成为瓶颈。系统级调优经常被忽视，取而代之的是模型优化，但系统级优化可以产生显著的性能提升。在某些情况下，通过对OS级配置进行小调整，你可以获得两位数的百分比改进。在大型AI项目的规模下，这可以节省数万或数十万美元的计算时间。
A well-tuned system ensures that models split across dozens of GPUs are not bottlenecked by I/O or OS overhead. System-level tuning is often overlooked in favor of model optimizations, but system-level optimizations can yield substantial performance gains. In some cases, you can get double-digit percentage improvements with small tweaks to your OS-level configuration. At the scale of a big AI project, this can save tens or hundreds of thousands of dollars in compute time.
3.3 为GPU环境配置CPU和OS (Configuring the CPUs and OS for GPU Environments)¶
GPU无法达到充分利用的最常见原因之一是CPU没有为它们提供有用的工作。在典型的训练循环中，CPU负责准备下一批数据，包括从磁盘加载数据、对数据进行分词、转换数据等。此外，CPU负责调度GPU内核并协调线程和进程之间的工作。
One of the most common reasons that GPUs don't reach full utilization is that the CPU isn't keeping them fed with useful work. In a typical training loop, the CPU is responsible for preparing the next batch of data, including loading the data from disk, tokenizing the data, transforming it, etc. In addition, the CPU is responsible for dispatching GPU kernels and coordinating between threads and processes.
如果这些主机端任务很慢--或者OS调度它们不当--昂贵的GPU可能会发现自己空闲，等待下一个任务或数据批次。为了避免这种情况，我们需要优化CPU和OS处理GPU工作负载的方式。
If these host-side tasks are slow-or if the OS schedules them poorly-the expensive GPU can find itself idle, twiddling its transistors and waiting for the next task or batch of data. To avoid this, we need to optimize how the CPU and OS handle GPU workloads.
这些优化包括设置CPU亲和性以避免跨NUMA节点流量，使用内存分配策略避免NUMA惩罚，以及应用OS级更改消除不必要的延迟。这样，GPU永远不会因数据而饥饿。
These optimizations include setting the CPU affinity to avoid cross-NUMA-node traffic so the right cores handle the right data, using memory-allocation strategies to avoid NUMA penalties and applying OS-level changes to eliminate unnecessary latency. This way, the GPU is never starved for data.
这包括将后台守护进程和OS任务隔离在它们自己的核心上--远离向GPU提供数据的核心，我们接下来将讨论这一点。
Part of this involves isolating background daemons and OS tasks on their own cores-and away from the cores that feed the GPUs, which we'll discuss next.
3.3.1 NUMA感知和CPU绑定 (NUMA Awareness and CPU Pinning)¶
现代服务器CPU有数十个核心，通常分为多个NUMA节点。NUMA节点是CPU、GPU、网络接口控制器（NIC）和内存的逻辑分组，它们在物理上彼此靠近。了解系统的NUMA架构对性能调优很重要。在单个NUMA节点内访问资源比访问其他NUMA节点中的资源更快。
Modern server CPUs have dozens of cores and are often split into multiple NUMA nodes. A NUMA node is a logical grouping of CPUs, GPUs, network interface controllers (NICs), and memory that are physically close to one another. Being aware of the system's NUMA architecture is important for performance tuning. Accessing resources within a single NUMA node is faster than accessing resources in other NUMA nodes.
例如，如果运行在NUMA节点0中CPU上的进程需要访问NUMA节点1中的GPU，它需要跨节点链路发送数据，这将产生更高的延迟。实际上，跨NUMA节点时内存访问延迟几乎会翻倍。
For example, if a process running on a CPU in NUMA node 0 needs to access a GPU in NUMA node 1, it will need to send data across an internode link, which will incur higher latency. In fact, memory access latency can nearly double when crossing to the other NUMA nodes.
在基于Grace的超级芯片（如GH200和GB200）上，CPU和GPU通过NVLink-C2C连接，该链路在Grace与其配对的加速器之间提供高达约900 GB/s的一致CPU到GPU内存访问。Linux仍将CPU DRAM视为CPU NUMA内存，将GPU HBM视为设备内存。因此，你应该继续将CPU线程绑定到本地Grace CPU并尊重数据局部性，即使一致性降低了软件开销。
On Grace-based superchips such as GH200 and GB200, the CPU and GPU are linked by NVLink-C2C, which provides coherent CPU-to-GPU memory access at up to ~900 GB/s between Grace and its paired accelerator. Linux still treats CPU DRAM as CPU NUMA memory and GPU HBM as device memory. As such, you should continue to bind CPU threads to the local Grace CPU and respect data locality, even though coherence reduces software overheads.
在许多双插槽系统上，远程内存访问延迟可能显著高于本地内存访问。在一个实验中，本地NUMA节点内存访问延迟约为80 ns，而远程（跨节点）内存访问延迟约为139 ns。这大约是75%的延迟增加，这是本地和远程NUMA节点内存访问速度之间的巨大差异。
On many dual-socket systems, remote memory access latency can be significantly higher than local memory access. In one experiment, local NUMA node memory access latency is ~80 ns compared to remote (cross-node) memory access latency of ~139 ns. This is roughly a 75% increase in latency, which is a huge difference in access speed between local and remote NUMA node memory access.
通过将进程绑定到与其GPU位于同一NUMA节点的CPU，我们可以避免这种额外开销。例如，你可以使用numactl --cpunodebind=
By binding a process to a CPU on the same NUMA node as its GPU, we can avoid this extra overhead. For instance, you can use numactl --cpunodebind=
--membind= to bind both CPU threads and memory allocations to the GPU's local NUMA node. You'll learn more about this in a bit. The key idea is to keep CPU execution and memory access local to the GPU that it's serving.
虽然Linux包含基本的NUMA平衡，但对于性能关键的AI工作负载通常不够。默认情况下，进程可能会跨NUMA节点迁移。这将导致远程内存访问带来的额外延迟。因此，重要的是将进程和内存显式绑定到与本地GPU相同的NUMA节点。你可以使用numactl、taskset或cgroups来实现，我们将在稍后展示。
While Linux includes basic NUMA balancing, it's usually not sufficient for performance-critical AI workloads. By default, processes may be migrated across NUMA nodes. This will lead to additional latency caused by remote memory accesses. As such, it's important to explicitly bind processes and memory to the same NUMA node as the local GPU. You can do this using numactl, taskset, or cgroups, as we'll show in a bit.
要显式指定NUMA亲和性，你需要将进程或线程"绑定"到与GPU连接到同一NUMA节点的特定CPU。这种类型的CPU亲和性称为CPU绑定。假设你在一个节点中有八个GPU，其中四个GPU连接到NUMA节点0，另外四个连接到NUMA节点1。
To explicitly specify NUMA-affinity, you need to "pin" processes or threads to specific CPUs that are connected to the same NUMA node as the GPU. This type of CPU affinity is called CPU pinning. Suppose you have eight GPUs in a node, with four GPUs connected to NUMA node 0 and the other four to NUMA node 1.
如果你启动八个训练进程，每个GPU一个，你应该将每个训练进程绑定到与GPU连接到同一NUMA节点的CPU核心或CPU核心集。在这种情况下，GPU 0-3连接到NUMA节点0，GPU 4-7连接到NUMA节点1的核心，如图3-4所示。
If you launch eight training processes, one per GPU, you should bind each training process to a CPU core-or set of CPU cores-connected to the same NUMA node as the GPUs. In this case, GPUs 0-3 are connected to NUMA node 0 and GPUs 4-7 are connected to NUMA node 1's cores, as shown in Figure 3-4.
Figure 3-4. Eight GPUs in a node, with four GPUs connected to NUMA node 0 and the other four to NUMA node 1
这样，当CPU进程想要向GPU 4提供数据时，它应该运行在连接到NUMA节点1的CPU上，因为GPU 4连接到NUMA节点1。Linux提供了执行此操作的工具，包括numactl --cpunodebind=
This way, when a CPU process wants to feed data to GPU 4, it should be running on a CPU connected to NUMA node 1 since GPU 4 is connected to NUMA node 1. Linux provides tools to do this, including numactl --cpunodebind=
--membind= , which launches a process pinned to the given NUMA node.
你也可以使用taskset将进程绑定到特定的核心ID。以下是使用numactl将train.py脚本绑定到与GPU 4在同一NUMA节点1中运行的CPU的示例：
You can also use taskset to pin processes to specific core IDs. Here is an example using numactl to bind the train.py script to a CPU running in the same NUMA node 1 as GPU 4:
这假设我们知道NUMA节点ID，并且我们只将脚本绑定到一个GPU。将train.py绑定到多个GPU到未知NUMA节点要复杂一些。
This assumes we know the NUMA node ID and that we are binding the script to only one GPU. Binding the train.py to multiple GPUs to an unknown NUMA node is a bit more complicated.
以下脚本使用nvidia-smi topo动态查询拓扑，并使用本地NUMA节点将脚本绑定到GPU：
#!/bin/bash
for GPU in 0 1 2 3; do
# Query NUMA node for this GPU
NODE=$(nvidia-smi topo -m -i $GPU \
| awk '/NUMA Affinity/ {print $NF}')
# Launch the training process pinned to that NUMA node
numactl --cpunodebind=$NODE --membind=$NODE \
bash -c "CUDA_VISIBLE_DEVICES=$GPU python train.py --gpu $GPU"
done
The following script dynamically queries the topology using nvidia-smi topo and binds the script to GPUs using the local NUMA node:
这里，我们使用topo -m获取CPU和NUMA亲和性。然后我们从NUMA Affinity列提取单个节点ID。最后，我们将--cpunodebind和--membind都绑定到该节点，以确保进程的线程和内存分配保持在GPU的NUMA域内本地。
Here, we use topo -m to get both CPU and NUMA affinities. We then extract a single-node ID from the NUMA Affinity column. Finally, we bind both --cpunodebind and --membind to that node to ensure your process's threads and memory allocations stay local to the GPU's NUMA domain.
许多深度学习框架还允许你以编程方式设置线程亲和性。例如，PyTorch的DataLoader暴露worker_init_fn，以便你可以在初始化期间为每个工作进程设置CPU亲和性，如下所示：
Many deep learning frameworks also let you set thread affinities programmatically. For instance, PyTorch's DataLoader exposes worker_init_fn so you can set CPU affinity for each worker process during initialization, as shown here:
import os
import re
import glob
import subprocess
import psutil
import ctypes
import torch
import torch.distributed as dist
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DataLoader, Dataset
from functools import partial
# Optional: NVML is preferred for GPU↔NUMA mapping
try:
import pynvml as nvml # pip install nvidia-ml-py3
_HAS_NVML = True
except Exception:
_HAS_NVML = False
# --- libnuma for memory binding
_libnuma = ctypes.CDLL("libnuma.so")
if _libnuma.numa_available() < 0:
raise RuntimeError("NUMA not available on this system")
_libnuma.numa_run_on_node.argtypes = [ctypes.c_int]
_libnuma.numa_set_preferred.argtypes = [ctypes.c_int]
def parse_physical_cpu_list(phys_str: str):
"""Parse '0-3,8-11' -> [0,1,2,3,8,9,10,11]."""
cpus = []
if not phys_str:
return cpus
for part in phys_str.split(','):
part = part.strip()
if not part:
continue
if '-' in part:
start, end = map(int, part.split('-'))
cpus.extend(range(start, end + 1))
else:
cpus.append(int(part))
return cpus
def get_numa_cpus_for_node(node: int):
"""Read /sys/devices/system/node/node{node}/cpulist."""
path = f"/sys/devices/system/node/node{node}/cpulist"
with open(path, "r") as f:
return parse_physical_cpu_list(f.read().strip())
def get_numa_cpus_and_memory():
"""Return (current_cpu_mask, preferred_node) from numactl --show."""
out = subprocess.run(["numactl", "--show"],
capture_output=True, text=True).stdout
phys = re.search(r"physcpubind:\s*([\d,\-\s]+)", out).group(1)
cpus = parse_physical_cpu_list(phys)
node = int(re.search(r"preferred node:\s*(-?\d+)", out).group(1))
return cpus, node
def get_gpu_numa_node(device: int) -> int:
"""
Determine NUMA node for a GPU (prefer NVML; fall back to sysfs;
final fallback to current preferred node).
"""
# NVML path (preferred)
if _HAS_NVML:
try:
nvml.nvmlInit()
props = torch.cuda.get_device_properties(device)
pci = props.pci_bus_id # '0000:03:00.0' or '00000000:03:00.0'
# Normalize to 8-hex-digit domain if needed for NVML
try:
domain, bus, devfn = pci.split(':')
if len(domain) < 8:
domain = domain.rjust(8, '0')
pci8 = f"{domain}:{bus}:{devfn}"
except ValueError:
pci8 = pci
try:
handle = nvml.nvmlDeviceGetHandleByPciBusId_v2(pci8)
except AttributeError:
handle = nvml.nvmlDeviceGetHandleByPciBusId(pci8)
# Direct NUMA ID if driver exposes it
try:
numa_id = nvml.nvmlDeviceGetNUMANodeId(handle)
if isinstance(numa_id, int) and numa_id >= 0:
return numa_id
except Exception:
pass
# Derive from NVML CPU affinity
cpu_count = psutil.cpu_count(logical=True)
elems = (cpu_count + 63) // 64
mask = nvml.nvmlDeviceGetCpuAffinity(handle, elems)
cpus = []
for i, m in enumerate(mask):
m = int(m)
for b in range(64):
if m & (1 << b):
cpu_id = i * 64 + b
if cpu_id < cpu_count:
cpus.append(cpu_id)
# Build CPU→NUMA map from sysfs and choose majority node
cpu2node = {}
for node_path in sorted(glob.glob("/sys/devices/system/node/node*")):
node_id = int(os.path.basename(node_path).replace("node", ""))
with open(os.path.join(node_path, "cpulist"), "r") as f:
for c in parse_physical_cpu_list(f.read().strip()):
cpu2node[c] = node_id
counts = {}
for c in cpus:
n = cpu2node.get(c)
if n is not None:
counts[n] = counts.get(n, 0) + 1
if counts:
return max(counts.items(), key=lambda kv: kv[1])[0]
except Exception:
pass
# sysfs fallback
try:
props = torch.cuda.get_device_properties(device)
pci = props.pci_bus_id
sysfs_path = f"/sys/bus/pci/devices/{pci}/numa_node"
with open(sysfs_path, "r") as f:
val = int(f.read().strip())
return val if val >= 0 else 0
except Exception:
pass
# last resort: current preferred node
_, node = get_numa_cpus_and_memory()
return node if node >= 0 else 0
def set_numa_affinity(node: int):
"""Bind current process to CPUs and memory of the given NUMA node."""
cpus = get_numa_cpus_for_node(node) # IMPORTANT: CPUs of target node
psutil.Process(os.getpid()).cpu_affinity(cpus)
_libnuma.numa_run_on_node(node)
_libnuma.numa_set_preferred(node)
print(f"PID={os.getpid()} bound to NUMA node {node} (CPUs={cpus})")
return cpus
def _worker_init_fn(worker_id: int, node: int, cpus: list):
"""Reapply binding in each DataLoader worker (no CUDA calls here)."""
psutil.Process(os.getpid()).cpu_affinity(cpus)
_libnuma.numa_run_on_node(node)
_libnuma.numa_set_preferred(node)
print(f"Worker {worker_id} (PID={os.getpid()}) bound to NUMA node {node}")
# ----- Example usage below -----
class MyDataset(Dataset):
def __len__(self): return 1024
def __getitem__(self, idx): return torch.randn(224*224*3, device='cpu')
def main():
# DDP setup
dist.init_process_group(backend="nccl", init_method="env://")
device = torch.cuda.current_device()
# Determine GPU's NUMA node and bind this process
gpu_node = get_gpu_numa_node(device)
cpus = set_numa_affinity(gpu_node)
# Build dataloader with closure-based worker_init_fn
dataset = MyDataset()
init_fn = partial(_worker_init_fn, node=gpu_node, cpus=cpus)
dataloader = DataLoader(
dataset,
batch_size=32,
num_workers=4,
pin_memory=True,
persistent_workers=True, # reduces worker respawn churn
worker_init_fn=init_fn,
prefetch_factor=2,
)
# Model and DDP
model = torch.nn.Linear(224*224*3, 10, bias=True).to("cuda")
ddp_model = DDP(model, device_ids=[device], static_graph=True)
for batch in dataloader:
batch = batch.to("cuda", non_blocking=True)
out = ddp_model(batch)
# ... loss, backward, optimizer ...
if __name__ == "__main__":
main()
此脚本将主训练进程和每个DataLoader工作进程绑定到GPU的本地NUMA节点，以防止跨NUMA内存访问。在DataLoader中，我们传递一个基于闭包的worker_init_fn，在每个工作进程内重新应用预先计算的NUMA绑定。我们在工作进程中不调用任何CUDA API。
This script binds the main training process and each DataLoader worker process to the GPU's local NUMA node to prevent cross-NUMA memory access. In the DataLoader, we pass a closure-based worker_init_fn that reapplies the precomputed NUMA binding inside each worker. And we do this without touching any CUDA APIs in the worker.
启动时，进程使用NVML将当前GPU映射到其NUMA节点和CPU亲和性掩码。当可用时，我们通过nvmlDeviceGetNUMANodeId直接读取节点。否则，我们从GPU的CPU亲和性掩码（nvmlDeviceGetCpuAffinity）推导它。如果NVML不可用或未暴露节点，我们回退到内核的sysfs条目/sys/bus/pci/devices/
At startup, the process uses NVML to map the current GPU to its NUMA node and CPU-affinity mask. When available, we read the node directly via nvmlDeviceGetNUMANodeId. Otherwise, we derive it from the GPU's CPU-affinity mask (nvmlDeviceGetCpuAffinity). If NVML is unavailable or does not expose the node, we fall back to the kernel's sysfs entry at /sys/bus/pci/devices/
/numa_node. As a last resort, we use the process's current preferred node.
然后我们从/sys/devices/system/node/node
We then compute the CPU list for that node from /sys/devices/system/node/node
/cpulist and apply CPU affinity to those cores with psutil. We also bind all future allocations to that node using libnuma (numa_run_on_node + numa_set_preferred).
由于某些启动器、容器运行时或内核不能可靠地将NUMA策略传播给子进程，我们在每个fork的工作进程中显式重新应用并验证绑定。仅依赖继承是不安全的。
Because some launchers, container runtimes, or kernels do not reliably propagate NUMA policy to children, we explicitly reapply and verify the binding in every forked worker. It's not safe to rely on inheritance alone.
记得设置pin_memory=True并在H2D复制上使用non_blocking=True，以便页锁定的主机缓冲区保持在正确的NUMA节点上。优先使用persistent_workers=True以避免重新fork工作进程并在epoch之间丢失它们的亲和性。不要在worker_init_fn中调用torch.cuda.*。相反，使用闭包或环境变量传递GPU索引。
Remember to set pin_memory=True and use non_blocking=True on H2D copies so that page-locked host buffers stay on the correct NUMA node. Prefer persistent_workers=True to avoid re-forking workers and losing their affinity between epochs. And do not call torch.cuda.* in worker_init_fn. Instead, pass the GPU index using a closure or environment variable.
结果是数据准备和批次加载完全在本地内存中发生。这样，你的GPU保持忙碌，永远不需要为远程NUMA跳转而暂停。使用此代码，你可以在任何安装了libnuma和numactl的Linux服务器上获得健壮的、拓扑感知的亲和性。
The result is that data preparation and batch loading happen entirely in local memory. This way, your GPUs stay busy and never need to pause for a remote‐NUMA hop. With this code, you get robust, topology‐aware affinity on any Linux server with libnuma and numactl installed.
默认情况下，numactl将其CPU和内存策略应用于进程，并记录为将策略继承给所有fork的子进程。然而，在实践中，Python框架生成的线程或exec的子进程并不总是在每个内核或Linux发行版上获得相同的设置。当使用框架管理的工作进程时，你应该在每个工作进程内显式重新声明CPU和内存策略。
By default, numactl applies its CPU and memory policy to a process and is documented to inherit that policy to all forked children. In practice, however, threads spawned by Python frameworks or exec'd subprocesses don't always pick up the same settings on every kernel or Linux distribution. When using framework-managed worker processes, you should explicitly reassert the CPU and memory policy inside of each worker.
对于像Grace Blackwell（和Vera Rubin）这样的超级芯片架构，CPU和GPU使用NVLink-C2C保持一致。然而，Linux仍将CPU DRAM和GPU HBM建模为独立的池。将CPU线程绑定到本地CPU NUMA节点对于局部性仍然有益。
With a superchip architecture like Grace Blackwell (and Vera Rubin), the CPU and GPU are coherent using NVLink-C2C. However, Linux still models CPU DRAM and GPU HBM as separate pools. Binding CPU threads to the local CPU NUMA node still remains beneficial for locality.
在实践中，绑定可以消除不可预测的CPU调度行为。它确保关键线程（如GPU的数据加载线程）不会在训练或推理过程中突然被OS迁移到不同NUMA节点上的核心。在实践中，仅通过消除跨NUMA流量和CPU核心迁移，就可能看到5%-10%的训练吞吐量提升。这也倾向于减少性能抖动和方差。
In practice, pinning can eliminate unpredictable CPU scheduling behavior. It ensures that a critical thread such as a data-loading thread for your GPU doesn't suddenly get migrated by the OS to a core on a different NUMA node in the middle of training or inferencing. In practice, it's possible to see 5%-10% training throughput improvements just by eliminating cross-NUMA traffic and CPU core migrations. This also tends to reduce performance jitter and variance.
许多高性能AI系统评估CPU同时多线程（SMT）--通常称为超线程--有时会禁用它以获得更可预测的每核性能，但好处取决于工作负载。这些系统还可能通过设置isolcpus内核参数将少量核心专门保留给OS后台任务，以将它们与通用调度器隔离。你也可以使用Kubernetes CPU隔离来处理系统守护进程。这确保剩余的核心完全专用于训练和推理线程并执行有用的工作。
Many high-performance AI systems evaluate CPU simultaneous multithreading (SMT), or hyperthreading as it's often called-and sometimes disable it for more predictable per-core performance, but the benefit is workload-dependent. These systems may also reserve a handful of cores exclusively for OS background tasks by setting the isolcpus kernel parameter to isolate them from the general scheduler. You can also use Kubernetes CPU isolation for system daemons. This ensures that the remaining cores are dedicated entirely to training and inference threads and doing useful work.
需要注意的是，对于像NVIDIA Grace Blackwell这样的集成CPU-GPU超级芯片，许多关于CPU到GPU数据传输的传统担忧得到了缓解，因为CPU和GPU通过NVLink-C2C暴露一致共享虚拟地址空间，而CPU DRAM和GPU HBM仍然是独立的内存池。这意味着跨NUMA延迟等问题被最小化，数据可以在CPU和GPU之间更直接地流动。
It's important to note that for integrated CPU-GPU superchips like NVIDIA's Grace Blackwell, many of the traditional concerns about CPU-to-GPU data transfer are alleviated because the CPU and GPU expose a coherent shared virtual address space over NVLink-C2C, while CPU DRAM and GPU HBM remain distinct memory pools. This means that issues like cross-NUMA delays are minimized, and the data can flow more directly between the CPU and GPU.
NVIDIA通过将CPU和GPU组合到单个超级芯片（如Grace Blackwell架构）上来解决CPU到GPU瓶颈并非巧合。在这种设计中，CPU和GPU甚至使用NVLink-C2C共享统一、一致的内存，速度高达900 GB/s，这最小化了数据传输开销。期望NVIDIA继续通过更多此类与软件和算法需求协同设计的硬件创新来解决系统瓶颈。
It's not a coincidence that NVIDIA tackled the CPU-to-GPU bottleneck by combining the CPU and GPU onto a single superchip such as the Grace Blackwell architecture. In this design, the CPU and GPU even share a unified, coherent memory using NVLink-C2C at up to 900 GB/s, which minimizes data transfer overhead. Expect NVIDIA to continue addressing system bottlenecks with more of these types of hardware innovations codesigned with the needs of software and algorithms.
即使采用紧密耦合的CPU-GPU超级芯片架构，仍然需要通过确保硬件和软件正确配置来优化栈，以便集成系统以峰值效率运行。即使在这些紧密耦合的架构中，你也希望最小化数据处理中的任何不必要延迟，以保持GPU充分利用。这包括配置大页内存、使用效率预取和固定内存，如下几节所示。
Even with the tightly coupled CPU-GPU superchip architecture, it's still important to optimize the stack by ensuring that the hardware and software are configured properly so that the integrated system operates at peak efficiency. Even in these tightly coupled architectures, you want to minimize any unnecessary delays in data handling to keep the GPU fully utilized. This includes configuring hugepages, using efficiency prefetching, and pinning memory, as you will see in the next sections.
3.3.2 NUMA友好的内存分配和内存固定 (NUMA-Friendly Memory Allocation and Memory Pinning)¶
默认情况下，进程将从其当前运行的CPU的NUMA节点分配内存。因此，如果你将进程绑定到NUMA节点0，其内存将自然来自NUMA节点0的本地RAM，这是理想的。但是，如果OS调度器迁移线程--或者如果你在绑定之前分配了一些内存--你可能会遇到进程运行在NUMA节点0但使用NUMA节点1内存的非理想情况。在这种情况下，每次内存访问都必须跳到另一个NUMA节点，这抵消了CPU绑定的好处。
By default, a process will allocate memory from the NUMA node of the CPU it's currently running on. So if you pin a process to NUMA node 0, its memory will naturally come from NUMA node 0's local RAM, which is ideal. However, if the OS scheduler migrates threads-or if some memory was allocated before you did the pinning-you could end up with the nonideal scenario in which a process running in NUMA node 0 is using memory from NUMA node 1. In this case, every memory access has to hop to the other NUMA node, negating the benefit of CPU pinning.
为了避免这种情况，numactl --membind选项强制从特定NUMA节点分配内存，如前面章节所述。在代码中，也有NUMA API甚至环境变量可以影响此配置。一般规则是保持内存靠近CPU，CPU靠近GPU。这样，从内存到CPU到GPU的数据移动链都在单个NUMA节点内。以下是使用--membind=1强制从包括NUMA节点1的首选NUMA节点分配内存的示例：
To avoid this, the numactl --membind option forces memory allocation from a specific NUMA node, as mentioned in an earlier section. In code, there are also NUMA APIs or even environment variables that can influence this configuration. The general rule is to keep memory close to the CPU, which is close to the GPU. That way the chain of data movement from memory to CPU to GPU is all within a single NUMA node. Here is the same example as before but with --membind=1 to force memory allocation from the preferred NUMA node that includes NUMA node 1:
需要注意的是，当你在numactl下启动进程时，其CPU（--cpunodebind）和内存策略（--membind）都应用于该进程，并由其所有子进程继承。因此，你的训练脚本fork的任何工作子进程将自动使用相同的NUMA内存绑定。但是，它们必须使用基于fork的模型创建。如果你切换到spawn启动方法，或以其他方式exec新程序，这些子进程不会继承父进程的内存策略。
It's important to note that when you launch a process under numactl, both its CPU (--cpunodebind) and memory policies (--membind) are applied to that process and inherited by all of its child processes. As such, any worker subprocesses forked by your training script will automatically use the same NUMA memory binding. However, they must be created using a fork-based model. If you switch to a spawn start method, or otherwise exec a new program, those child processes do not inherit the parent's memory policy.
此外，固定内存（也称为页锁定内存）对于高效的直接GPU访问至关重要。当内存被固定时，OS不会交换或移动它。这导致更快的直接内存访问（DMA）传输。从固定主机内存复制数据到GPU可以比从常规可分页内存快2-3倍，因为GPU或NIC可以直接执行DMA。
In addition, pinned memory, also called page-locked memory, is essential for efficient and direct GPU access. When memory is pinned, the OS won't swap or move it. This leads to faster direct memory access (DMA) transfers. Copying data from pinned host memory to GPU can be 2-3x faster than from regular pageable memory since the GPU or NIC can perform DMA directly.
你可以使用已安装的CUDA工具中的bandwidthTest --memory=
You can test the data-transfer bandwidth between CPU memory and GPU memory using bandwidthTest --memory=
from the installed CUDA utilities.
事实上，这是NVIDIA GPUDirect技术的基础，例如GPUDirect RDMA，它允许InfiniBand等NIC直接与GPU内存交换数据。类似地，GPUDirect Storage（GDS）允许NVMe驱动器将数据流式传输到GPU内存，而无需额外的CPU开销。
In fact, this is the basis of NVIDIA's GPUDirect technologies such as GPUDirect RDMA, which allows NICs like InfiniBand to directly exchange data with GPU memory. Similarly, GPUDirect Storage (GDS) allows NVMe drives to stream data into GPU memory without extra CPU overhead.
深度学习框架提供选项为数据加载器使用固定内存。例如，PyTorch的DataLoader有一个标志pin_memory=True，当为true时，意味着加载的批次将被放置在固定RAM中，如图3-5所示。
Deep learning frameworks provide options to use pinned memory for data loaders. For example, PyTorch's DataLoader has a flag pin_memory=True, which, when true, means the batches loaded will be placed in pinned RAM, as shown in Figure 3-5.
Figure 3-5. Pinned memory (aka page-locked or nonpageable) is a type of memory that cannot be swapped out to disk
内存固定加速了tensor.to(device)操作，因为CUDA驱动程序不必即时固定页面。当你使用大批量大小或每次迭代读取大量数据时，这尤其有益。许多从业者注意到，在PyTorch中仅启用pin_memory=True就可以通过减少数据传输瓶颈和提高主机到设备传输吞吐量来提高性能10%-20%。
Memory pinning speeds up the tensor.to(device) operations because the CUDA driver doesn't have to pin pages on the fly. It's especially beneficial when you are using large batch sizes or reading a lot of data in each iteration. Many practitioners have noticed that just turning on pin_memory=True in PyTorch can improve performance up to 10%-20% by reducing data transfer bottlenecks and increasing host-to-device transfer throughput.
简而言之，你应该确保数据加载器使用固定内存（例如，PyTorch DataLoader中的pin_memory=True），并为支持的硬件启用GPUDirect RDMA和GDS。这将减少数据传输延迟。
In short, you should make sure that your data loader uses pinned memory (e.g., pin_memory=True in PyTorch DataLoader) and that GPUDirect RDMA and GDS are enabled for supported hardware. This will reduce data transfer latency.
需要注意的是，OS对用户可以锁定（固定）的内存量有限制。这是通过ulimit -l
It's important to note that the OS has a limit on how much memory a user can lock (pin). This is set with the ulimit -l
command. In containerized environments, you can adjust the container's security context and Docker --ulimit memlock setting accordingly. This way, the container can lock sufficient memory.
如果你计划使用大型固定缓冲区，请确保ulimit值较高--或将其设置为无限制。否则分配可能会失败。通常，对于大型AI工作负载和高性能计算（HPC）应用程序，将其设置为无限制。
If you plan to use large, pinned buffers, ensure the ulimit value is high-or set it to unlimited. Otherwise the allocation might fail. Typically, one sets it to unlimited for large AI workloads and high-performance computing (HPC) applications.
3.3.3 透明大页内存 (Transparent Hugepages)¶
除了固定内存并将其绑定到NUMA节点外，我们还应该讨论透明大页内存（THP）。Linux内存管理通常使用4 KB页面，但当你有使用数十或数百GB内存的进程时（如深度学习数据集、预取批次、模型参数等），管理数百万个小页面是低效的。
In addition to pinning memory and binding it to NUMA nodes, we should talk about transparent hugepages (THPs). Linux memory management typically uses 4 KB pages, but managing millions of tiny pages is inefficient when you have processes using tens or hundreds of gigabytes of memory, as in the case of deep learning datasets, prefetched batches, model parameters, etc.
大页内存-2 MB甚至1 GB页面-可以通过使内存块更大来减少虚拟内存管理的开销。主要好处是更少的页面错误和更少的转换后备缓冲区（TLB）压力。TLB是CPU用于将虚拟地址映射到物理地址的缓存。更少、更大的页面意味着TLB可以用相同数量的条目覆盖更多内存，减少未命中。
Hugepages-2 MB or even 1 GB pages-can reduce the overhead of virtual memory management by making memory chunks bigger. The main benefits are fewer page faults and less pressure on the translation lookaside Buffer (TLB). The TLB is a cache that the CPU uses to map virtual addresses to physical ones. Fewer, larger pages means the TLB can cover more memory with the same number of entries, reducing misses.
大页内存通常产生适度的改进--通常约3%-5%的吞吐量提升。它们通过减少页面错误开销和TLB压力来实现这一点。启用THP在大多数系统上是一个简单的胜利，因为内核会自动用2 MB页面支持大型分配。在具有非常大内存池的场景中（例如，用于I/O的预分配固定缓冲区），你也可以考虑使用vm.nr_hugepages或hugetlbfs进行显式大页内存，以获得更确定性的性能。
Hugepages typically produce modest gains-often on the order of ~3%-5% throughput improvement. They do this by reducing page-fault overhead and TLB pressure. Enabling THP is a simple win on most systems since the kernel will automatically back large allocations with 2 MB pages. In scenarios with very large memory pools (e.g., preallocated pinned buffers for I/O), you may also consider explicit hugepages using vm.nr_hugepages or hugetlbfs for more deterministic performance.
当使用大型固定内存区域时，你应该将ulimit -l设置（最大锁定内存）提高到较高值或无限制。如果此限制太低，你固定内存的尝试可能会失败，导致回退到可交换内存--或内存不足（OOM）错误。
Remember that, when using large, pinned memory regions, you should raise the ulimit -l setting (max locked memory) to a high value or unlimited. If this limit is too low, your attempt to pin memory can fail, leading to fallback on swappable memory-or out-of-memory (OOM) errors.
需要注意的是，THP的后台压缩可能会引入不可预测的停顿，这对延迟敏感的LLM推理工作负载是灾难性的。Linux默认配置为使用THP在可能的情况下自动分配2 MB页面。这通常足够，但值得为你的工作负载进行测试。
It's important to note that THP's background compaction can introduce unpredictable pauses that are disastrous for latency-sensitive LLM inference workloads. Linux is configured by default to use THP to automatically allocate 2 MB pages whenever possible. This is often sufficient, but it's worth testing for your workload.
你可以禁用THP，但你需要手动分配和控制大页内存。这会带来额外的复杂性，但对于推理等低延迟工作负载可能是必需的。禁用THP后，你的系统将避免内核驱动的碎片整理引起的停顿。
You can disable THP, but you will need to manually allocate and control hugepages. This will incur extra complexity, but it might be needed for low-latency workloads like inference. With THP disabled, your system will avoid stalls caused by kernel-driven defragmentations.
现代共识是为大多数基于GPU的训练工作负载启用THP，其中吞吐量很重要；对于推理等延迟重要的工作负载，完全禁用THP（transparent_hugepage=never）--或使用madvise。对于许多rank（GPU）同时分配内存的分布式训练工作负载也是如此。
The modern consensus is to enable THP for most GPU-based training workloads in which throughput is important and to disable THP completely (transparent_hugepage=never)-or use madvise-for workloads like inference in which latency is important. This is also true for distributed training workloads in which many ranks (GPUs) allocate memory simultaneously.
除了CPU/内存绑定和大页内存外，还有一些其他OS级调整值得一提。这些包括线程调度、虚拟内存管理、文件系统缓存和CPU频率设置，我们将在接下来的几节中介绍。
Beyond CPU/memory pinning and hugepages, there are a few other OS-level tweaks worth mentioning. These include thread scheduling, virtual memory management, filesystem caching, and CPU frequency settings, which we'll cover in the next few sections.
3.3.4 调度器和中断亲和性 (Scheduler and Interrupt Affinity)¶
在繁忙的系统上，你需要确保重要线程（如数据管道线程）不会被频繁中断。Linux默认使用完全公平调度器（CFS），在大多数情况下工作良好。但如果你有一个非常延迟敏感的线程向GPU提供数据，你可以考虑为该线程使用实时先进先出（FIFO）或轮询（RR）优先级调度。这将确保高优先级线程运行而不被普通优先级线程抢占。
On a busy system, you want to make sure that important threads such as data-pipeline threads aren't interrupted frequently. Linux by default uses the Completely Fair Scheduler (CFS) that works well for most cases. But if you have a very latency-sensitive thread that feeds the GPU with data, for example, you could consider using real-time first in, first out (FIFO) or round-robin (RR) priority scheduling for that thread. This would make sure that the high-priority thread runs without being preempted by normal-priority threads.
但是，请谨慎使用，因为如果管理不当，实时线程可能会使其他进程饥饿。然而，在实践中，如果你已经将线程绑定到专用核心，通常不需要调整实时线程优先级，但值得留意。
However, use this with caution, as real-time threads can starve other processes if not managed properly. In practice, however, if you've pinned your threads to dedicated cores, you often don't need to mess with real-time thread priorities, but it's worth keeping an eye on.
另一个选择是隔离核心或创建单独的CPU分区，以进一步减少这些专用计算资源上的中断。为此，你可以使用cset、内核参数如isolcpus和nohz_full，或cgroup cpuset隔离。通过隔离，OS调度器将这些CPU核心留给你随意使用。
Another option is to isolate cores or create separate CPU partitions to further reduce interruptions on these dedicated compute resources. To do this, you can use cset, kernel parameters like isolcpus and nohz_full, or cgroup cpuset isolation. With isolation, the OS scheduler leaves those CPU cores for you to use as you wish.
在生产环境中强烈推荐使用cgroup CPU和内存亲和性。使用这些，每个AI工作负载都在自己的物理核心和内存区域上隔离。这将防止跨工作负载争用和NUMA惩罚。应该使用cpuset cgroups或容器运行时（docker --cpuset-cpus）等工具来强制执行。
cgroup CPU and memory affinity is strongly recommended in production environments. Using these, each AI workload is isolated on its own physical cores and memory regions. This will prevent cross-workload contention and NUMA penalties. Tools like cpuset cgroups or container runtimes (docker --cpuset-cpus) should be used to enforce this.
你可以将每个设备的硬件中断分配给同一NUMA节点上的核心。这将防止跨节点中断处理，否则会产生额外延迟并驱逐远程节点上有用的缓存行。例如，如果NUMA节点0上的GPU或NIC引发中断，你应该将其绑定到节点0上的核心，以便没有其他节点处理它。如果没有此绑定，不同NUMA节点上的CPU可能会处理该中断。这将强制缓存一致性流量和跨节点通信。
You can assign each device's hardware interrupts to cores on the same NUMA node. This will prevent cross-node interrupt handling that would otherwise incur extra latency and evict useful cache lines on a remote node. For example, if your GPU or NIC on NUMA node 0 raises an interrupt, you'd bind it to a core on node 0 so that no other node handles it. Without this binding, a CPU on a different NUMA node might process the interrupt. This would force cache coherency traffic and cross-node communication.
在实践中，性能敏感的系统通常会禁用默认的irqbalance守护进程或使用定制规则运行它。另一个选择是使用/proc/irq/*/smp_affinity手动设置每个中断的亲和性掩码。通过将每个GPU和NIC中断绑定到最近的核心，你可以保证这些设备中断总是在最佳NUMA节点上处理。
In practice, performance-sensitive systems often disable the default irqbalance daemon or run it with bespoke rules. The other option is to manually set each interrupt's affinity mask using /proc/irq/*/smp_affinity. By pinning every GPU and NIC interrupt to the nearest cores, you guarantee that those device interrupts are always serviced on the optimal NUMA node.
简而言之，专用核心、适当的调度优先级和NUMA感知的硬件中断绑定的组合可以帮助最小化为GPU提供数据的数据加载线程的抖动。
In short, the combination of dedicated cores, appropriate scheduling priorities, and NUMA-aware hardware interrupt bindings can help minimize jitter for data loading threads that are feeding the GPUs.
3.3.5 虚拟内存和交换 (Virtual Memory and Swapping)¶
不言而喻，但你应该始终尝试避免内存交换。如果进程的任何部分被交换到磁盘，你将看到灾难性的、多个数量级的减速。GPU程序倾向于分配大量主机内存用于数据缓存。如果OS决定将一些数据从内存交换到磁盘，当GPU需要访问该数据时将经历巨大延迟。
It goes without saying, but you should always try to avoid memory swapping. If any part of your process's memory gets swapped to disk, you will see a catastrophic, multiple-orders-of-magnitude slowdown. GPU programs tend to allocate a lot of host memory for data caching. If the OS decides to swap some data out of memory and onto disk, the GPU will experience huge delays when it needs to access that data.
我们建议设置vm.swappiness=0，这告诉Linux避免交换，除非在极端内存压力下。它有效地使用cgroup限制隔离训练作业的内存，以防止容器化环境中的任何交换。
We recommend setting vm.swappiness=0, which tells Linux to avoid swapping except under extreme memory pressure. It effectively isolates your training job's memory with cgroup limits to prevent any swapping.
你应该通过Docker或Kubernetes使用cgroups v2将内存和CPU绑定到AI进程。这将在容器化环境中强制执行NUMA亲和性和无交换策略。
You should use cgroups v2 through Docker or Kubernetes to pin memory and CPUs to the AI process. This will enforce NUMA affinity and no-swap policies in containerized environments.
你也可以使用sudo swapoff -a临时禁用所有交换设备和文件，直到下次重启。只需确保你有足够的RAM用于工作负载--或设置限制以防止过度提交。否则，OOM killer可能会终止进程。使用vmstat或free -m监控交换使用情况，确保交换保持为零。
You can also use sudo swapoff -a to temporarily disable all swap devices and files until the next reboot. Just make sure you have enough RAM for your workload-or put limits to prevent overcommit. Otherwise, the OOM killer may reap the process. Monitor swap usage using vmstat or free -m to make sure swap stays at zero.
另一个相关设置是前面提到的ulimit -l，用于固定内存。如果你想防止内存被交换，你应该将该限制设高，否则可能会经历过多的内存交换。同样，通常将此限制设为无限制，用于利用大量内存的大型AI工作负载。
Another related setting is ulimit -l, as mentioned earlier for pinned memory. If you want to prevent memory from swapping, you should set that limit high or you may experience excessive memory swapping. Again, typically one sets this limit to unlimited for large AI workloads that utilize a lot of memory.
3.3.6 文件系统缓存和回写 (Filesystem Caching and Write-Back)¶
大型训练作业的最佳实践是频繁将检查点写入磁盘，以便你可以从已知良好检查点重新启动失败的作业。然而，在检查点期间，巨大的数据突发可能会填满OS页面缓存并导致停顿。
A best practice for large training jobs is to write frequent checkpoints to disk in case you need to restart a failed job from a known good checkpoint. During checkpointing, however, huge bursts of data might fill up the OS page cache and cause stalls.
对于存储，你可以调整vm.dirty_ratio和vm.dirty_background_ratio来调整页面缓存大小以缓冲写入。例如，对于多GB检查点，使用更高的脏数据比率让OS在刷新到磁盘之前在RAM中批处理更多数据。这将平滑大型检查点写入并减少训练循环中的停顿。
For storage, you can adjust vm.dirty_ratio and vm.dirty_background_ratio to tune the page-cache size for buffering writes. For example, with multi-GB checkpoints, using a higher dirty ratio lets the OS batch more data in RAM before flushing to disk. This will smooth out large checkpoint writes and reduce stalls in your training loop.
另一个选择是在单独的线程中执行检查点。PyTorch中一个较新的选项是从集群中的节点写入分布式检查点分区。在这种情况下，检查点分区将在失败作业重启后加载检查点时合并。
Another option is to perform checkpointing in a separate thread. A more recent option in PyTorch is to write distributed checkpoint partitions from nodes across the cluster. In this case, the checkpoint partitions will be combined when the checkpoint is loaded after a failed-job restart.
在延迟敏感的训练工作流中，最好完全绕过页面缓存。例如，使用O_DIRECT打开检查点文件或使用Linux的io_uring进行异步I/O以避免页面缓存停顿。在写入每个检查点后，调用posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED)立即从缓存中删除这些页面，并防止后续迭代的内存压力。
In latency-sensitive training workflows, it's best to bypass the page cache entirely. For example, open checkpoint files with O_DIRECT or use Linux's io_uring for asynchronous I/O to avoid page-cache stalls. After writing each checkpoint, call posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED) to immediately drop those pages from cache and prevent memory pressure on subsequent iterations.
3.3.7 CPU频率和C状态 (CPU Frequency and C-states)¶
默认情况下，许多计算节点将以省电模式运行CPU，这会降低CPU频率或在空闲时使其休眠。这有助于节省能源、减少热量并降低成本。在模型训练期间，当GPU正在处理其数据集的最后批次时，CPU可能不会始终100%利用。然而，当新工作到达时，这些电源管理功能可能会导致额外延迟。
By default, many compute nodes will run CPUs in a power-saving mode, which either downclocks a CPU or puts it to sleep when it's idle. This helps save energy and reduce heat and lowers the cost. During model training, the CPUs might not always be 100% utilized as the GPUs are churning through the final batches of their dataset. However, these power management features could cause extra latency when the system wakes the CPUs up again when new work arrives.
为获得最大和一致的性能，AI系统通常将CPU频率调控器配置为"performance"模式，这使CPU始终保持最大频率。这可以使用cpupower frequency-set -g performance或在BIOS中完成。
For maximum and consistent performance, AI systems often configure the CPU frequency governor to "performance" mode, which keeps the CPU at max frequency all the time. This can be done using cpupower frequency-set -g performance or in the BIOS.
同样，禁用深度C状态可以防止核心进入低功耗休眠状态。CPU C状态是由系统ACPI规范定义的省电模式。当CPU核心空闲时，它可以进入C状态以节省能源。C状态越深，节省的功率越多，但当工作到达时核心唤醒可能需要更长时间。禁用更深的C状态可以消除过多的延迟峰值。C0是活动状态；C0以上的所有状态都代表更深层次的休眠状态。
Likewise, disabling deep C-states can keep cores from going into a low-power sleep state. CPU C-states are power-saving modes defined by the system's ACPI specification. When a CPU core is idle, it can enter a C-state to save energy. The deeper the C-state, the more power is saved but the longer it may take for the core to wake up when work arrives. Disabling deeper C-states can remove excessive latency spikes. C0 is active; everything above C0 represents a deeper state of sleep.
在实践中，许多服务器BIOS/UEFI（统一可扩展固件接口）提供高性能配置文件，可自动将CPU调控器设置为"Performance"并禁用深度C状态。
In practice, many server BIOS/UEFI (Unified Extensible Firmware Interface) offer a high-performance profile that automatically sets the CPU governor to "Performance" and disables deep C-states.
本质上，我们可以用一点额外的功耗换取更响应的CPU行为。在GPU是主要功耗者的训练场景中，如果能让GPU保持供应充足，多一点CPU功耗通常是可以接受的。例如，如果数据加载线程在等待数据时休眠，CPU进入深度C6状态，CPU的大部分都会断电以最大化节能。
Essentially, we can trade a bit of extra power draw for more responsive CPU behavior. In a training scenario where GPUs are the big power consumers, a bit more CPU power usage is usually fine if it keeps the GPUs fed. For example, if a data loader thread sleeps while waiting for data and the CPU goes into the deep C6 state, significant portions of the CPU are powered down to maximize energy savings.
如果CPU进入更深的休眠状态，唤醒可能需要几微秒。虽然这不是很长时间，但许多微秒累积起来，如果管理不当可能会导致GPU气泡。气泡是GPU等待CPU恢复数据处理的时间段。通过保持CPU就绪，我们减少这种中断。许多服务器BIOS都有禁用C状态的设置--或至少限制它们。
If the CPU enters a deeper sleep state, it might take a few microseconds to wake up. While this is not a long time, many microseconds can add up and can cause GPU bubbles if not managed properly. Bubbles are periods of time when the GPU is waiting for the CPU to resume data processing. By keeping the CPU ready, we reduce such hiccups. Many BIOSes for servers have a setting to disable C-states-or at least limit them.
你应该始终关闭系统中任何可能引入不可预测延迟的东西，例如过多的上下文切换、CPU频率调频和内存到磁盘交换。结果应该是你的CPU以GPU能够消耗的速度向GPU提供数据，而不会让OS在错误的核心上调度事情或在错误的时间占用CPU周期。
You should always turn off anything in your system that might introduce unpredictable latency, such as excess context switching, CPU frequency scaling, and memory-to-disk swapping. The result should be that your CPUs deliver data to the GPUs as fast as the GPUs can consume it, without the OS scheduling things on the wrong core or taking CPU cycles away at the wrong time.
3.3.8 调优主机CPU内存分配器 (Tune Host CPU Memory Allocator)¶
在调优良好的GPU服务器上，CPU使用率可能不会很高，因为GPU处理大部分计算。然而，CPU使用率应保持稳定并与GPU活动同步。CPU必须保持忙碌准备每个传入批次，而当前批次正被GPU处理。正确的CPU到GPU交接对于维持高GPU利用率至关重要。
On a well-tuned GPU server, CPU usage may not be very high since GPUs handle most of the computation. However, CPU usage should remain steady and in lockstep with GPU activity. The CPUs must stay busy preparing each incoming batch while the current batch is being processed by the GPU. Proper CPU‐to‐GPU handoff is crucial for sustaining high GPU utilization.
通过调优主机的内存分配器（jemalloc或tcmalloc），你可以消除数据准备中不可预测的停顿。这将使GPU以峰值运行--除了有意的同步点。
By tuning your host's memory allocator (jemalloc or tcmalloc), you can eliminate unpredictable pauses in data preparation. This will keep GPUs running at their peak-except for intentional synchronization points.
调优后，你应该看到每个GPU的利用率接近100%，仅在必要的同步屏障处下降。GPU永远不应因CPU端延迟而停顿等待数据。使用jemalloc，你可以将分配分片到每CPU arena（narenas），启用background_thread用于离路径清除，并延长dirty_decay_ms/muzzy_decay_ms，以便释放的页面不会立即返回给OS。这将最小化锁争用和碎片。
After tuning, you should see each GPU's utilization hover near 100% and drop only at required synchronization barriers. The GPUs should never stall for data due to CPU-side delays. With jemalloc, you can shard allocations into per-CPU arenas (narenas), enable background_thread for off-path purging, and lengthen dirty_decay_ms/muzzy_decay_ms so that freed pages aren't immediately returned to the OS. This will minimize lock contention and fragmentation.
你可以使用MALLOC_CONF环境变量调优jemalloc：
You can tune jemalloc with the MALLOC_CONF environment variable:
同样，tcmalloc受益于调优TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES和TCMALLOC_RELEASE_RATE环境变量。这些将提供更大的每线程缓存，以便小分配避免全局锁和系统调用--保持CPU线程准备好以低、可预测的延迟为GPU提供数据。
Similarly, tcmalloc benefits from tuning the TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES and TCMALLOC_RELEASE_RATE environment variables. These will provide larger per-thread caches so that small allocations avoid global locks and syscalls-keeping CPU threads ready to feed the GPU with low, predictable latency.
3.4 GPU驱动和运行时性能设置 (GPU Driver and Runtime Settings for Performance)¶
我们已经优化了CPU端，但GPU驱动和运行时也有一些重要设置可能影响性能--特别是在多GPU和多用户场景中。NVIDIA GPU有一些旋钮，当正确调优时，可以减少开销并改善多个工作负载共享GPU的方式。
We've optimized the CPU side, but there are also important settings for the GPU driver and runtime that can affect performance-especially in multi-GPU and multiuser scenarios. NVIDIA GPUs have a few knobs that, when tuned properly, can reduce overhead and improve how multiple workloads share a GPU.
接下来，我们将介绍GPU持久化模式、MPS分区、MIG，以及时钟设置、ECC内存和内存不足行为等其他注意事项。
Next, we'll cover GPU persistence mode, the partitions of MPS, MIG, and a few other considerations like clock settings, ECC memory, and out-of-memory behavior.
3.4.1 GPU持久化模式 (GPU Persistence Mode)¶
默认情况下，如果没有应用程序使用GPU，驱动可能会将GPU置于较低功耗状态并卸载部分驱动上下文。下次应用程序想要使用GPU时，初始化它需要成本。驱动启动所有内容可能需要一两秒。
By default, if no application is using a GPU, the driver may put the GPU into a lower-power state and unload some of the driver's context. The next time an application comes along and wants to use the GPU, there's a cost to initialize it. This can take on the order of a second or two for the driver to spin everything up.
GPU初始化开销可能会对定期释放和重新获取GPU的工作负载产生负面影响。例如，考虑一个作业频繁启动和停止的训练集群。或者一个低容量推理集群，每次新推理请求到达时都必须唤醒GPU。在这两种情况下，开销都会降低整体工作负载性能。
GPU initialization overhead can negatively impact performance for workloads that periodically release and reacquire the GPU. For instance, consider a training cluster where jobs are starting and stopping frequently. Or a low-volume inference cluster that has to wake up the GPU every time a new inference request arrives. In both of these cases, the overhead will reduce overall workload performance.
持久化模式通过运行nvidia-persistenced守护进程启用。这保持GPU驱动加载和硬件处于就绪状态，即使没有应用程序活动。这请求系统在空闲时不要完全关闭GPU电源，这可以防止电源门控。
Persistence mode is enabled by running the nvidia-persistenced daemon. This keeps the GPU driver loaded and the hardware in a ready state even when no application is active. This requests that the system not fully power down the GPU when idle, which prevents power gating.
持久化保持GPU唤醒，以便下一个作业零启动延迟。这通常推荐用于长时间运行和延迟敏感的工作负载。你可以在启动时使用以下命令启用持久化守护进程：
Persistence keeps the GPU awake so that the next job has zero startup delay. This is generally recommended for long-running and latency-sensitive workloads. You can enable the persistence daemon at boot time using the following command:
在Kubernetes环境中，可以配置NVIDIA GPU Operator在所有GPU上自动启用持久化模式。
In Kubernetes environments, the NVIDIA GPU Operator can be configured to enable persistence mode on all GPUs automatically.
在AI集群上，通常在服务器启动时在所有GPU上启用持久化模式。这样，当作业开始时，GPU已经初始化并可以立即开始处理。它不会使你的实际计算更快，因为它不会加速数学运算，但它可以减少作业启动延迟并防止冷启动延迟。
On AI clusters, it's common to just enable persistence mode on all GPUs at server boot time. This way, when a job begins, the GPUs are already initialized and can start processing immediately. It won't make your actual compute any faster, as it doesn't speed up the math operations, but it shaves off job-startup latency and prevents cold start delays.
GPU持久化模式也有助于交互式使用，因为没有持久化时，空闲一段时间后你进行的第一个CUDA调用可能会在驱动重新初始化GPU时停滞。启用持久化后，该调用会快速返回。
GPU persistence mode also helps with interactive usage, as without persistence, the first CUDA call you make after some idle time might stall while the driver reinitializes the GPU. With persistence on, that call returns quickly.
持久化的唯一缺点是空闲时功耗略高，因为GPU保持在更高的就绪状态。但对于大多数数据中心GPU来说，这是为了更好的性能一致性而可以接受的权衡。
The only downside of persistence is a slightly higher power draw when idle since the GPU stays in a higher readiness state. But, for most data center GPUs, this is an acceptable trade-off for better performance consistency.
一旦由具有sudo权限的管理员设置了GPU持久化模式，你就可以享受其好处，并继续处理其他优化。
Once GPU persistence mode is set by an admin with sudo access, you can enjoy the benefits and move on to tackle other optimizations.
3.4.2 MPS¶
通常，当多个进程共享单个GPU时，GPU的调度器在它们之间进行时间分片。例如，如果两个Python进程各自有一些内核要在同一GPU上运行，GPU可能会执行一个进程的内核，然后执行另一个进程的内核，依此类推。如果这些内核很短，并且它们之间有空闲间隙，GPU可能会最终利用不足，因为它在进行"乒乓"上下文切换而没有重叠工作。
Normally, when multiple processes share a single GPU, the GPU's scheduler time-slices between them. For example, if two Python processes each have some kernels to run on the same GPU, the GPU might execute one process's kernel, then the other process's kernel, and so on. If those kernels are short and there's an idle gap between them, the GPU can end up underutilized as it's doing "ping-pong" context switches and not overlapping the work.
NVIDIA的MPS是一个功能，它创建一种伞，多个进程可以在其下同时在GPU上运行，而无需严格的时间分片。使用MPS，只要GPU资源（流式多处理器[SM]、Tensor Core等）可用，GPU就可以同时执行来自不同进程的内核。MPS本质上将进程的上下文合并到一个调度器上下文中。
NVIDIA's MPS is a feature that creates a sort of umbrella under which multiple processes can run on the GPU concurrently and without strict time-slicing. With MPS, the GPU can execute kernels from different processes at the same time as long as the GPU resources (streaming multiprocessors [SMs], Tensor Cores, etc.) are available. MPS essentially merges the contexts of the processes into one scheduler context.
MPS何时有用？对于模型训练，如果你通常每个GPU运行一个进程，你可能不会使用MPS。但如果你有在一个大GPU上运行许多推理作业的场景，MPS是一个游戏规则改变者。想象你有一个强大的GPU或GPU集群，但你的推理作业没有完全使用它。
When is MPS useful? For model training, if you normally run one process per GPU, you might not use MPS. But if you have scenarios like running many inference jobs on one big GPU, MPS is a game changer. Imagine you have a powerful GPU or GPU cluster, but your inference job-or set of multiple inference jobs-doesn't fully use it.
例如，考虑在一个40 GB GPU上运行四个独立的推理作业，每个使用5-10 GB和仅30%的GPU计算。默认情况下，每个推理作业获得一个时间片，因此在任何时刻，只有一个作业的工作实际在GPU上运行。这使得GPU平均70%空闲。
For instance, consider running four separate inference jobs on one 40 GB GPU, each using 5-10 GB and only 30% of GPU compute. By default, each inference job gets a time-slice, so at any moment, only one job's work is actually running on the GPU. That leaves the GPU 70% idle on average.
如果你为这些推理作业启用MPS，GPU可以交错它们的工作，以便当一个作业在等待内存时，另一个作业的内核可能填充GPU等。结果是更高的整体GPU利用率。在实践中，如果两个进程各使用40%的GPU，使用MPS你可能会看到GPU以80%-90%的利用率同时服务两者。
If you enable MPS for these inference jobs, the GPUs can interleave their work so that while one job is waiting on memory, another job's kernel might fill the GPU, etc. The result is higher overall GPU utilization. In practice, if two processes each use 40% of a GPU, with MPS you might see the GPU at 80%-90% utilization serving both.
例如，两个训练进程各自需要一小时--在同一GPU上顺序运行--可以使用MPS一起运行，并行完成总共略多于一小时，而不是顺序两小时。当来自并发客户端的内核和内存带宽互补时，MPS的加速可以接近翻倍。为了可视化，想象进程A和进程B各自定期启动内核，没有MPS。GPU调度可能看起来像A-B-A-B，中间有间隙，而每个进程在等待，如图3-6所示。
For instance, two training processes that each would take one hour on their own-on the same GPU, running sequentially-can run together with MPS and finish in a bit over one hour total in parallel instead of two hours sequentially. The speedup from MPS can approach a near-doubling when kernels and memory bandwidth from concurrent clients complement one another. To visualize, imagine Process A and Process B each launching kernels periodically without MPS. The GPU schedule might look like A-B-A-B with gaps in between while each one waits, as shown in Figure 3-6.
Figure 3-6. GPU schedule when processes A and B launch kernels without MPS
使用MPS，调度重叠A和B，以便每当A不使用GPU的某些部分时，B的工作可以同时使用它们，反之亦然。这种重叠消除了空闲间隙，如图3-7所示。
With MPS, the schedule overlaps A and B so that whenever A isn't using some parts of the GPU, B's work can use them simultaneously, and vice versa. This overlapping eliminates idle gaps, as shown in Figure 3-7.
Figure 3-7. Reducing idle gaps for processes A and B using MPS
设置MPS涉及运行MPS控制守护进程（nvidia-cuda-mps-control），然后启动一个MPS服务器进程来代理GPU访问。在现代GPU上，MPS更加简化，因为客户端（进程）可以直接与硬件对话，计算节点本身的干扰最小。
Setting up MPS involves running an MPS control daemon (nvidia-cuda-mps-control), which then launches an MPS server process that brokers GPU access. On modern GPUs, MPS is more streamlined as clients (the processes) can talk directly to the hardware with minimal interference from the compute node itself.
通常，你在节点上启动MPS服务器--通常每个GPU一个或每个用户一个--然后使用环境变量运行GPU作业，将它们连接到MPS。该服务器下的所有作业将并发共享GPU。
Typically, you start the MPS server on a node-often one per GPU or one per user-and then run your GPU jobs with an environment variable that connects them to MPS. All jobs under that server will share the GPU concurrently.
MPS的另一个功能是能够为每个客户端设置活动线程百分比。这限制了客户端可以使用多少SM（本质上是GPU核心）。如果你想保证服务质量（QoS），例如两个作业各获得最多50%的GPU执行资源，这可能很有用。在这种情况下，你可以设置CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50将客户端限制为约50%的SM执行容量。如果未显式设置，作业将只是竞争并使用它们能获得的任何GPU资源。
Another feature of MPS is the ability to set an active thread percentage per client. This limits how many SMs (GPU cores, essentially) a client can use. This can be useful if you want to guarantee quality of service (QoS) where two jobs, for example, each get at most 50% of the GPU's execution resources. In this case, you can set CUDA_MPS_ACTIVE_THREAD_PERCENTAGE=50 to cap a client to about 50% of SM execution capacity. If not explicitly set, the jobs will just compete and use whatever GPU resources they can.
请注意，MPS不分区GPU内存，因此所有进程将共享完整的GPU内存空间。MPS主要关于计算共享和调度。问题是一个进程可能请求大量GPU RAM，导致GPU上的OOM错误，并导致终止GPU上运行的所有其他进程。这非常具有破坏性。此外，如果一个程序自己使GPU饱和100%，MPS不会神奇地使其更快，因为你无法超过100%利用率。它仅在单个作业留下一些其他作业可以填补的空闲时才有益。
Note that MPS does not partition GPU memory, so all processes will share the full GPU memory space. MPS is mainly about compute sharing and scheduling. The issue is that one process could request a massive amount of GPU RAM, cause an OOM error on the GPU, and result in terminating all of the other processes running on the GPU. This is very disruptive. Also, if one program saturates the GPU 100% on its own, MPS won't magically make it go faster, as you can't exceed 100% utilization. It's beneficial only when individual jobs leave some slack that others can fill.
MPS的另一个限制是，默认情况下，所有MPS客户端必须作为同一Unix用户运行，因为它们共享上下文。在多用户集群中，这通常意味着MPS在调度器级别设置，以便一次只有一个用户的作业共享GPU。否则，你可以配置所有用户共享的系统级MPS，但要理解作业之间没有从安全角度的隔离。
Another limitation of MPS is that, by default, all MPS clients must run as the same Unix user since they share a context. In multiuser clusters, this means MPS is usually set up at the scheduler level such that only one user's jobs share a GPU at a time. Otherwise, you can configure a system-wide MPS that's shared by all users, but understand that the jobs are not isolated from a security standpoint.
现代NVIDIA驱动支持多用户MPS，允许来自不同Unix用户的进程共享单个MPS服务器。这提高了可用性，但不提供内存隔离。当需要强隔离时，优先使用MIG。
Modern NVIDIA drivers support multiuser MPS so that processes from different Unix users can share a single MPS server. This improves usability but does not provide memory isolation. Prefer MIG when strong isolation is required.
MPS的一个具体替代方案是Kubernetes中的GPU时间分片功能。Kubernetes时间分片允许设备插件按时间在同一GPU上调度不同的pod。例如，如果你将单个GPU配置为时间分片复制因子为4，该GPU上的4个pod可以各自获得一个时间份额。
One specific alternative to MPS is a feature for time-slicing GPUs in Kubernetes. Time-slicing on Kubernetes allows the device plugin to schedule different pods on the same GPU by time. For instance, if you configure a single GPU with a time-slicing replication factor of four, four pods on that GPU can each receive a time share.
Kubernetes时间分片是一种自动化的时间共享算法，不需要MPS。然而，这不会重叠执行。相反，它只是比默认驱动更快地切换。时间分片可能对交互式工作负载有用，你更喜欢以一些空闲时间为代价获得隔离。对于高吞吐量作业，使用MPS重叠或使用MIG分割GPU通常比细粒度时间分片更好，如下所述。
Kubernetes time-slicing is sort of an automated time-sharing algorithm that doesn't require MPS. However, this doesn't overlap execution. Instead, it just switches more rapidly than the default driver would. Time-slicing may be useful for interactive workloads where you prefer isolation at the cost of some idle time. For high-throughput jobs, overlapping with MPS or splitting the GPU with a MIG is usually better than fine-grained time-slicing, as discussed next.
3.4.3 MIG¶
现代GPU可以使用MIG在硬件级别分区为多个实例。MIG是一种虚拟化形式，但在硬件中完成。这样，开销非常低--可能几个百分点--由于损失了一些灵活性。如果一个实例空闲，它不能将其资源借给另一个，因为它们是硬分区的。
Modern GPUs can be partitioned at the hardware level into multiple instances using MIG. MIG is a form of virtualization but done in hardware. This way, the overhead is very low-maybe a few percent-due to the loss of some flexibility. If one instance is idle, it can't lend its resources to another, as they are hard partitioned.
MIG允许将GPU切片为多达七个较小的逻辑GPU--每个都有自己的专用内存部分和计算单元或SM，如图3-8所示。
MIG allows a GPU to be sliced into as many as seven smaller logical GPUs-each with its own dedicated portion of memory and compute units, or SMs, as shown in Figure 3-8.
Figure 3-8. Seven MIG slices on a modern GPU
按照惯例，NVIDIA的MIG配置文件命名使用前缀
By convention, NVIDIA's MIG profile naming uses the prefix
g to denote the number of compute slices between 1 (min) and 7 (max) on modern GPUs. Each slice number represents a number of SM groups allocated to that partition. Each SM group is roughly a 1/7 slice of the total number of SMs.
如果GPU有132个SM，每个1/7切片代表132 SMs x 1/7 = ~19 SMs在一个组中。因此，1g代表~19 SMs，2g代表~38 SMs，一直到7g，代表总共~132 SMs。
If a GPU has 132 SMs, each 1/7 slice represents 132 SMs x 1/7 = ~19 SMs in a group. As such, 1g represents ~19 SMs, 2g represents ~38 SMs, all the way up to 7g, which represents the total of ~132 SMs.
相比之下，后缀
In contrast, and somewhat confusingly, the suffix
gb specifies the exact amount of HBM GPU RAM in gigabytes that is reserved for that profile. The MIG profile values are fixed for each GPU generation and type and listed in the NVIDIA documentation.
表3-1显示了Blackwell B200 GPU可用的MIG配置文件。
Table 3-1 shows the MIG profiles available for the Blackwell B200 GPU.
表3-1 Blackwell B200的MIG配置文件
| 配置文件 | 内存比例 | SM比例 | L2缓存大小 | 复制引擎 | 实例数量 |
|---|---|---|---|---|---|
| MIG 1g.23gb | ⅛ | 1/7 | ⅛ | 2 | 7 |
| MIG 1g.45gb | 2/8 | 1/7 | 2/8 | 2 | 4 |
| MIG 2g.45gb | 2/8 | 2/7 | 2/8 | 3 | 3 |
| MIG 3g.90gb | 4/8 | 3/7 | 4/8 | 6 | 2 |
| MIG 4g.90gb | 4/8 | 4/7 | 4/8 | 8 | 1 |
| MIG 7g.180gb | Full | Full | Full | 16 | 1 |
Table 3-1. MIG Profiles for Blackwell B200 | Profile name | Fraction of memory | Fraction of SMs | L2 cache size | Copy engines | Number of instances | | ------------ | ------------------ | --------------- | ------------- | ------------ | ------------------- | | MIG 1g.23gb | ⅛ | 1/7 | ⅛ | 2 | 7 | | MIG 1g.45gb | 2/8 | 1/7 | 2/8 | 2 | 4 | | MIG 2g.45gb | 2/8 | 2/7 | 2/8 | 3 | 3 | | MIG 3g.90gb | 4/8 | 3/7 | 4/8 | 6 | 2 | | MIG 4g.90gb | 4/8 | 4/7 | 4/8 | 8 | 1 | | MIG 7g.180gb | Full | Full | Full | 16 | 1 |
该表格还显示了每个配置文件的硬件单元数量、复制引擎和L2缓存比例。这些固定配置文件与GPU的硬件内存控制器对齐，使得每个内存切片映射到连续的HBM通道。
The table also shows the number of hardware units, copy engines, and L2 cache fractions for each profile. These fixed profiles align with the GPU's hardware memory controllers such that each memory slice maps to contiguous HBM channels.
这种两部分方案将计算容量（SM组数量）与内存容量（总GB数）分离。管理员可以选择MIG配置文件的组合。分配的SM和HBM总和不需要完全匹配GPU的全部容量。但是，某些组合受到硬件分区的限制。例如，他们不能发明新的切片大小。
This two-part scheme separates compute capacity (number of SM groups) from memory capacity (total GB). Administrators can choose combinations of MIG profiles. The sum of allocated SMs and HBM does not need to exactly match the full GPU capacity. However, certain combinations are constrained by hardware partitioning. They cannot invent new slice sizes, for instance.
管理员可以使用 nvidia-smi -mig 等工具或使用NVIDIA Kubernetes GPU Operator的 nvidia.com/mig.config 配置映射在每个GPU上仅启用或禁用支持的MIG配置文件（例如1g.23gb、2g.45gb、4g.90gb等）。重新配置MIG需要排空工作负载并调用MIG的动态重新配置功能来应用更改。
Administrators can enable or disable only the supported MIG profiles (e.g., 1g.23gb, 2g.45gb, 4g.90gb, etc.) on each GPU using tools like nvidia-smi -mig or using the NVIDIA Kubernetes GPU Operator's nvidia.com/mig.config config map. Reconfiguring MIG requires draining workloads and invoking MIG's dynamic reconfiguration capability to apply the changes.
注意，这些配置文件代表GPU资源的固定分区。你不能任意选择SM和内存组合；你必须使用预定义的配置文件之一。此外，GPU上的MIG实例组合不能超过GPU的总资源。
Note that these profiles represent fixed partitions of the GPU's resources. You cannot arbitrarily choose combinations of SMs and memory; you must use one of the predefined profiles. Additionally, the combination of MIG instances on a GPU cannot exceed the GPU's total resources.
例如，在B200上，你可以创建一个7g.141gb实例（使用几乎所有资源），或者你可以创建多个较小的实例，如两个3g.71gb实例（使用84个SM和142 GB内存），但创建三个3g.71gb实例是不可能的，因为这需要126个SM--超过了B200上可用的98个SM。
For example, on a B200, you could create one 7g.141gb instance (using nearly all resources), or you could create multiple smaller instances like two 3g.71gb instances (using 84 SMs and 142 GB of memory), but creating three 3g.71gb instances would not be possible as that would require 126 SMs-more than the 98 available on the B200.
一旦GPU处于MIG模式，现代GPU可以在不重启整个系统的情况下动态创建和销毁MIG分区。你可以在排空现有工作负载后即时调整MIG实例，但要在GPU上启用或禁用MIG模式本身，需要重置该GPU。
Once a GPU is in MIG mode, modern GPUs can create and destroy MIG partitions dynamically without rebooting the entire system. You can adjust MIG instances on the fly after draining existing workloads, but to enable or disable MIG mode itself on a GPU, a reset of that GPU is needed.
每个MIG实例从软件的角度看就像一个单独的GPU，因为它有自己的内存、自己的SM，甚至独立的引擎上下文。MIG的好处是强隔离和为每个作业保证资源。如果你有多个用户或多个服务，每个只需要比如10 GB的逻辑GPU内存，你可以将它们打包到一个物理GPU上，而不会相互干扰彼此的内存或计算。
Each MIG instance acts like a separate GPU from the perspective of software since it has its own memory, its own SMs, and even separate engine contexts. The benefit of MIG is strong isolation and guaranteed resources for each job. If you have multiple users or multiple services that need only, say, 10 GB of logical GPU memory each, you can pack them onto one physical GPU without them interfering with one another's memory or compute.
作业可以专门请求MIG设备，但你必须小心地以一种使用所有切片的方式进行调度。例如，如果你有一个7切片设置，而一个作业只占用1个切片，其他6个应该用其他作业填充，否则你会让大量资源闲置。可以将集群中的某些节点配置为使用MIG进行小型推理作业，而将其他节点配置为非MIG工作负载用于大型训练作业。
Jobs can request MIG devices specifically, but you have to be careful to schedule in a way that uses all slices. For instance, if you have a 7-slice setup and a job takes only 1 slice, the other 6 should be packed with other jobs, or you're leaving a lot idle. It's possible to configure certain nodes in your cluster to use MIG for small inference jobs, for example, and configure other nodes for non-MIG workloads for large training jobs.
一个重要的操作注意事项是，为了使用MIG，你通常需要在系统级别--或至少在节点级别--进行配置。GPU必须被置于MIG模式，创建切片，并重置GPU。一旦这些步骤完成，切片就会作为独立的设备出现在系统中--每个都有自己的唯一设备ID。
One important operational note is that, in order to use MIG, you typically configure it at the system level-or at least at the node level. The GPU has to be put into MIG mode, the slices created, and the GPU reset. Once these happen, the slices appear as separate devices to the system-each with their own unique device ID.
如果你因为前期规划不当而没有使用所有可用的MIG切片，你最终会因让它们碎片化和未使用而浪费资源。重要的是提前规划分区大小以匹配你的工作负载--并在工作负载变化时调整分区大小。你需要重置GPU才能使更改生效。
If you're not using all of the available MIG slices because of poor upfront planning, you'll end up wasting resources by leaving them fragmented and unused. It's important to plan partition sizes upfront to match your workloads-and adjust the partition sizes when the workload changes. You will need to reset the GPUs to pick up the change.
对于跨越多个GPU的大规模模型训练作业和推理服务器，MIG通常没有用处，因为我们需要访问完整的GPU集合。另一方面，对于可以运行较小GPU分区的多租户、小模型推理服务器，MIG及其隔离功能可能会有用。
For large-scale model training jobs and inference servers that span many GPUs, MIG is typically not useful since we want access to the full set of GPUs. On the other hand, for multitenant, small-model inference servers that can run smaller GPU partitions, MIG and its isolation features could be useful.
截至本文撰写时，当GPU处于MIG模式时，GPU到GPU的点对点通信（包括NVLink）被禁用。这适用于跨GPU的NVLink和PCIe P2P。MIG实例无法与其他GPU进行P2P通信。跨MIG实例的CUDA IPC也受到限制。这可能会降低分布式训练的吞吐量。在启用MIG之前，请确认你的训练或推理拓扑不依赖于GPU点对点路径。大规模训练作业（以及稀疏MoE专家推理系统）需要大量的GPU到GPU通信，通常不适合使用MIG。GPU MIG实例之间的通信必须通过主机或网络结构进行。
As of this writing, when a GPU is in MIG mode, GPU-to-GPU peer-to-peer communication (including NVLink) is disabled. This applies to both NVLink and PCIe P2P across GPUs. MIG instances cannot engage in P2P with other GPUs. CUDA IPC across MIG instances is also limited. This can reduce distributed training throughput. Confirm that your training or inference topology does not rely on GPU peer paths before enabling MIG. Large-scale training jobs (and sparse MoE expert inference systems) require massive GPU-to-GPU communication and are typically not good candidates for MIG. Communication between GPU MIG instances must travel through either the host or network fabric.
MIG的一个限制是，给定GPU上的所有MIG实例必须属于同一配置文件类型。例如，你不能在单个GPU上混合使用1g.24gb和3g.71gb实例。这是一个硬件限制，旨在简化资源管理并确保隔离。
One limitation of MIG is that all MIG instances on a given GPU must be of the same profile type. For example, you cannot mix 1g.24gb and 3g.71gb instances on a single GPU. This is a hardware constraint designed to simplify resource management and ensure isolation.
此外，MIG实例不能跨多个GPU。每个MIG实例完全包含在单个物理GPU内。如果你需要跨GPU扩展工作负载，应该使用其他方法，如分布式训练或推理，而不是MIG。
Additionally, MIG instances cannot span multiple GPUs. Each MIG instance is entirely contained within a single physical GPU. If you need to scale a workload across GPUs, you should use other methods, such as distributed training or inference, rather than MIG.
简而言之，只有当你需要在同一GPU上运行多个独立作业并具有强隔离时，才启用MIG。不要将MIG用于跨越GPU的大规模分布式训练或推理，因为你需要访问GPU的全部能力及其快速互连。
In short, enable MIG only when you need to run multiple independent jobs on the same GPU with strong isolation. Do not use MIG for large-scale distributed training or inferencing that spans GPUs, as you want access to the full power of the GPUs and their fast interconnects.
在我们的大型基于transformer的模型训练和推理的背景下，我们将保持MIG关闭。但了解这个功能的存在是好的。也许一个集群可能会动态切换模式，在白天运行MIG，此时有许多小型训练或推理实验正在进行，然后在晚上关闭MIG以运行使用整个GPU的大型训练作业。
In our context of large transformer-based model training and inferencing, we will leave MIG off. But it's good to know that this feature exists. Perhaps a cluster might dynamically switch modes and run MIG during the day when lots of small training or inferencing experiments are happening, then turn MIG off at night to run big training jobs that use whole GPUs.
Kubernetes设备插件将MIG设备列为资源，例如nvidia.com/mig-1g.23gb，表示具有总共23 GB的1/7 GPU切片。
The Kubernetes device plugin will list MIG devices as resources like nvidia.com/mig-1g.23gb in the case of a 1/7 GPU slice with a total 23 GB for the slice.
3.4.4 GPU时钟速度和ECC (GPU Clock Speeds and ECC)¶
NVIDIA GPU有一个称为GPU Boost的功能，它在功率和温度限制内自动调整核心时钟。大多数时候，你应该让GPU自己工作。但有些用户喜欢锁定时钟以保持一致性，使GPU始终以固定最大频率运行。这样，运行之间的性能是稳定的，不会因功率或温度变化而变化。
NVIDIA GPUs have something called GPU Boost, which automatically adjusts the core clock within power and thermal limits. Most of the time, you should let the GPU just do its thing. But some users like to lock the clocks for consistency so that the GPU always runs at a fixed maximum frequency. This way, run-to-run performance is stable and not subject to variations in power or temperature.
固定时钟在执行基准测试时极其重要，因为后续运行可能会因过热而受到限制。如果你不考虑这一点，你可能会错误地解释后续运行的糟糕结果，因为这些后续运行的GPU可能因先前运行产生的过热而受到限制。
Fixing the clock is extremely important when performing benchmarks since later runs may be throttled due to excessive heat. If you do not account for this, you may inadvertently interpret the poor results of later runs incorrectly since the GPUs of these subsequent runs may be throttled due to excessive heat caused by previous runs.
具体来说，NVIDIA的GPU Boost会在功率/温度限制内上下调整核心时钟。使用nvidia-smi -lgc锁定核心时钟，使用-ac锁定内存时钟，将时钟锁定在最大稳定频率。这将确保GPU以恒定频率运行--并防止GPU的Boost默认功能在后续运行中降频。
Specifically, NVIDIA's GPU Boost will vary the core clock up or down to stay within power/thermal limits. Locking the clock at the max stable frequency using nvidia-smi -lgc to lock the core clock and -ac to lock the memory clock. This will make sure the GPU runs at a constant frequency-and prevents the GPU's Boost default functionality from downclocking in later runs.
锁定时钟对于可重复基准测试特别有用。然而，对于生产工作负载，让GPU Boost管理时钟通常更好，因为它可以根据实时条件动态优化性能。
Locking clocks is particularly useful for reproducible benchmarking. However, for production workloads, it's often better to let GPU Boost manage the clocks, as it can dynamically optimize performance based on real-time conditions.
一些团队故意将GPU降频以减少热量--特别是如果他们运行非常长的作业，不想随着时间的推移遭受最终的热减速。数据中心GPU通常有足够的温度余量--以及适当的空气和液体冷却--所以你不需要这样做，但了解这是一个选项是好的。
Some teams purposely underclock the GPUs to reduce heat-especially if they are running very long jobs and don't want to suffer the eventual thermal slowdown over time. Data center GPUs typically have enough temperature headroom-as well as proper air and liquid cooling-so that you don't need to do this, but it's good to know that it's an option.
另一种方法是使用nvidia-smi -pl设置略低于GPU最大热设计功耗（TDP）的功率限制。TDP是GPU在持续负载下可以产生的最大热量（以瓦特为单位）。这决定了必须散发的热量以防止过热。
Another approach is to use nvidia-smi -pl to set a power limit slightly below the maximum thermal design power (TDP) of the GPU. TDP is the maximum amount of heat, measured in watts, that a GPU can generate under sustained load. This dictates the amount of heat that must be dissipated to prevent overheating.
如果你将功率限制设置为低于TDP，GPU Boost将自动调整时钟低于热限制点。这可以减少峰值热量产生，防止限制，并产生最小的性能影响。
If you set the power limit below TDP, the GPU Boost will auto-adjust clocks below the thermal throttle point. This can reduce peak heat generation, prevent throttling, and incur minimal performance impact.
GPU上的ECC内存是另一个考虑因素。ECC确保如果存在单位内存错误（例如由宇宙射线引起），内存可以即时纠正。如果存在双位错误，则会检测到错误并向调用代码抛出错误。ECC通常在NVIDIA数据中心GPU上默认启用。
ECC memory on GPUs is another consideration. ECC ensures that if there's a single-bit memory error caused by cosmic rays, for example, the memory can be corrected on the fly. And if there's a double-bit error, the error is detected and will throw an error to the calling code. ECC is usually enabled by default on NVIDIA data center GPUs.
禁用ECC可以释放少量内存，因为ECC需要额外的位用于错误检查。这可能会产生边际性能提升，但通常只有几个百分点。然而，关闭ECC也会移除关键的内存错误保护，这可能导致系统不稳定或未检测到的数据损坏。
Disabling ECC can free up a small amount of memory since ECC requires extra bits for error checking. This might yield a marginal performance gain by reducing the overhead associated with on-the-fly error checking, but typically just a few percent. However, turning off ECC also removes critical memory-error protection, which can lead to system instability or undetected data corruption.
对于NVIDIA的数据中心GPU，包括Hopper和Blackwell，ECC默认启用，旨在保持启用以确保可靠、纠错的计算和数据完整性。对于大型模型上的长时间训练或推理作业，单个内存错误可能会完全崩溃作业，或者更糟糕的是，在没有警告的情况下静默损坏你的模型。
For NVIDIA's data center GPUs, including Hopper and Blackwell, ECC comes enabled by default and is intended to remain enabled to ensure reliable, error-corrected computation and data integrity. For long training or inference jobs on huge models, a single memory error could crash the job completely or, even worse, silently corrupt your model without a warning.
建议始终为任何严肃的AI工作负载保持ECC开启。唯一可能考虑关闭它的情况是研究环境中你愿意承担风险，因为你需要那一点点额外的内存让你的模型适应有限的GPU集群。
It's recommended to always keep ECC on for any serious AI workload. The only time you'd possibly consider turning it off is in a research setting where you are fine with taking the risk because you need that extra sliver of memory for your model to fit into your limited-memory GPU cluster.
切换ECC模式需要重置GPU，并可能需要重启当前在该GPU上运行的作业。所以这不是你想频繁切换的开关。保持ECC开启以获得稳定性和可靠性。安心胜过关闭ECC带来的微不足道的加速。
Toggling ECC mode requires resetting the GPU and likely restarting jobs that are currently running on that GPU. So it's not a toggle that you want to switch frequently. Keep ECC on for stability and reliability. The peace of mind outweighs the negligible speedup of turning ECC off.
3.4.5 GPU内存过度订阅、碎片化和内存不足处理 (GPU Memory Oversubscription, Fragmentation, and Out-of-Memory Handling)¶
与CPU RAM不同，默认情况下不存在GPU"交换"内存。如果你尝试分配超过可用GPU内存，你将收到不友好的OOM错误以及更不友好的进程崩溃。有几种机制可以缓解这个问题：允许内存动态增长、拥抱CPU和GPU之间的统一内存，以及利用内存池和缓存分配器。
Unlike CPU RAM, by default there is no such thing as GPU "swap" memory. If you try to allocate more GPU memory than available, you will get an unfriendly OOM error along with an even-unfriendlier process crash. There are a couple of mechanisms to mitigate this issue: allow memory to grow dynamically, embrace unified memory across CPU and GPU, and utilize memory pools and caching allocators.
默认情况下，某些框架（例如TensorFlow）在启动时获取所有可用GPU内存以避免碎片化并提高性能。如果你不知道这一点，在共享GPU的场景中这可能非常糟糕。PyTorch默认仅在需要时分配GPU内存。
By default, some frameworks (e.g., TensorFlow) grab all of the available GPU memory at startup to avoid fragmentation and improve performance. If you don't know this, it can be very bad in scenarios where you are sharing the GPU. PyTorch, by default, allocates GPU memory only as needed.
TensorFlow有一个选项（TF_FORCE_GPU_ALLOW_GROWTH=true）使其从小开始并按需动态增长GPU内存使用--类似于PyTorch。然而，PyTorch和TensorFlow都不允许你分配超过GPU可用内存的内存。
TensorFlow has an option (TF_FORCE_GPU_ALLOW_GROWTH=true) to make it start small and dynamically grow the GPU memory usage as needed-similar to PyTorch. However, neither PyTorch nor TensorFlow lets you allocate more memory than the GPU has available.
CUDA的统一内存系统允许你分配内存而无需预定义它驻留在CPU还是GPU上。CUDA运行时根据需要移动页面。现代NVIDIA GPU如Hopper和Blackwell包括使用页面迁移引擎（PME）进行按需分页的硬件支持。
CUDA's Unified Memory system lets you allocate memory without predefining whether it resides on the CPU or GPU. The CUDA Runtime handles moving pages as needed. Modern NVIDIA GPUs like Hopper and Blackwell include hardware support for on-demand paging using the Page Migration Engine (PME).
PME在GPU可用内存不足时自动在GPU内存和主机CPU RAM之间迁移内存页面。然而，虽然PME提供了灵活性，但依赖它可能会引入性能惩罚，相比之下为工作负载提供足够的GPU内存。
PME automatically migrates memory pages between GPU memory and host CPU RAM when the GPU runs low on available memory. However, while PME provides flexibility, relying on it can introduce performance penalties compared to having enough GPU memory for your workload.
这种GPU到CPU的内存卸载可能很慢，因为CPU内存I/O比GPU高带宽内存（HBM）I/O慢，正如我们在第2章学到的。这个机制主要是为尝试运行不适合GPU RAM的模型的从业者提供的便利。
This GPU-to-CPU memory offloading can be slow, however, since CPU memory I/O is slower than GPU high-bandwidth memory (HBM) I/O, as we learned in Chapter 2. This mechanism is mostly a convenience for practitioners trying to run models that don't fit into GPU RAM.
对于性能关键的工作负载，你通常希望尽可能避免依赖统一内存过度订阅。它作为安全网存在，而不是直接崩溃你的脚本，但当GPU内存过度订阅时，你的作业会运行得更慢。
For performance-critical workloads, you generally want to avoid relying on unified memory oversubscription where possible. It's there as a safety net instead of outright crashing your script, but your job will run slower when GPU memory is oversubscribed.
像PyTorch这样的库使用缓存分配器，以便当你释放GPU内存时，它不会立即将内存返回给OS。相反，它保留它以供将来分配重用。这避免了内存碎片化和重复请求OS分配同一内存块的开销。
Libraries like PyTorch use a caching allocator so that when you free GPU memory, it doesn't return the memory to the OS immediately. Instead, it keeps it to reuse for future allocations. This avoids memory fragmentation and the overhead of asking the OS to repeatedly allocate the same block of memory.
你可以使用PYTORCH_CUDA_ALLOC_CONF（以前称为PYTORCH_CUDA_ALLOC_CONF）设置最大池大小来限制缓存分配器。我们将在后面的章节中介绍PyTorch内存分配机制的优化。
You can limit the caching allocator using PYTORCH_CUDA_ALLOC_CONF (formerly PYTORCH_CUDA_ALLOC_CONF) to set a max pool size. We'll cover optimizations to PyTorch's memory-allocation mechanism in a later chapter.
如果你遇到GPU OOM错误（你肯定会在某个时候遇到），这很可能是由内存碎片化或过度内存缓存引起的。你可以尝试使用PyTorch的torch.cuda.empty_cache()清除缓存，但这几乎总是意味着你的工作负载确实需要那么多内存。
If you run into the GPU OOM error, which you surely will at some point, it's likely caused by memory fragmentation or excessive memory caching. You can try to clear the cache using PyTorch's torch.cuda.empty_cache(), but it almost always means your workload legitimately needs that much memory.
PyTorch还提供torch.cuda.memory_stats()和torch.cuda.memory_summary()等工具，通过显示已分配与保留内存来帮助诊断碎片化。NVIDIA的Nsight Systems还显示GPU内存使用模式，以帮助识别内存泄漏、与泄漏相关的长期分配、CPU-GPU互连活动和GPUDirect Storage时间线跟踪。此外，Nsight Compute分析器提供低级内核分析，包括占用率、吞吐量和NVLink使用情况。我们将在接下来的章节中介绍所有这些内容。
PyTorch also provides tools like torch.cuda.memory_stats() and torch.cuda.memory_summary() to help diagnose fragmentation by showing allocated versus reserved memory. NVIDIA's Nsight Systems also shows GPU memory usage patterns to help identify memory leaks, long-lived allocations that correlate with leaks, CPU-GPU interconnect activity, and GPUDirect Storage timeline tracing. Additionally, the Nsight Compute profiler provides low-level kernel analysis, including occupancy, throughput, and NVLink usage. We'll cover all of these in the upcoming chapters.
Docker提供--gpus标志来选择并将GPU暴露给容器，但它不支持设置GPU内存限制。如果你需要GPU内存或计算的硬隔离，使用MIG分区设备或使用带活动线程百分比的多进程服务（MPS）进行公平共享。在Kubernetes中使用MIG资源如nvidia.com/mig-2g.45gb配置限制，当你需要严格分区时。
Docker provides the --gpus flag to select and expose GPUs to a container, but it does not support setting a GPU memory limit. If you need hard isolation for GPU memory or compute, use MIG to partition the device or use Multi-Process Service (MPS) with active thread percentage for fair sharing. Configure limits in Kubernetes using MIG resources like nvidia.com/mig-2g.45gb when you require strict partitioning.
在多租户节点上，这对于隔离作业可能很有用。在单作业每GPU的情况下，设置内存限制并不常见，因为你希望让作业使用尽可能多的GPU内存。
In multitenant nodes, this could be useful to isolate jobs. In a single-job-per-GPU situation, it's not common to set a memory limit, as you want to let the job use as much of the GPU memory as it can get.
一般来说，GPU内存不足是可以在应用层面管理的问题。例如，你可以减少数据批处理大小、模型权重精度，甚至模型参数数量（如果这是一个选项）。
In general, running out of GPU memory is something you can manage at the application level. For instance, you can reduce the data batch size, model weight precision, or even the model parameter count, if that's an option.
最佳实践是在模型训练和推理期间使用nvidia-smi或NVML API监控GPU内存使用。如果你接近内存限制，考虑变通方法，如减少批处理大小、使用激活检查点进行训练，或其他降低内存使用的技术。
A best practice is to monitor GPU memory usage with nvidia-smi or NVML APIs during model training and inferencing. If you're close to the memory limit, consider workarounds like reducing batch size, using activation checkpointing for training, or other techniques to lower memory usage.
此外，你应该确保你的CPU内存没有被交换，因为这会间接损害你的GPU利用率和有效吞吐量，因为每次你的GPU尝试从CPU主机获取东西，但主机内存页已被交换到磁盘，你的性能将被更慢的磁盘I/O所限制。因此，将这些内存减少最佳实践与之前关于固定内存、增加ulimit和禁用交换性等建议结合起来很重要。
Also, you should ensure that your CPU memory isn't being swapped, as this would indirectly hurt your GPU utilization and goodput because each time your GPU tries to fetch something from the CPU host, but the host memory page has been swapped to disk, your performance will be bottlenecked by the much slower disk I/O. So it's important to combine these memory-reduction best practices with the earlier advice about pinning memory, increasing the ulimit, and disabling swappiness, etc.
简而言之，建议始终保持GPU驱动加载，而不是在作业之间卸载GPU驱动。这类似于GPU持久化模式，但在更深层面上。一些集群配置为在没有作业运行时卸载驱动以释放OS内核内存并出于安全考虑。然而，如果你这样做，下一个作业必须支付重新加载GPU驱动的成本，如果使用MIG，还要重新配置MIG切片。
In short, it's recommended to always keep the GPU driver loaded instead of unloading the GPU driver between jobs. This is similar to GPU persistence mode but at a deeper level. Some clusters are configured to unload the driver when no jobs are running in order to free OS kernel memory and for security. However, if you do that, the next job has to pay the cost of reloading the GPU driver and, if MIG is used, reconfiguring MIG slices.
建议在作业之间保持驱动和任何MIG配置持久化。唯一需要卸载GPU驱动的时候是故障排除或升级驱动。因此，集群管理员通常设置系统，使NVIDIA驱动模块在机器启动后始终存在。
It's recommended to keep the driver and any MIG configuration persistent across jobs. The only time you want to unload the GPU driver is for troubleshooting or upgrading the driver. As such, cluster admins often set up the system so that the NVIDIA driver modules are always present once the machine boots.
3.5 GPU容器运行时优化 (Container Runtime Optimizations for GPUs)¶
许多AI系统使用编排工具和容器运行时来管理软件环境。Kubernetes和Docker在AI基础设施中很流行。使用容器确保所有依赖项（包括CUDA和库版本）一致。这避免了"但在我机器上能工作"的问题。容器引入了一点复杂性和极小的开销，但通过正确配置，你可以使用容器为GPU工作负载获得接近裸机的性能。
Many AI systems use orchestration tools and container runtimes to manage the software environment. Kubernetes and Docker are popular in AI infrastructure. Using containers ensures that all dependencies, including CUDA and library versions, are consistent. This avoids the "but it works on my machine" problem. Containers introduce a bit of complexity and a tiny amount of overhead, but with the right configuration, you can get near bare-metal performance for GPU workloads using containers.
容器运行在节点上不是传统的虚拟机（VM）。与VM相比，容器共享主机OS内核，以便CPU和内存操作以接近本机的速度执行。使用NVIDIA Container Toolkit，从Docker容器内部访问GPU是直接的，不会产生开销。
A container running on a node is not a traditional virtual machine (VM). In contrast to VMs, containers share the host OS kernel so that CPU and memory operations perform at near-native speed. And with the NVIDIA Container Toolkit, GPU access from within a Docker container is direct and does not incur overhead.
对于使用最新NVIDIA Container Toolkit的现代GPU，在正确配置的环境中GPU性能实际上与在容器外直接在裸机主机上运行代码相同（差异< 2%）。事实上，Red Hat OpenShift和Kubernetes被用于MLPerf Inference v5.0结果，这表明现代容器和编排配置不会损害效率或延迟。
For modern GPUs running with the latest NVIDIA Container Toolkit, GPU performance within a properly configured environment is virtually identical (< 2% difference) to running the code directly on the bare-metal host outside of the container. In fact, Red Hat OpenShift and Kubernetes were used in the MLPerf Inference v5.0 results, which demonstrates that modern containers and orchestration configuration do not compromise efficiency or latency.
3.5.1 NVIDIA Container Toolkit和CUDA兼容性 (NVIDIA Container Toolkit and CUDA Compatibility)¶
使用GPU容器时的一个挑战是确保容器内的CUDA库与主机上的驱动匹配。NVIDIA通过其Container Toolkit和基础Docker镜像解决了这个问题。主机提供NVIDIA驱动，它与内核和硬件紧密集成。在容器内，你通常会找到某个版本的CUDA运行时库。
One challenge when using containers with GPUs is making sure that the CUDA libraries inside the container match the driver on the host. NVIDIA solves this through their Container Toolkit and base Docker images. The host provides the NVIDIA driver, which, remember, is tightly integrated with the kernel and hardware. Inside the container, you typically find the CUDA runtime libraries of a certain version.
一般规则是主机的NVIDIA驱动版本必须至少与容器内CUDA版本所需的最小驱动版本一样新。对于CUDA 13.x，所需的最小Linux主机驱动分支是R580或更新。对于CUDA 12.x，所需的最小Linux主机驱动分支是R525或更新。在旧驱动上使用较新的CUDA运行时将导致CUDA初始化失败。
The general rule is that the host's NVIDIA driver version must be at least as recent as the minimum driver version required by the CUDA version inside the container. For CUDA 13.x, the minimum required Linux host driver branch is R580 or newer. For CUDA 12.x, the minimum required Linux host driver branch is R525 or newer. Using a newer CUDA runtime with an older driver will cause the CUDA initialization to fail.
对于Docker和Kubernetes环境，最简单的方法是使用NVIDIA官方基础Docker镜像，来自NVIDIA GPU Cloud（NGC）或DockerHub镜像仓库。这些镜像（例如，nvcr.io/nvidia/pytorch或类似）捆绑了正确版本的CUDA运行时、cuDNN、NCCL等。此外，这些Docker镜像根据CUDA版本列出了所需的最小CUDA驱动。这样，你可以获得最新硬件的支持，而无需担心依赖问题。
For Docker and Kubernetes environments, the simplest approach is to use NVIDIA's official base Docker images from the NVIDIA GPU Cloud (NGC) or DockerHub image repositories. These images (e.g., nvcr.io/nvidia/pytorch or similar) bundle the proper versions of the CUDA runtime, cuDNN, NCCL, etc. In addition, these Docker images list the minimum required CUDA driver, depending on the CUDA version. This way, you get support for the latest hardware without dependency headaches.
每个新CUDA版本都需要最低NVIDIA驱动版本。在更新CUDA工具包时，请务必查阅NVIDIA官方兼容性矩阵并升级主机驱动。
Each new CUDA version requires a minimum NVIDIA driver version. Always consult NVIDIA's official compatibility matrix and upgrade the host driver when you update the CUDA toolkit.
3.5.2 NVIDIA Container Runtime¶
或者，NVIDIA的容器运行时实际上可以在运行时将主机驱动库注入容器中，因此你甚至不需要在镜像中包含NVIDIA驱动。相反，你只需依赖主机的驱动。同样，这是因为容器不像传统VM那样完全隔离。Docker容器被允许使用主机设备、卷和库。
Alternatively, NVIDIA's container runtime can actually inject the host driver libraries into the container at runtime, so you don't even need to ship the NVIDIA driver inside the image. Instead, you just rely on the host's driver. Again, this works because the container isn't fully isolated like a traditional VM. Docker containers are allowed to use host devices, volumes, and libraries.
在容器内部，你的应用程序使用CUDA运行时库，如来自容器镜像的libcudart.so，而NVIDIA Container Toolkit在容器启动时注入主机驱动库，如libcuda.so和libnvidia-ml.so。主机驱动库直接在主机上调用，以便一切正常工作。
Inside the container, your application uses the CUDA runtime libraries, such as libcudart.so from the container image, while the NVIDIA Container Toolkit injects the host's driver libraries such as libcuda.so and libnvidia-ml.so at container start. The host driver libraries are invoked directly on the host so that everything just works.
只要主机驱动满足镜像中CUDA Toolkit所需的最低版本，CUDA运行时库（容器）和NVIDIA Container Toolkit（主机）之间的分割就受支持。如果你不匹配并尝试在容器中使用较新的CUDA版本而主机上是旧驱动，你可能会收到错误。匹配CUDA和驱动版本很重要。
The split between CUDA runtime libraries (container) and NVIDIA Container Toolkit (host) is supported as long as the host driver meets the minimum version required by the CUDA Toolkit in the image. If you were to mismatch and try to use a newer CUDA version in the container with an old driver on the host, you'd likely get an error. It's important to match the CUDA and driver versions.
关键要点是，使用容器运行GPU时不涉及虚拟机监控程序或虚拟化层。容器直接共享主机内核和驱动，因此当内核在GPU上启动时，就像它是从主机启动的一样。
The key takeaway is that there is no hypervisor or virtualization layer involved when using containers for GPUs. The container is sharing the host kernel and driver directly, so when a kernel launches on the GPU, it's as if it launched from the host.
换句话说，你不会因为基于Docker的虚拟化而损失性能--除非你使用的是VMware或单根I/O虚拟化（SR-IOV）虚拟GPU之类的东西，这是一种需要一些调优的特殊场景。使用Docker加NVIDIA，基本上相当于裸机性能。
In other words, you aren't losing performance to Docker-based virtualization-unless you are using something like VMware or Single Root Input/Output Virtualization (SR-IOV) virtual GPUs, which is a special scenario that requires some tuning. With Docker plus NVIDIA, it's basically the equivalent of bare metal performance.
NVIDIA Container Toolkit也适用于containerd和Podman，不仅仅是Docker。这对于使用containerd作为默认容器运行时的现代Kubernetes环境很有意义。
The NVIDIA Container Toolkit works with containerd and Podman as well, not only Docker. This is relevant for modern Kubernetes environments that use containerd as the default container runtime.
3.5.3 避免容器覆盖文件系统开销 (Avoiding Container Overlay Filesystem Overhead)¶
在Docker容器中运行与直接在主机上运行的主要区别可能在I/O方面。容器通常使用联合文件系统，透明地将多个底层文件系统（如主机文件系统和容器文件系统）覆盖到单一统一视图中。
The main difference when running in a Docker container versus running directly on the host might be in I/O. Containers often use a union filesystem that transparently overlays multiple underlying filesystems, like the host filesystem and the container filesystem, into a single, unified view.
使用覆盖文件系统（如OverlayFS）会有一些开销。这种额外延迟出现是因为文件系统必须检查多个底层--只读和可写--以确定应该返回哪个版本的文件。额外的元数据查找和合并这些层的逻辑可能比从单个简单文件系统读取增加少量开销。
There is some overhead when using an overlay filesystem, however. This extra latency arises because the filesystem must check multiple underlying layers-both read-only and writable-to determine which version of a file should be returned. The additional metadata lookups and the logic for merging these layers can add a small amount of overhead compared to reading from a single, simple filesystem.
在联合文件系统（如OverlayFS）中，来自多个源的文件和目录将显示为属于一个文件系统。这种机制对容器特别有用，其中来自基础镜像层的只读文件系统与可写容器层结合。
In a union filesystem such as OverlayFS, files and directories from multiple sources will appear as if they belong to one filesystem. This mechanism is especially useful for containers, where the read-only filesystem from the base image layer is combined with a writable container layer.
此外，写入覆盖使用的写时复制（CoW）机制会有开销。CoW意味着当你修改只读层中的文件（例如，基础镜像）时，文件必须首先被复制到可写层。然后写入发生在复制的可写文件上--而不是原始的只读文件。如前所述，读取修改后的文件需要查看只读和可写层以确定哪个是正确的返回版本。
Furthermore, there is overhead when writing to the copy-on-write (CoW) mechanism used by the overlay. CoW means that when you modify a file in the read-only layer (e.g., the base image), the file must first be copied to the writable layer. The write then happens to the copied writable file-instead of the original, read-only file. As mentioned earlier, reading a modified file requires looking at both the read-only and writable layers to determine which is the correct version to return.
模型训练通常在读取数据集、加载模型和写入模型检查点时涉及繁重的I/O操作。要解决此问题，你可以使用绑定挂载将主机目录或网络文件系统挂载到容器中。绑定挂载绕过覆盖，因此执行类似于主机上直接的磁盘I/O。如果主机文件系统是NVMe SSD或NFS挂载之类的东西，你可以获得该底层存储设备的全部性能。我们特意不在镜像中打包数TB的数据集。相反，我们通过挂载引入数据。
Model training often involves heavy I/O operations when reading datasets, loading a model, and writing model checkpoints. To work around this, you can mount a host directory-or network filesystem-into the container using bind mounts. Bind mounts bypass the overlay and therefore perform similarly to disk I/O directly on the host. If the host filesystem is something like an NVMe SSD or an NFS mount, you get the full performance of that underlying storage device. We purposely do not package a multi-terabyte dataset inside the image. Instead, we bring the data in through the mounts.
例如，如果你的训练数据在主机的/data/dataset上，你可以使用-v /data/dataset:/mnt/dataset:ro运行容器，其中ro表示只读挂载。然后你的训练脚本从/mnt/dataset读取。这样，你直接从主机文件系统读取。事实上，最佳实践是避免对容器的可写层进行繁重的数据读/写。相反，将数据目录和输出目录从主机挂载到容器中。你要确保I/O不会成为容器CoW机制开销的瓶颈。
For example, if your training data is on /data/dataset on the host, you'd run the container with -v /data/dataset:/mnt/dataset:ro, where ro means read-only mount. Then your training script reads from /mnt/dataset. This way, you're reading directly from the host filesystem. In fact, it's a best practice to avoid heavy data reads/writes against the container's writable layer. Instead, mount your data directory and output directory from the host into the container. You want to ensure that I/O is not bottlenecked by the overhead of the container's CoW mechanism.
3.5.4 减小镜像大小以加快容器启动 (Reduce Image Size for Faster Container Startup)¶
如果镜像很大并需要通过网络拉取，容器启动时间可能会相当慢。但在典型的长时间运行训练循环中，几分钟的启动时间与数小时、数天或数月的训练时间相比可以忽略不计。仍然值得通过不包括不必要的构建工具或临时构建文件来保持镜像合理精简。这节省磁盘空间并改善容器启动时间。
Container startup times can be quite a bit slower if the image is huge and needs to be pulled over the network. But in a typical long-running training loop, a startup time of a few minutes is negligible compared to the hours, days, or months of training time. It's still worth keeping images reasonably slim by not including unnecessary build tools or temporary build files. This saves disk space and improves container startup time.
一些HPC中心更喜欢Singularity（Apptainer）而不是Docker，因为它可以在用户空间运行镜像而无需root守护进程。它还直接使用主机文件系统，除了OS已有的开销外几乎没有开销。
Some HPC centers prefer Singularity (Apptainer) over Docker, because it can run images in user space without a root daemon. It also uses the host filesystem directly and tends to have virtually zero overhead beyond what the OS already has.
无论是Docker还是Apptainer（前身为Singularity），研究和基准测试表明，一旦正确配置，这些容器解决方案与直接在主机上运行相比只有几个百分点的差异。基本上，如果有人给你GPU利用率和吞吐量的日志，仅从日志中很难判断作业是否在容器中运行。
In either case, Docker or Apptainer (formerly Singularity), studies and benchmarks have shown that once properly configured, these container solutions measure only a couple percent difference between running a container or directly on the host. Essentially, if someone gave you a log of GPU utilization and throughput, it would be difficult to tell from the log alone whether the job ran in a container or not.
3.6 用于拓扑感知容器编排和网络的Kubernetes (Kubernetes for Topology-Aware Container Orchestration and Networking)¶
Kubernetes（也称为K8s）是AI训练和推理的流行容器编排器。NVIDIA Kubernetes设备插件是一个轻量级组件，向调度器广播GPU硬件（/dev/nvidia0、/dev/nvidiactl等）。当你请求nvidia.com/gpu时，它将这些设备节点挂载到你的pod中。
Kubernetes (also known as K8s) is a popular container orchestrator for AI training and inference. The NVIDIA device plugin for Kubernetes is a lightweight component that advertises GPU hardware (/dev/nvidia0, /dev/nvidiactl, etc.) to the scheduler. It mounts those device nodes into your pods when you request nvidia.com/gpu under resources.limits.
NVIDIA Kubernetes GPU Operator自动化所有NVIDIA软件的安装和生命周期，包括驱动库、NVIDIA Kubernetes设备插件和NVIDIA Container Toolkit。它还负责使用NVIDIA的GPU Feature Discovery进行节点标记，用其NUMA节点和NVLink/NVSwitch ID标记每个GPU。然后调度器可以使用这些标签智能地将GPU分配给作业。
The NVIDIA Kubernetes GPU Operator automates the installation and lifecycle of all NVIDIA software, including driver libraries, the NVIDIA Kubernetes device plugin mentioned previously, and the NVIDIA Container Toolkit. It's also responsible for node labeling using NVIDIA's GPU Feature Discovery to label each GPU with its NUMA node and NVLink/NVSwitch ID. The scheduler can then use these labels to intelligently allocate GPUs to jobs.
3.6.1 使用Kubernetes拓扑管理器编排容器 (Orchestrating Containers with Kubernetes Topology Manager)¶
Kubernetes拓扑管理器可以提供详细的拓扑信息。例如，它可以检测GPU 0连接到NUMA节点0、NVLink域A和PCIe总线Z。然后Kubernetes调度器可以使用此信息以最佳方式将容器分配给GPU，以实现高效处理和通信。
Kubernetes Topology Manager can provide detailed topology information. For example, it can detect that GPU 0 is connected to NUMA node 0, NVLink domain A, and PCIe bus Z. The Kubernetes scheduler can then use this information to allocate containers to GPUs in an optimal way for efficient processing and communication.
拓扑感知GPU调度仍在成熟中。在许多集群中，管理员使用Kubernetes标签显式标记节点以捕获GPU和系统拓扑。这些标签确保多GPU pod落在GPU共享相同NVLink互连或位于相同NUMA域内的服务器上。
Topology-aware GPU scheduling is still maturing. In many clusters, administrators explicitly label nodes using Kubernetes labels to capture the GPU and system topology. These labels ensure that multi-GPU pods land on servers whose GPUs share the same NVLink interconnect or reside within the same NUMA domain.
对于我们来说，如果你在Kubernetes中运行多GPU作业，请确保启用拓扑感知调度。这通常涉及将--topology-manager-policy配置为best-effort、restricted，或在某些情况下为single-numa-node。此策略配置通过避免远程内存访问帮助多GPU和CPU + GPU工作负载实现更低延迟。这与OS级NUMA调优相辅相成。
For our purposes, if you're running multi-GPU jobs in Kubernetes, make sure to enable topology-aware scheduling. This typically involves configuring --topology-manager-policy to best-effort, restricted, or, in some cases, single-numa-node. This policy configuration helps multi-GPU and CPU + GPU workloads achieve lower latency by avoiding remote memory access. This complements the OS-level NUMA tuning.
此外，请务必使用上一节提到的最新NVIDIA GPU设备插件和NVIDIA Kubernetes GPU Operator，因为它们是拓扑感知的，并支持将多GPU pod打包到连接到同一NUMA节点的GPU上。这些通过最小化跨NUMA节点通信和减少多节点GPU工作负载中的延迟来帮助优化性能。
Also, be sure to use the latest NVIDIA GPU device plugin and NVIDIA Kubernetes GPU Operator, mentioned in the previous section, as these are topology aware and support packing multi-GPU pods onto GPUs that are connected to the same NUMA node. These help optimize performance by minimizing cross-NUMA-node communication and reducing latency in multinode GPU workloads.
在NVLink-5 NVL72系统上，单个机架级NVLink域提供高达130 TB/s的聚合双向GPU到GPU带宽，相当于每GPU约1.8 TB/s。在调度集合密集型训练时，优先选择在跨越较慢的网络结构之前将流量保持在快速NVLink域内的放置。
On NVLink-5 NVL72 systems, a single rack-level NVLink domain provides up to 130 TB/s of aggregate bidirectional GPU-to-GPU bandwidth, equivalent to about 1.8 TB/s per GPU. When scheduling collective-heavy training, prefer placements that keep traffic inside the fast NVLink domain before crossing the slower network fabric.
拓扑管理器策略的工作方式如下：best-effort策略尝试将资源对齐到NUMA节点，但如果无法完美对齐，仍会调度pod。restricted策略更严格--如果无法满足对齐要求，pod将被拒绝。single-numa-node策略最严格，要求所有资源必须来自单个NUMA节点。
The Topology Manager policies work as follows: The best-effort policy tries to align resources to NUMA nodes but will still schedule the pod if perfect alignment isn't possible. The restricted policy is stricter-if alignment requirements can't be met, the pod is rejected. The single-numa-node policy is the most strict, requiring all resources to come from a single NUMA node.
对于AI工作负载，single-numa-node策略通常提供最佳性能，因为它保证所有CPU、内存和GPU资源都来自同一NUMA节点，消除跨NUMA流量。然而，这也意味着调度可能更困难，特别是在资源紧张的多租户集群中。
For AI workloads, the single-numa-node policy often provides the best performance because it guarantees that all CPU, memory, and GPU resources come from the same NUMA node, eliminating cross-NUMA traffic. However, this also means scheduling can be more difficult, especially in busy, multitenant clusters.
3.6.2 使用Kubernetes和SLURM进行作业调度 (Job Scheduling with Kubernetes and SLURM)¶
在多节点部署中，作业调度器对于最大化所有节点的资源利用率至关重要。通常，简单Linux实用程序资源管理（SLURM）用于训练集群，而Kubernetes通常用于推理集群。然而，已经出现了集成SLURM与Kubernetes的混合解决方案。开源Slinky项目是一个简化跨训练和推理工作负载集群管理的示例解决方案。
In multinode deployments, job schedulers are essential for maximizing resource utilization across all nodes. Commonly, the Simple Linux Utility for Resource Management (SLURM) is used for training clusters, while Kubernetes is typically favored for inference clusters. However, hybrid solutions have emerged that integrate SLURM with Kubernetes. The open source Slinky project is an example solution to simplify cluster management across training and inference workloads.
这些系统处理GPU到作业的分配，并协调跨节点的进程启动。如果训练作业请求8个节点，每个节点8个GPU，调度器将识别符合条件的节点并使用mpirun或Docker等容器运行时等工具启动作业。这样，每个进程都知道作业中所有可用的GPU。许多集群还依赖经过良好测试的Docker仓库，如NVIDIA的NGC Docker仓库，以保证跨所有节点的一致软件环境--包括GPU驱动、CUDA工具包、PyTorch库和其他Python包。
These systems handle the allocation of GPUs to jobs and coordinate the launch of processes across nodes. If a training job requests 8 nodes with 8 GPUs per node, the scheduler will identify eligible nodes and start the job using tools like mpirun or container runtimes such as Docker. This way, each process is aware of all available GPUs in the job. Many clusters also rely on well-tested Docker repositories like NVIDIA's NGC Docker repository to guarantee a consistent software environment-including GPU drivers, CUDA toolkits, PyTorch libraries, and other Python packages-across all nodes.
在SLURM中，也存在类似的问题。SLURM有GPU"通用资源"的概念，你可以定义某些GPU附加到某些NUMA节点或NVLink/NVSwitch。然后在你的作业请求中，你可以请求连接到同一NUMA节点的GPU。
With SLURM, similar issues exist. SLURM has the concept of "generic resources" for GPUs, and you can define that certain GPUs are attached to certain NUMA nodes or NVLinks/NVSwitches. Then in your job request, you can ask for GPUs that are, say, connected to the same NUMA node.
如果设置不当，调度器可能会将所有GPU视为相同，并为你的多GPU容器请求提供非理想的分配。正确的配置可以避免不必要的跨NUMA节点和跨NVLink GPU通信开销。
If not properly set, a scheduler might treat all GPUs as identical and provide nonideal allocations for your multi-GPU container requests. Proper configuration can avoid unnecessary cross-NUMA-node and cross-NVLink GPU communication overhead.
SLURM也支持将MIG分区作为不同资源进行调度。这对于将多个作业打包到一个GPU上很有用。这类似于Kubernetes如何使用Kubernetes设备插件调度GPU切片。接下来，我们将讨论如何在Kubernetes中使用MIG切片。
SLURM supports scheduling MIG partitions as distinct resources as well. This can be useful for packing multiple jobs onto one GPU. This is analogous to how Kubernetes can schedule GPU slices using the Kubernetes device plugin. Next, we'll discuss how to use MIG slices with Kubernetes.
3.6.3 使用MIG切片GPU (Slicing a GPU with MIG)¶
当你启用NVIDIA的MIG模式时，单个物理GPU被切片为更小的、固定的和硬件隔离的分区，称为MIG实例。以下是请求两个nvidia.com/mig-2g.45gb MIG切片的Kubernetes pod配置示例：
When you enable NVIDIA's MIG mode, introduced in an earlier section, a single physical GPU is sliced into smaller, fixed, and hardware-isolated partitions called MIG instances. Next is an example Kubernetes pod configuration for two of the nvidia.com/mig-2g.45gb MIG slices:
这里，配置指定在一个GPU上至少有两个空闲2g.45gb实例的节点上运行pod；换句话说，2个切片，每个切片是SM的2/7（2g）。如果GPU总共有132个SM，每个是2/7 x 132 SMs = ~38 SMs。乘以2，pod总共分配~76 SMs。总内存分配是45 GB GPU RAM。
Here, the configuration specifies running a pod on a node with at least two free 2g.45gb instances on one GPU; in other words, 2 slices in which each slice is 2/7 of the SMs (2g). If a GPU has a total of 132 SMs, each is 2/7 x 132 SMs = ~38 SMs. Multiply this by 2), and the pod is allocating a total of ~76 SMs. The total memory allocation is 45 GB of GPU RAM.
请注意，调度器不能将这些分割到多个GPU或节点上。Kubernetes只有在单个节点可以提供两个分区时才会调度pod。这是因为pod不能跨越多个节点。如果没有单个节点有两个空闲的2g.45gb切片可用（总共76个SM和45 GB GPU RAM，如前计算），pod将保持Kubernetes Pending（未调度）状态，因此不会运行--即使其他节点集体有足够的MIG容量。
Note that the scheduler cannot split these across GPUs or nodes. Kubernetes will schedule the pod only if a single node can provide both partitions. This is because pods cannot span multiple nodes. If no single node has two free 2g.45gb slices available for a total of 76 SMs and 45 GB of GPU RAM (as calculated previously), the pod remains in a Kubernetes Pending (unscheduled) state and therefore won't run-even if other nodes collectively have enough MIG capacity.
这个约束突出了根据典型工作负载需求规划MIG大小的重要性。例如，如果许多作业请求2g.45gb切片，你可能会将每个GPU配置为托管三个2g.45gb实例--在其七个可能的切片中--以便两个这样的实例可以在一个GPU上共存于单个pod。
This constraint highlights the importance of planning your MIG sizes according to typical workload needs. For instance, if many jobs request 2g.45gb slices, you might configure each GPU to host three 2g.45gb instances-among its seven possible slices-so that two such instances can co-reside on one GPU for a single pod.
这种单节点约束可能导致pod永远无法运行--即使可以在集群的不同节点上找到组合的MIG资源。只有当请求的MIG资源在单个节点上可用时，请求才能被满足。
This single-node constraint can cause pods to never run-even if the combined MIG resources can be found across different nodes of the cluster. The request can be satisfied only if the requested MIG resources are available on a single node.
MIG的一个管理缺点是在MIG模式和正常（非MIG）模式之间切换GPU需要重置GPU--或重启计算节点。所以这不是调度器可以轻松按作业动态执行的操作。但是，你通常会提前创建MIG分区并让配置运行一段时间。
An administrative drawback with MIG is that switching a GPU between MIG mode and normal (non-MIG) mode requires resetting the GPUs-or rebooting the compute node. So it's not something the scheduler can easily do dynamically per job. However, you usually create MIG partitions in advance and leave the configuration running for some period of time.
在Kubernetes环境中，NVIDIA Kubernetes GPU Operator的MIG Manager可以自动配置并保留节点上的MIG分区。这样，MIG切片在重启和驱动重新加载后保持活动。
In a Kubernetes environment, the NVIDIA Kubernetes GPU Operator's MIG Manager can automatically configure and preserve MIG partitions on nodes. This way, the MIG slices remain active across reboots and driver reloads.
你可以将一个K8s节点标记为"mig-enabled"，另一个标记为"mig-disabled"，让调度器相应地放置作业/pod。这更多是一个操作细节，但了解MIG是真正的静态分区--而不是动态调度器的产物是好的。
You can label one K8s node with "mig-enabled" and another as "mig-disabled" and let the scheduler place jobs/pods accordingly. This is more of an operational detail, but it's good to know that MIG is a truly static partition-and not a product of a dynamic scheduler.
使用MIG时建议启用持久化模式，以便即使没有作业运行，MIG配置也在GPU上保持活动。这样，GPU不必在运行每个周期性作业之前不断重建切片。
Persistence mode is recommended when using MIG so that the MIG configuration remains active on the GPU even if no jobs are running. This way, the GPU doesn't have to keep rebuilding the slices before running each periodic job.
3.6.4 优化Kubernetes的网络通信 (Optimizing Network Communication for Kubernetes)¶
当你使用Kubernetes运行多节点GPU工作负载时，pod需要相互通信。在Kubernetes中，默认情况下，pod有自己的IP，不同节点上的pod之间可能存在覆盖网络或网络地址转换（NAT）。这可能引入复杂性和额外开销。
When you run multinode GPU workloads using containers with Kubernetes, the pods need to talk to one another. In Kubernetes, by default, pods have their own IP, and there might be an overlay network or network-address translation (NAT) between pods on different nodes. This can introduce complications and additional overhead.
通常，GPU集群最简单的解决方案是为这些性能敏感的作业使用主机网络。这意味着容器的网络不是隔离的，因为它直接使用主机的网络接口。要在Kubernetes中启用此功能，你在pod规范上设置hostNetwork: true。在Docker中，你可以使用--network=host运行。
Often, the simplest solution for GPU clusters is to use host networking for these performance-sensitive jobs. That means the container's network is not isolated, as it uses the host's network interface directly. To enable this in Kubernetes, you set hostNetwork: true on the pod specification. In Docker, you could run with --network=host.
使用主机网络允许容器完全像主机一样访问InfiniBand互连--没有任何额外的转换或防火墙层。这对MPI作业特别有用，因为它消除了为每个MPI rank配置端口映射的需要。
Using host networking allows a container to access the InfiniBand interconnect exactly as the host does-without any additional translation or firewall layers. This is particularly useful for MPI jobs because it eliminates the need to configure port mappings for every MPI rank.
然而，如果由于安全策略无法使用主机网络，你必须确保你的Kubernetes容器网络接口（CNI）和任何覆盖网络能够处理所需的流量。在这种情况下，你可能需要打开特定端口以支持NCCL握手和数据交换，使用NCCL_PORT_RANGE和NCCL_SOCKET_IFNAME等环境变量来帮助建立连接。
However, if host networking is not an option due to security policies, you must ensure that your Kubernetes container network interface (CNI) and any overlay network can handle the required traffic. In such cases, you may need to open specific ports to support the handshake of NCCL and data exchange, using environment variables like NCCL_PORT_RANGE and NCCL_SOCKET_IFNAME to help establish connections.
在覆盖网络上运行时，关键是延迟保持低且操作在内核空间运行。此外，确保没有用户空间代理限制节点之间的流量。这些因素可能会显著影响性能。
When operating over an overlay network, it's critical that latency remains low and operations run in kernel space. Also, make sure that no user-space proxies throttle traffic between nodes. These factors can significantly impact performance.
在使用Kubernetes环境并希望启用RDMA时，考虑安装Mellanox的Kubernetes RDMA设备插件。该插件在pod接口上暴露InfiniBand和GPUDirect RDMA端点，以实现低延迟、零拷贝网络。
When using a Kubernetes environment and you want to enable RDMA, consider installing the Kubernetes RDMA device plugin from Mellanox. This plugin exposes InfiniBand and GPUDirect RDMA endpoints on the pods interface to enable low-latency, zero-copy networking.
如果你有InfiniBand或RoCE网络，请记住如果你的NIC支持，在NVIDIA驱动中启用GPUDirect RDMA。这允许GPU直接与NIC交换数据--绕过CPU进行节点间通信。这对于在多节点环境中保持高性能至关重要。
If you have InfiniBand or RoCE networking, remember to enable GPUDirect RDMA in the NVIDIA driver if your NIC supports it. This allows GPUs to directly exchange data with the NIC-bypassing the CPU for internode communication. This is essential for maintaining high performance in a multinode environment.
3.6.5 减少Kubernetes编排抖动 (Reducing Kubernetes Orchestration Jitter)¶
运行像Kubernetes这样的编排器意味着每个节点上运行一些后台进程（例如，Kubernetes "kubelet"）、容器运行时守护进程和（理想情况下）监控代理。虽然这些服务消耗CPU和内存，但消耗量大约是单个核心的几个百分点。因此，它们不会从基于GPU的训练作业中窃取明显的时间，该作业使用这些核心进行数据加载和预处理。
Running an orchestrator like Kubernetes means there are some background processes running on every node (e.g., the Kubernetes "kubelet"), container runtime daemons, and (ideally) monitoring agents. While these services consume CPU and memory, the consumption is on the order of a few percent of a single core. So they won't steal noticeable time from a GPU-based training job, which uses these cores for data loading and preprocessing.
然而，如果训练作业运行在同时运行推理工作负载的节点上，你可能会在执行时间和吞吐量方面遇到一些抖动或不可预测的变化。这在任何多租户情况下都很常见。如果同一机器上的另一个容器意外使用大量CPU或I/O，它将通过竞争相同资源影响你的容器--无论是训练还是推理。
However, if the training job is running on a node that is also running an inference workload, you may experience some jitter, or unpredictable variation, in the execution timing and throughput. This is common in any multitenancy situation, though. If another container on the same machine unexpectedly uses a lot of CPU or I/O, it will affect your container-whether training or inference-by competing for the same resources.
为减少抖动，你可以使用Kubernetes的CPU Manager静态策略将CPU核心专门分配给特定容器。这确保其他容器无法使用这些核心。你还可以使用隔离核心来运行性能关键的工作负载。
To reduce jitter, you can use Kubernetes' CPU Manager with the static policy to assign CPU cores exclusively to specific containers. This ensures that other containers cannot use those cores. You can also use isolated cores for running performance-critical workloads.
同构工作负载（如全部训练或全部推理）从系统角度来看比训练和推理混合的异构工作负载更容易调试和调优。
Homogeneous workloads such as all training or all inference are much easier to debug and tune from a system's perspective than a heterogeneous mix of both training and inference.
另一个抖动来源是后台内核线程和中断，正如我们在第2章中讨论的使用中断请求（IRQ）亲和性的上下文。与Kubernetes类似，如果其他pod使用与你作业相同的网络或磁盘，这些pod可能会在托管你作业的计算节点上引起大量中断和额外的内核工作。这将导致抖动并影响你作业的性能。
Another source of jitter is background kernel threads and interrupts, as we discussed in Chapter 2 in the context of using interrupt request (IRQ) affinity. Similar to Kubernetes, if other pods are using the same network or disks as your job, the other pods might cause a lot of interrupts and extra kernel work on the compute nodes that host your job. This will cause jitter and affect your job's performance.
理想情况下，GPU节点完全专用于你的作业。然而，如果不是这样，你应该确保使用Linux cgroup控制器对I/O和CPU进行仔细分区，以便其他工作负载不会干扰。幸运的是，Kubernetes支持CPU隔离，这确保pod获得它们请求的专用CPU核心和内存--并防止其他pod被调度在与你相同的CPU核心上。这避免了上下文切换和资源争用的额外开销。
Ideally, a GPU node is fully dedicated to your job. However, if it's not, you should ensure that the node is carefully partitioned using Linux cgroup controllers for I/O and CPU so that other workloads don't interfere. Fortunately, Kubernetes supports CPU isolation, which ensures that pods get the dedicated CPU cores and memory they request-and prevents other pods from being scheduled on the same CPU core as yours. This avoids extra overhead from context switching and resource contention.
在实践中，性能敏感的Kubernetes作业应该请求给定节点的所有CPU和GPU，以便没有其他东西干扰或争用作业的资源。说起来容易做起来难，但从性能和一致性角度来看，这是理想的作业配置。
In practice, performance-sensitive Kubernetes jobs should request all of the CPUs and GPUs of a given node so that nothing else interferes or contends with the jobs' resources. Easier said than done, but this is the ideal job configuration from a performance and consistency standpoint.
3.6.6 改善资源保证 (Improving Resource Guarantees)¶
为了防止资源争用，Kubernetes允许你为pod定义资源请求和限制。例如，你可以指定你的训练作业需要16个CPU核心和64 GB RAM。然后Kubernetes将专门为你的作业保留这些资源，并避免在同一CPU上调度其他pod。
To safeguard against resource contention, Kubernetes lets you define resource requests and limits for pods. For example, you can specify that your training job requires 16 CPU cores and 64 GB of RAM. Kubernetes will then reserve those resources exclusively for your job and avoid scheduling other pods on the same CPUs.
这些限制使用Linux cgroups强制执行，因此如果你的容器超过其分配，它可能会被限制甚至被OOM killer终止。常见做法是使用资源请求--以及可选的CPU Manager功能来绑定核心--以确保性能关键作业获得对必要CPU资源的独占访问，以便其他进程无法从你保留的核心窃取CPU时间。
These limits are enforced using Linux cgroups, so if your container exceeds its allocation, it can be throttled or even terminated by the OOM killer. It's common practice to use resource requests-and optionally the CPU Manager feature to pin cores-to ensure that performance-critical jobs get exclusive access to the necessary CPU resources so that other processes cannot steal CPU time from your reserved cores.
Kubernetes还支持服务质量（QoS）类别，根据pod的资源请求和限制为其分配。Guaranteed QoS类别为同时设置了CPU和内存请求等于限制的pod提供最强保证。Burstable QoS为设置了某些资源请求的pod提供中间保证。BestEffort QoS为没有设置资源请求或限制的pod提供最低保证。
Kubernetes also supports Quality of Service (QoS) classes that are assigned to pods based on their resource requests and limits. The Guaranteed QoS class provides the strongest guarantees for pods that have both CPU and memory requests set equal to limits. The Burstable QoS class provides intermediate guarantees for pods that have some resource requests set. The BestEffort QoS class provides the weakest guarantees for pods that have no resource requests or limits set.
对于AI工作负载，通常建议使用Guaranteed QoS类别，以确保资源被严格保留和隔离。这减少了因资源争用导致的性能变化可能性。
For AI workloads, it's generally recommended to use the Guaranteed QoS class to ensure resources are strictly reserved and isolated. This reduces the likelihood of performance variability due to resource contention.
3.6.7 内存隔离和避免OOM Killer (Memory Isolation and Avoiding the OOM Killer)¶
如果未正确限制，内存干扰也可能发生。Kubernetes提供一流的内存隔离支持（使用Linux cgroups）。然而，贪婪的容器（如果不受约束）可能会在主机上分配过多内存。这将导致主机将其部分内存交换到磁盘。
Memory interference can also occur if not properly limited. Kubernetes provides first-class memory isolation support (using Linux cgroups). However, a greedy container, if unconstrained, could allocate too much memory on the host. This would cause the host to swap some of its memory to disk.
如果无界容器在主机上使用过多内存，臭名昭著的Linux "OOM killer"将开始终止进程--可能是你的Kubernetes作业--即使你的作业不是使用过多内存的那个。OOM killer在决定终止哪些pod时使用启发式方法。有时它决定终止最大的运行中pod，这可能是你的大型训练或推理作业，它在CPU RAM中保存大量数据以提供给GPU。
If an unbounded container uses too much memory on the host, the infamous Linux "OOM killer" will start killing processes-and potentially your Kubernetes job-even if your job wasn't the one using too much memory. The OOM killer uses heuristics when deciding which pods to kill. Sometimes it decides to kill the largest running pod, which is likely your large training or inference job holding lots of data in CPU RAM to feed the GPUs.
为了避免这种情况，你可以故意不为训练或推理容器设置严格的内存限制。这样，它们可以根据需要使用所有可用内存。通过适当的监控和告警，你可以确保作业不会尝试超出你预期的过度分配。如果你确实设置了内存限制，请确保它高于你实际预期使用的量。这提供了一些余量，避免在长时间运行的训练作业进行到第三天时被OOM killer终止。
To avoid this, you can purposely not set strict memory limits on training or inference containers. This way, they can use all available memory, if needed. With proper monitoring and alerting, you can ensure the job doesn't try to over-allocate beyond what you expect. If you do set a memory limit, make sure it's above what you actually expect to use. This provides a bit of headroom to avoid getting killed by the OOM killer three days into a long-running training job.
在Kubernetes中，没有设置请求/限制的Pod被视为BestEffort，最可能被驱逐。要获得Guaranteed QoS，每个容器必须为CPU和内存都设置requests == limits。仅设置高限制将导致Burstable QoS，而不是Guaranteed。
In Kubernetes, a Pod with no requests/limits is treated as BestEffort and is the most likely to be evicted. To obtain Guaranteed QoS, every container must set requests == limits for both CPU and memory. Setting a high limit alone will result in a Burstable QoS, not Guaranteed.
3.6.8 处理I/O隔离 (Dealing with I/O Isolation)¶
遗憾的是，在撰写本文时，Kubernetes不提供原生的、一流的I/O隔离功能。虽然Linux确实支持使用cgroup控制器进行I/O控制，但Kubernetes本身不会像对CPU和内存那样自动强制执行I/O限制。
As of this writing, Kubernetes does not offer native, first-class I/O isolation out of the box, unfortunately. While Linux does support I/O controls using cgroup controllers, Kubernetes itself does not automatically enforce I/O limits in the same way it does for CPU and memory.
如果你需要确保GPU节点上的繁重I/O工作负载不会相互干扰，你可能需要在节点级别手动配置I/O控制。这可能涉及调整cgroup v2 I/O控制器或使用其他OS级配置来分区I/O资源。
If you need to ensure that heavy I/O workloads on a GPU node don't interfere with one another, you might need to manually configure I/O controls at the node level. This can involve adjusting the cgroup v2 I/O controller or using other OS-level configurations to partition I/O resources.
简而言之，虽然Kubernetes通过调度和资源请求防止CPU争用，但I/O隔离通常需要对底层Linux系统进行额外的手动调优。
In short, while Kubernetes prevents CPU contention through scheduling and resource requests, I/O isolation usually requires additional, manual tuning of the underlying Linux system.
需要注意的是，在容器内部，某些系统设置是从主机继承的。例如，如果主机将CPU频率调频设置为性能模式，容器将继承该设置。但如果容器运行在虚拟化环境（如云实例）中，你可能无法更改这些设置。
It's important to note that, inside a container, some system settings are inherited from the host. For instance, if the host has CPU frequency scaling set to performance mode, the container will inherit that setting. But if the container is running in a virtualized environment such as a cloud instance, you might not be able to change these settings.
最好始终确保主机已调优，因为容器无法更改内核参数，如大页内存设置或CPU调控器限制。通常，集群管理员通过基础OS镜像设置这些参数和设置。或者在Kubernetes环境中，他们可能使用NVIDIA GPU Operator之类的东西在每个节点上设置持久化模式和其他sysctl旋钮。
It's a good idea to always ensure that the host machine is tuned since containers can't change kernel parameters like hugepage settings or CPU governor limits. Usually, cluster admins set these parameters and settings through the base OS image. Or, in a Kubernetes environment, they might use something like the NVIDIA GPU Operator to set persistence mode and other sysctl knobs on each node.
3.7 关键要点 (Key Takeaways)¶
以下是本章的关键要点，包括跨操作系统、驱动、GPU、CPU和容器层的优化：
Here is a list of key takeaways from this chapter, including optimizations across the operating system, driver, GPU, CPU, and container layers:
数据和计算局部性至关重要：确保数据存储和处理尽可能靠近计算单元。使用本地高速存储（如NVMe SSD缓存）最小化延迟并减少对远程文件系统或网络I/O的依赖。
Data and compute locality is critical: Ensure that data is stored and processed as close to the computation units as possible. Use local, high-speed storage such as NVMe SSD caches to minimize latency and reduce reliance on remote filesystems or network I/O.
实现NUMA感知配置和CPU亲和性：通过在同一NUMA节点内对齐进程和内存分配来优化CPU到GPU数据流。使用numactl和taskset等工具绑定CPU可防止跨节点内存访问，从而降低延迟并提高吞吐量。
Implement NUMA-aware configuration and CPU affinity: Optimize CPU-to-GPU data flow by aligning processes and memory allocations within the same NUMA node. Pin the CPU with tools like numactl and taskset prevents cross-node memory access. This will lead to lower latency and improved throughput.
最大化GPU驱动和运行时效率：微调GPU驱动设置，例如启用持久化模式以保持GPU处于就绪状态。考虑多进程服务（MPS）等功能，用于在单个GPU上重叠多个进程的工作。对于多租户环境，探索MIG分区以有效隔离工作负载。
Maximize GPU driver and runtime efficiency: Fine-tune the GPU driver settings, such as enabling persistence mode to keep GPUs in a ready state. Consider features like Multi-Process Service (MPS) for overlapping work from multiple processes on a single GPU. For multitenant environments, explore MIG partitions to isolate workloads effectively.
有效预取和批处理数据：通过提前预取数据和将小I/O操作批处理为更大、更高效的读取来保持GPU供应充足。利用PyTorch的DataLoader prefetch_factor（连同num_workers）等预取机制提前加载多个批次。
Prefetch and batch data effectively: Keep the GPUs fed by prefetching data ahead of time and batching small I/O operations into larger, more efficient reads. Leverage prefetching mechanisms like PyTorch's DataLoader prefetch_factor (along with num_workers) to load multiple batches in advance.
数据加载时固定内存：将数据预取与使用PyTorch的DataLoader pin_memory=True进行内存固定相结合，使用固定的CPU内存（页锁定，不可交换到磁盘）进行更快、异步的数据传输到GPU。因此，数据加载和模型执行可以重叠，空闲时间减少，CPU和GPU资源得到持续利用。
Pin memory when data loading: Combining data prefetching with memory pinning using PyTorch's DataLoader pin_memory=True uses pinned CPU memory (page-locked, not swappable to disk) for faster, asynchronous data transfers to the GPU. As a result, data loading and model execution can overlap, idle times are reduced, and both CPU and GPU resources are continuously utilized.
优化内存传输：利用固定、页锁定内存和大页内存等技术加速主机和GPU之间的数据传输。这有助于减少复制开销，并允许异步传输与计算重叠。
Optimize memory transfers: Leverage techniques such as pinned, page-locked memory and hugepages to accelerate data transfers between the host and GPU. This helps reduce copy overhead and allows asynchronous transfers to overlap with computations.
重叠通信与计算：通过将梯度同步和数据暂存等内存操作与正在进行的GPU计算重叠，减少数据传输的等待时间。这种重叠有助于保持高GPU利用率和更好的整体系统效率。
Overlap communication with computation: Reduce the waiting time for data transfers by overlapping memory operations like gradient synchronization and data staging with ongoing GPU computations. This overlap helps maintain high GPU utilization and better overall system efficiency.
调优和扩展网络栈：在多节点环境中，使用启用RDMA的网络（例如InfiniBand/Ethernet），并调优TCP缓冲区、MTU和中断亲和性等网络设置，以在分布式训练和推理期间保持高吞吐量。
Tune and scale the networking stack: In multinode environments, use RDMA-enabled networks (e.g., InfiniBand/Ethernet), and tune network settings such as TCP buffers, MTU, and interrupt affinities to maintain high throughput during distributed training and inference.
使用容器化和编排确保一致性：使用Docker等容器运行时配合NVIDIA Container Toolkit，以及Kubernetes等编排平台配合NVIDIA GPU Operator和设备插件，以便整个软件栈--包括驱动、CUDA库和应用程序代码--在节点间保持一致。这些解决方案有助于对齐CPU-GPU亲和性，并基于硬件拓扑管理资源分配。
Use containerization and orchestration for consistency: Use container runtimes like Docker with the NVIDIA Container Toolkit and orchestration platforms like Kubernetes with the NVIDIA GPU Operator and device plugin so that the entire software stack-including drivers, CUDA libraries, and application code-is consistent across nodes. These solutions help align CPU-GPU affinities and manage resource allocation based on hardware topology.
消除容器运行时开销：虽然容器增加了可重现性和部署便捷性，但请确保正确配置CPU和GPU亲和性、主机网络和资源隔离，以最小化任何容器开销。
Eliminate container runtime overhead: While containers increase reproducibility and ease of deployment, ensure that CPU and GPU affinities, host networking, and resource isolation are correctly configured to minimize any container overhead.
使用编排和调度最佳实践：像Kubernetes这样强大的容器编排器是确保高效资源分配的重要组成部分。高级调度技术--如Kubernetes拓扑管理器--有助于确保具有快速互连的GPU聚集在一起。
Use orchestration and scheduling best practices: Robust container orchestrators like Kubernetes are essential components for ensuring efficient resource allocation. Advanced scheduling techniques-such as the Kubernetes Topology Manager-help ensure that GPUs with fast interconnects are clustered together.
通过动态适应性和扩展实现灵活性：编排层分配工作并动态管理跨节点的工作负载分段。这种灵活性对于扩展训练任务和在数据负载及请求模式变化很大的推理场景中确保高效运行都至关重要。
Strive for flexibility through dynamic adaptability and scaling: The orchestration layer distributes work and dynamically manages workload segmentation across nodes. This flexibility is crucial for both scaling up training tasks and ensuring efficient runtime in inference scenarios where data loads and request patterns vary widely.
持续增量调优：系统级优化不是一劳永逸的。定期监控性能指标；随着工作负载演变调整CPU亲和性、批处理大小和预取设置；并累积使用这些小改进来实现显著的性能提升。
Tune continuously and incrementally: System-level optimizations are not one-and-done. Regularly monitor performance metrics; adjust CPU affinities, batch sizes, and prefetch settings as workloads evolve; and use these small improvements cumulatively to achieve significant performance gains.
减少跨栈瓶颈：最终目标是确保从OS和CPU到GPU驱动和运行时的所有组件协调工作。消除某一层的瓶颈（如CPU内存分配或驱动初始化）可释放GPU的全部潜力，这直接转化为更快的训练、更低的成本和更高效的资源使用。
Reduce bottlenecks across the stack: The ultimate goal is to ensure that all components, from the OS and CPU to the GPU driver and runtime, work in harmony. Eliminating bottlenecks in one layer, such as CPU memory allocation or driver initialization, unlocks the full potential of the GPUs, which directly translates to faster training, lower costs, and more efficient resource usage.
这些策略共同作用，最小化数据传输摩擦，减少等待时间，并确保你的硬件被充分利用于高效的训练和推理。
Together, these strategies work to minimize data transfer friction, reduce wait times, and ensure that your hardware is used to its fullest potential for efficient training and inference.
3.8 结论 (Conclusion)¶
本章表明，即使是最先进的GPU也可能受到其周围环境低效的影响。调优良好的操作系统、容器运行时、集群编排器和软件栈构成了高性能AI系统的基础。通过对齐数据与计算（通过NUMA感知绑定和本地存储解决方案）、重叠通信与计算，以及微调主机系统和GPU驱动，你可以降低延迟并增加吞吐量。
This chapter has demonstrated that even the most advanced GPUs can be hindered by inefficiencies in their surrounding environment. A well-tuned operating system, container runtime, cluster orchestrator, and software stack form the backbone of high-performance AI systems. By aligning data with compute through NUMA-aware pinning and local storage solutions, overlapping communication with computation, and fine-tuning both the host system and GPU drivers, you can reduce latency and increase throughput.
将你的整个系统想象成一辆精密设计的跑车，每个组件（CPU、内存、GPU、网络、容器、编排器和编程栈）必须无缝协作才能提供最大性能。小的调整（如启用持久化模式或优化CPU调度）单独看起来可能微不足道，但当组合起来并在大型GPU集群上扩展时，它们可以在时间和成本上带来显著节省。这些优化确保GPU在训练大规模transformer模型和运行复杂推理管道时始终以接近峰值效率运行。
Think of your entire system as a precision-engineered sports car where each component (CPU, memory, GPU, network, containers, orchestrators, and programming stack) must work seamlessly together to deliver maximum performance. Small tweaks, such as enabling persistence mode or optimizing CPU scheduling, may seem minor on their own, but when combined and scaled across a large GPU cluster, they can lead to substantial savings in time and cost. These optimizations ensure that GPUs are consistently operating near their peak efficiency when training massive transformer models and running complex inference pipelines.
随着领域的发展和模型的持续增长，系统级调优的重要性只会增加。本章讨论的技术使性能工程师和系统架构师能够利用硬件的每一点潜力。这实现了更快的迭代周期和更具成本效益的AI部署。最终，深度优化的系统加速了研究，并使尖端AI应用对更广泛的受众更加可及。
As the field evolves and models continue to grow, the importance of system-level tuning will only increase. The techniques discussed in this chapter empower performance engineers and system architects to leverage every bit of hardware potential. This enables faster iteration cycles and more cost-effective AI deployments. Ultimately, a deeply optimized system accelerates research and makes cutting-edge AI applications more accessible to a broader audience.
最后，请记住，虽然硬件和软件栈可能看起来像大量相互连接的旋钮和开关，但小的调整可以转化为时间和成本的显著节省。通过持续监控性能指标并增量完善栈的每一层，你可以将潜在瓶颈转化为效率提升的机会。让数据指导你，你将释放AI系统的全部潜力。
Finally, remember that while the hardware and software stack may seem like an unmanageable amount of interconnected knobs and switches, small tweaks can translate into significant savings in time and cost. By continuously monitoring performance metrics and incrementally refining each layer of the stack, you can transform potential bottlenecks into opportunities for efficiency gains. Let the data guide you, and you will unlock the full potential of your AI system.