第20章 AI辅助的性能优化与向百万GPU集群的扩展 (AI-Assisted Performance Optimizations and Scaling Toward Multimillion GPU Clusters)

本章汇集了一系列案例研究和未来趋势，展示了人类和AI如何协同优化AI系统的性能。具体而言，AI可以协助微调底层GPU代码，创建比人工努力产生的内核运行更快的内核。在更广泛的背景下，这些示例表明，即使在矩阵乘法等核心操作中，算法创新也能产生类似于通过获取新硬件所实现的性能提升。在高层次上，考虑一个使用来自一系列强化学习rollout（例如迭代）的奖励反馈的工作流程。这可以帮助为您的环境找到最优的GPU内核代码，如图20-1所示。

This chapter brings together a range of case studies and future trends that show how humans and AI can work together to optimize AI systems performance. Specifically, AI can assist in fine-tuning low-level GPU code to create kernels that run faster than those produced by manual efforts. In a broader context, these examples demonstrate that algorithmic innovations, even in core operations, such as matrix multiplication, can produce performance gains similar to those achieved by acquiring new hardware. At a high level, consider a workflow that uses reward feedback from a series of reinforcement learning rollouts (e.g., iterations). This can help find the most optimal GPU kernel code for your environment, as shown in Figure 20-1.

这些AI辅助的方法可以帮助提高性能、减少训练时间并降低运营成本。它们还可以实现在较小系统上高效部署更大的模型，这将解锁AI领域的未来进步。换句话说，这是AI帮助创造更好的AI。我们喜欢这个！

These AI-assisted approaches can help improve performance, reduce training time, and lower operating costs. They can also enable the efficient deployment of larger models on smaller systems, which will unlock future advances in AI. In other words, this is AI helping to create better AI. We love it!

Figure 20-1. Using reinforcement learning to find the most optimal GPU kernel code for your environment

AlphaTensor AI发现的算法提升GPU性能 (AlphaTensor AI-Discovered Algorithms Boosting GPU Performance) (Google DeepMind)

并非所有AI优化都发生在代码层面。有时，优化会深入到算法和数学领域。一个突破性的例子来自DeepMind在2022年的AlphaTensor项目，其中AI被用于发现新的通用矩阵乘法（GEMM）技术。

Not all AI optimization happens at the code level. Sometimes, the optimizations go deeper into the realm of algorithms and math. A groundbreaking example comes from DeepMind's AlphaTensor project from 2022, in which AI was used to discover new general matrix multiply (GEMM) techniques.

GEMM是几乎所有模型训练和推理工作负载的核心操作。即使GEMM效率的微小提升也能对整个AI领域产生巨大影响。AlphaTensor将快速算法的搜索形式化为一个单人游戏，使用强化学习来探索许多不同的可能性。

GEMMs are core operations that underpin almost all model training and inference workloads. Even a slight improvement in GEMM efficiency can have a huge impact across the entire AI field. AlphaTensor formalized the search for fast algorithms as a single-player game using reinforcement learning to explore many different possibilities.

令人惊讶的结果是，它发现了比当时存在的任何人类推导方法都更好的矩阵乘法公式。例如，它重新发现了Strassen著名的用于2x2矩阵的次二次算法，如图20-2所示，但也针对更大的矩阵尺寸进行了改进。

The astonishing result was that it found formulas for multiplying matrices that proved better than any human-derived method in existence at the time. For instance, it rediscovered Strassen's famous subquadratic algorithm for 2 x 2 matrices, as shown in Figure 20-2, but also improved it for larger matrix sizes.

但真正的证明是在实际硬件上测试这些算法时。AlphaTensor发现了一种特定于NVIDIA Volta V100 GPU代的方法，该方法乘法大矩阵的速度比当时标准的NVIDIA V100时代cuBLAS库快10%-20%。GEMM性能提升10%-20%是巨大的。这就像为每个模型的前向和后向传递获得了额外10%-20%的免费计算。

But the real proof came when those algorithms were tested on actual hardware. AlphaTensor discovered a method specific to the NVIDIA Volta V100 GPU generation, which multiplied large matrices 10%-20% faster than the standard NVIDIA V100-era cuBLAS library could at the time. A 10%-20% speedup in GEMM performance is huge. It's like gaining an extra 10%-20% in free compute for every model's forward and backward pass.

Figure 20-2. Strassen's subquadratic algorithm for multiplying 2 x 2 matrices (source: https://oreil.ly/5jzLn)

这种收益通常来自新的硬件代--或数月的底层CUDA调优。然而，在这种情况下，AI在相对较短的时间内找到了数学上更好的方法。

Such gains typically come from a new hardware generation-or months of low-level CUDA tuning. Yet, in this case, the AI found a better way mathematically in a relatively short amount of time.

学到的教训是，在人类工程师认为新颖的基本算法和数学操作中，可能仍有未开发的效率等待发现。AI可以筛选人类无法在合理时间内尝试的成千上万甚至数百万种算法变体。对于性能工程师来说，AlphaTensor的成功表明算法创新尚未结束。在未来，AI可能会为我们提供一套用于卷积、排序或注意力等基本操作的更快算法的新工具包。

The lesson learned is that there may still be untapped efficiency left to discover in fundamental algorithmic and mathematical operations that human engineers consider novel. The AI can sift through many thousands and millions of variations of algorithms that humans could never try in a reasonable amount of time. For performance engineers, AlphaTensor's success suggests that algorithmic innovation is not over. In the future, an AI might hand us a new toolkit of faster algorithms for fundamental operations like convolutions, sorting, or attention.

这种情况下的ROI有些间接但非常有影响力。通过将AlphaTensor的矩阵乘法算法纳入GPU库，任何大规模训练作业或推理工作负载都会看到速度的即时提升。这可能影响从图形渲染到LLM性能再到科学计算的所有方面。AlphaTensor证明，在数百个GPU上进行数千次训练迭代的15%速度提升转化为巨大的时间和能源节省。这是每次运行代码时都会回报的收益。此外，这种加速是在没有额外硬件的情况下实现的--只有更智能的软件。

The ROI in this case is somewhat indirect but very impactful. By incorporating AlphaTensor's matrix-multiply algorithm into a GPU library, any large-scale training job or inference workload would see an instantaneous boost in speed. This could influence everything from graphics rendering to LLM performance to scientific computing. AlphaTensor demonstrated that a 15% speed improvement-over thousands of training iterations on hundreds of GPUs-translates to massive time and energy savings. It's a return that pays back every time you run the code. Moreover, this speedup was achieved without additional hardware-only smarter software.

对于超大规模性能工程师来说，要点是对堆栈所有层级的AI驱动优化保持开放态度。即使是最基本的、经过良好优化的操作（如GEMM）也可能留有改进空间。让AI在没有人类偏见的情况下探索优化空间--可以通过全面减少运行时间来产生高额回报。

For the ultrascale performance engineer, the takeaway is to remain open to AI-driven optimizations at all levels of the stack. Even the most fundamental, well-optimized operations like GEMMs might leave room for improvement. Letting an AI explore the optimization space-without human bias-can yield high dividends by slashing runtimes across the board.

截至撰写本文时，AlphaTensor的矩阵乘法算法仍处于实验阶段。主流GPU库（如cuBLAS）尚未纳入这些技术，有待进一步验证和泛化。

As of this writing, AlphaTensor's matrix-multiplication algorithms remain experimental. Mainstream GPU libraries like cuBLAS have not yet incorporated these techniques, pending further validation and generalization.

使用DeepSeek-R1自动优化GPU内核 (Automated GPU Kernel Optimizations with DeepSeek-R1) (NVIDIA)

优化底层GPU代码长期以来一直是被称为CUDA Ninjas的专家人类的艺术，但已经证明AI能够执行这些专家任务。NVIDIA工程师尝试使用强大的DeepSeek-R1推理模型，看看它是否能为复杂的注意力机制生成与高性能、手工调优实现相媲美的高性能CUDA内核。

Optimizing low-level GPU code has long been an art reserved for expert humans called CUDA Ninjas, but it's been shown that AI is capable of performing these expert tasks. NVIDIA engineers experimented with the powerful DeepSeek-R1 reasoning model to see if it could generate a high-performance CUDA kernel for the complex attention mechanism that rivaled high-performance, hand-tuned implementations.

作为推理模型，DeepSeek-R1使用"推理时"扩展策略，其中，它不是在生成响应之前对模型进行一次快速传递，而是在一段时间内完善其输出--给定的时间越长，结果越好。像DeepSeek-R1这样的推理模型经过微调，可以更长时间地思考并迭代其答案--就像一个人在脱口而出回答之前花时间思考答案一样。

Being a reasoning model, DeepSeek-R1 uses an "inference-time" scaling strategy in which, instead of performing one quick pass through the model before generating a response, it refines its output over a period of time-the longer it's given, the better. Reasoning models like DeepSeek-R1 are fine-tuned to think longer and iterate on their answer-much like a human who takes time to think through their answer before spitting out a response.

在这个实验中，NVIDIA在H100上部署了R1，给它15分钟时间生成优化的注意力内核代码。他们在生成器循环中插入了一个验证器程序，这样每次R1提出内核时，验证器都会检查生成的内核代码的正确性并测量代码的效率。生成→验证→反馈→迭代循环看起来像以下伪代码：

In this experiment, NVIDIA deployed R1 on an H100 and gave it 15 minutes to generate an optimized attention kernel code. They inserted a verifier program into the generator loop so that each time R1 proposed a kernel, the verifier checked the correctness of the generated kernel code and measured the code's efficiency. The generation → verification → feedback → iteration loop looks something like the following pseudocode:

for iteration in range(max_iters):
    code = R1_model.generate_code(prompt)
    valid, runtime = verifier.verify(code)
    if valid and runtime < target_time:
        break  # 接受此内核
    prompt = refine_prompt(prompt, verifier.feedback)
    ...

这个反馈循环为下一次内核代码迭代提供了改进提示的指导。循环继续，直到代码满足给定标准，如图20-3所示。

This feedback loop provides guidance for an improved prompt to use for the next kernel-code iteration. The loop continues until the code meets the given criteria, as shown in Figure 20-3.

Figure 20-3. Inference-time scaling with DeepSeek-R1 on the NVIDIA Hopper platform (source: Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling | NVIDIA Technical Blog)

使用了以下提示：

请编写一个GPU注意力内核来支持相对位置编码。
在内核中即时实现相对位置编码。
应返回完整的代码，包括必要的修改。
使用以下函数计算相对位置编码：
def relative_positional(score, b, h, q_idx, kv_idx):
    return score + (q_idx - kv_idx)
在实现内核时，请记住，由于qk_scale = sm_scale * 1.44269504，
应将常数缩放因子1.44269504应用于相对位置编码。
PyTorch参考不需要缩放相对位置编码，但在GPU内核中，使用：
    qk = qk * qk_scale + rel_pos * 1.44269504
请提供包含这些更改的完整更新内核代码，
确保相对位置编码在内核操作中高效应用。

The following prompt was used:

Please write a GPU attention kernel to support relative position encodings. Implement the relative positional encoding on the fly within the kernel. The complete code should be returned, including the necessary modifications. Use the following function to compute the relative positional encoding: def relative_positional(score, b, h, q_idx, kv_idx): return score + (q_idx - kv_idx) When implementing the kernel, keep in mind that a constant scaling factor 1.44269504 should be applied to the relative positional encoding due to qk_scale = sm_scale * 1.44269504. The PyTorch reference does not need to scale the relative positional encoding, but in the GPU kernel, use: qk = qk * qk_scale + rel_pos * 1.44269504 Please provide the complete updated kernel code that incorporates these changes, ensuring that the relative positional encoding is applied efficiently within the kernel operations.

有了这个提示，AI生成了一个功能正确的注意力CUDA内核。（注意，1.44269504 = 1/ln(2)。使用这个值，提示在形成qk时相应地缩放相对位置项。）除了正确性之外，生成的内核还比内置的PyTorch FlexAttention API实现了1.1-2.1倍的加速。图20-4显示了生成的内核与PyTorch优化的FlexAttention在各种注意力模式（包括因果掩码和长文档掩码）下的性能比较。

With this prompt, the AI produced a functionally correct CUDA kernel for attention. (Note that 1.44269504 = 1/ln(2). Using this value, the prompt scales the relative-position term accordingly when forming qk. In addition to correctness, the generated kernel also achieved a 1.1-2.1x speedup over the built-in PyTorch FlexAttention API. Figure 20-4 shows the performance comparison between the generated kernel and PyTorch's optimized FlexAttention across various attention patterns, including causal masks and long-document masks.

Figure 20-4. Automatically generated attention kernels achieved 1.1x-2.1x speedups compared to PyTorch FlexAttention (source: Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling | NVIDIA Technical Blog)

更令人印象深刻的是，AI生成的内核在Stanford的KernelBench套件（注意力任务）的100%基本测试用例（Level-1）和96%复杂用例（Level-2）上验证准确。这基本上与人类工程师的可靠性相匹配。

Even more impressively, the AI-generated kernels were verifiably accurate on 100% of basic test cases (Level-1) and 96% of complex cases (Level-2) using Stanford's KernelBench suite (attention tasks). This essentially matches the reliability of a human engineer.

在实践中，您应该将这样的验证器系统与强大的测试套件集成--就像KernelBench所做的那样--这样罕见的边缘情况不会在生成的代码中引入错误。

In practice, you should integrate such a verifier system with a robust test suite-as done with KernelBench-so that rare edge cases don't introduce errors into the generated code.

学到的教训是，给LLM适当的工具来验证、批评和完善其输出可以提高代码质量。直观地说，这个工作流程相当于人类工程师如何反复分析、调试和改进自己的代码。在生成→验证→完善的循环下，仅用15分钟，原本粗糙的代码草稿就演变成了生产质量的注意力机制。这说明了AI辅助性能调优的一个强大范式。

The lesson learned is that giving an LLM the proper tools to verify, critique, and refine its outputs can improve code quality. Intuitively, this workflow is equivalent to how a human engineer profiles, debugs, and improves their own code repeatedly. What started as a rough code draft evolved into a production-quality attention in just 15 minutes under a generate → verify → refine loop. This illustrates a powerful paradigm for AI-assisted performance tuning.

ROI是革命性的，因为即使NVIDIA的顶级CUDA工程师也可能需要数小时或数天来手工制作和测试新型注意力内核变体。通过这种AI辅助优化方法，AI可以在很短的时间内生成相当高效的底层CUDA内核。这使工程师能够专注于更高级的AI系统优化机会和AI可能难以检测和修复的边缘情况。

The ROI is game-changing, as even NVIDIA's top CUDA engineers might spend hours or days to handcraft and test a new type of attention kernel variant. With this AI-assisted optimization approach, an AI can generate a comparably efficient, low-level CUDA kernel in a fraction of the time. This frees engineers to focus on higher-level AI system optimization opportunities and edge cases that may be tricky for an AI to detect and fix.

虽然仍需要一些人工监督，但这个实验展示了一条可行的路径，可以减少GPU优化软件的开发成本，同时实现显著的运行时性能加速。对于AI系统性能工程师来说，这种AI辅助表明未来的工作流程可能涉及与AI副驾驶合作，快速协同设计跨硬件、软件和算法的优化。AI副驾驶是人类生产力的力量倍增器。将这些副驾驶视为经过预训练和微调的AI实习生，它们能够利用从现有代码库中获得的丰富CUDA技巧和诀窍知识来推理复杂问题。

While some human oversight was still needed, this experiment showed a viable path to reduce development costs for GPU-optimized software with significant runtime performance speedups. For AI systems performance engineers, this type of AI assistance hints that future workflows may involve partnering with AI copilots to rapidly codesign optimizations across hardware, software, and algorithms. The AI copilot is a force-multiplier for human productivity. Think of these copilots as pretrained and fine-tuned AI interns capable of reasoning through complex problems using their vast knowledge of CUDA tips and tricks derived from existing code bases.

强化学习方法生成优化的GPU内核 (Reinforcement Learning Approach to Generating Optimized GPU Kernels) (Predibase)

另一家初创公司Predibase通过使用强化学习采取略有不同的方法展示了自动化GPU编程。他们提出了一个更大胆的问题：是否可以使用许多PyTorch和Triton代码示例来训练LLM成为高级OpenAI Triton程序员？

Another startup, Predibase, demonstrated automated GPU programming by taking a slightly different approach using reinforcement learning. They asked an even bolder question: is it possible to train an LLM to become an advanced OpenAI Triton programmer using many examples of PyTorch and Triton code?

请记住，OpenAI Triton是一种类似Python的GPU编程语言（和编译器），简化了GPU编程。任务是看AI是否可以生成高效的Triton代码来替代PyTorch代码--并且在NVIDIA GPU上运行得比PyTorch的TorchInductor编译器（使用Triton进行GPU代码生成）快得多。

Remember that OpenAI Triton is a Python-like GPU programming language (and compiler) that simplifies GPU programming. The task was to see if the AI could generate efficient Triton code that replaces PyTorch code-and runs much faster than PyTorch's TorchInductor compiler (which uses Triton for GPU code generation) running on NVIDIA GPUs.

在他们的实验中，Predibase使用了H100 GPU集群和一个称为组相对偏好优化（GRPO）的基于RL的微调过程，在一个中等规模的320亿参数Qwen2.5-Coder-32B-Instruct LLM上进行了训练。Predibase的RL调优模型能够为所有13个任务生成正确的Triton内核。值得注意的是，他们的环境针对正确性而非运行时性能进行了优化。

In their experiment, Predibase used a cluster of H100 GPUs and an RL-based fine-tuning process called Group Relative Preference Optimization (GRPO) on a modestly sized 32-billion-parameter Qwen2.5-Coder-32B-Instruct LLM. Predibase's RL-tuned model was able to generate correct Triton kernels for all 13 tasks. Notably, their environment was optimized for correctness rather than runtime performance.

为此，Predibase创建了一个奖励函数，使用强化学习指导模型持续生成更好的代码。具体而言，LLM首先生成候选内核。系统会自动编译和测试内核的正确性和速度。如果内核运行无错误、产生正确结果且比基线内核运行得更快，模型就会收到正奖励，如图20-5所示。

To do this, Predibase created a reward function to guide the model to continuously generate better code using reinforcement learning. Specifically, the LLM would first generate a candidate kernel. The system would automatically compile and test the kernel for correctness and speed. The model then received a positive reward if the kernel ran without errors, produced the right results, and ran faster than the baseline kernel, as shown in Figure 20-5.

通过这种基于RL的试错方法的许多迭代，模型稳步改进。在几天的训练中，AI从接近0%的成功率，在仅5000个训练步骤后，约40%的时间产生可工作的内核。一些生成的Triton内核运行速度比基线快达3倍。此外，随着训练的进行，模型继续改进。

Through many iterations of this RL-based trial-and-error approach, the model steadily improved. Within a few days of training, the AI went from near-0% success to producing working kernels ~40% of the time after only 5,000 training steps. Some of the generated Triton kernels ran up to 3x faster than baseline. Additionally, the model continued to improve as training progressed.

Figure 20-5. Assigning an RL-based reward for generating correct and high-performing OpenAI Triton code (relative to a baseline) (source: https://oreil.ly/JBxdW)

这个结果表明，AI可以通过测试、观察反馈和进行调整来优化代码。这类似于工程师如何迭代完善他们的代码。强化学习可以通过奖励正确性和速度，将AI生成的代码与真实世界的性能指标对齐。这促使AI探索优化，如使用warp级并行或最小化全局内存访问以提高整体性能。

This outcome shows that an AI can optimize code by testing, observing feedback, and making adjustments. This is similar to how engineers iteratively refine their code. Reinforcement learning can align AI-generated code with real-world performance metrics by rewarding both correctness and speed. This prompts the AI to explore optimizations like using warp-level parallelism or minimizing global memory access to improve overall performance.

Predibase演示的经验教训和ROI是，这种类型的AI辅助非常引人注目，因为它在内核代码级别自动化性能优化，潜在地减少了对人工调优的需求。工程师不再需要为新模型手工创建自定义内核，训练有素的AI助手可以生成多个变体并选择最佳的一个。这缩短了开发周期，使工程师能够专注于探索新的模型架构，例如，这样各种规模的公司都可以实现前沿、前沿模型的性能。

The lesson learned and ROI from Predibase's demonstration is that this type of AI assistance is compelling because it automates performance optimization at the kernel-code level, potentially reducing the need for manual tuning. Instead of engineers manually creating custom kernels for new models, a trained AI assistant can generate multiple variants and select the best one. This shortens development cycles and allows engineers to focus on exploring new model architectures, for example, so that companies of all sizes can achieve cutting-edge, frontier model performance.

这种方法还暗示了一个未来，即Triton和Python等更高级的语言和框架可能会取代手工CUDA编程。这种方法降低了GPU编程的门槛，从长远来看，可能导致一个自动化管道，其中AI代理持续编写和改进计算内核，成为性能工程师的必备工具。

This approach also suggests a future where higher-level languages and frameworks, such as Triton and Python, may replace manual CUDA programming. Such methods lower the barrier to GPU programming and, in the long term, could lead to an automated pipeline where an AI agent continuously writes and improves computational kernels, becoming an essential tool for performance engineers.

自我改进的AI代理 (Self-Improving AI Agents) (AI Futures Project)

到目前为止，案例研究为我们提供了真实世界超大规模AI优化的快照。展望未来，AI系统性能工程师面临着令人兴奋的挑战和机遇的混合。下一代AI模型将需要更大、更快的硬件--以及更智能、更高效的方式来使用该硬件。现在让我们转向一些关键的未来趋势--保持关注性能工程师的实用见解和最佳实践。

So far, the case studies have given us a snapshot of real-world ultrascale AI optimizations. Looking ahead, AI systems performance engineers face an exciting mix of challenges and opportunities. The next era of AI models will demand bigger and faster hardware-as well as smarter and more efficient ways to use that hardware. Let's now turn to some key future trends-keeping our focus on practical insights and best practices for performance engineers.

2025年初，AI Futures Project的一份报告描述了一系列里程碑和AI模型/代理，用于衡量技术进步、提高研究速度，并在未来几年为AI研究和开发提供变革性好处。该报告描述了前沿AI实验室目前正在设计和建造世界上有史以来最大的一些AI数据中心。这些超级集群将提供比以前系统多得多的计算，并实现模型性能的巨大飞跃。

In early 2025, a report from the AI Futures Project described a series of milestones and AI models/agents that measure technological progress, enhance research speed, and provide transformative benefits for AI research and development over the next few years. The report describes how the frontier AI labs are currently designing and building some of the biggest AI data centers the world has ever seen. These superclusters will provide exponentially more compute than previous systems and enable a massive leap in model performance.

作为背景，训练GPT-3需要大约3 x 10²³ FLOPS，GPT-4大约需要2 x 10²⁵ FLOPS。即将到来的超大规模AI工厂正在被设计用于处理大约10²⁷-10²⁸ FLOPS的训练--比GPT-4使用的计算量多约100倍，如图20-6所示。

For context, training GPT-3 required on the order of 3 x 10²³ FLOPS, and GPT-4 roughly 2 x 10²⁵ FLOPS. The upcoming ultrascale AI factories are being engineered to handle on the order of 10²⁷-10²⁸ FLOPS for training-about 100x more compute than was used for GPT-4, as shown in Figure 20-6.

研究人员正在设想一个Agent-1模型，它将使用比前一代模型多两个数量级的计算进行训练。这为持续更快的训练运行和更快的反馈循环奠定了基础。结果是一个强大的平台，解锁了前所未有的吞吐量和效率，大幅缩短了研究周期时间，加速了机器学习的突破性发现。

Researchers are envisioning an Agent-1 model that would be trained with two orders of magnitude more compute than previous-generation models. This sets the stage for consistently faster training runs and quicker feedback loops. The result is a robust platform that unlocks unprecedented throughput and efficiency and drastically cuts research cycle times and accelerates breakthrough discoveries in machine learning.

根据AI Futures Project的情景，Agent-1被设想为一个可以实时生成和优化代码的自我改进模型。通过自动化从常规调试到复杂内核融合的编码任务，这个前沿AI系统减少了洞察时间，扩展了全球研究工程师的创意视野。自动化编码充当力量倍增器，实现快速迭代，允许研究人员以较少的人工开销探索更有雄心的想法。

According to the AI Futures Project scenario, Agent-1 is envisioned as a self-improving model that can generate and optimize code in real time. By automating coding tasks ranging from routine debugging to complex kernel fusion, this frontier AI system reduces time-to-insight and expands the creative horizon for research engineers all across the world. Automated coding acts as a force multiplier that enables rapid iteration and allows researchers to explore more ambitious ideas with less manual overhead.

这些大规模AI系统预计将允许持续的模型微调和改进。后续模型Agent-2可能是一个永远学习的AI，实际上从未完成训练。因此，Agent-2不是检查点并部署静态模型，而是设计为每天根据新生成的合成数据更新其权重。

These massive AI systems are expected to allow continuous model fine-tuning and improvement. The follow-up model, Agent-2, might be an always-learning AI that never actually finishes training. So instead of checkpointing and deploying a static model, Agent-2 is designed to update its weights every day based on freshly generated synthetic data.

Figure 20-6. Amount of compute needed to train GPT-3 and GPT-4 compared to the expected compute for the "next-generation" model called Agent-1 by the researchers at the AI Futures Project (source: https://ai-2027.com)

这种永久的或持续的学习过程确保系统通过不断完善其性能和适应新信息来保持前沿。如果实现，这种方法将使我们从当前部署静态训练和微调模型的范式转变。

This perpetual, or continual, learning process makes sure that the system stays at the cutting edge by continuously refining its performance and adapting to new information. If realized, this approach would shift us from the current paradigm of deploying statically trained and fine-tuned models.

这种类型的持续重新训练（Agent-2的方法）仍然是一个活跃的研究领域，因为在保持模型稳定性和避免灾难性遗忘方面存在挑战。当模型执行先前任务的能力随着其专门从事新任务而退化时，就会发生灾难性遗忘。

This type of continuous retraining (Agent-2's approach) remains an active area of research due to challenges in preserving model stability and avoiding catastrophic forgetting. Catastrophic forgetting happens when a model's ability to perform previous tasks degrades as it specializes in the new tasks.

Agent-3被描述为一个利用算法突破来大幅提高编码效率的AI系统。通过集成高级神经暂存器和迭代蒸馏与放大技术，Agent-3转变为一个快速、成本效益高的超人编码器。在AI Futures Project提出的假设情景中，Agent-3可以并行运行200,000个副本，创建相当于数万名顶级人类程序员--并以30倍速度运行的虚拟劳动力。这种大规模并行性将加速研究周期，并使高级AI算法和系统的设计和实现民主化。

Agent-3 is described as an AI system that leverages algorithmic breakthroughs to drastically enhance coding efficiency. By integrating advanced neural scratchpads and iterated distillation and amplification techniques, Agent-3 transforms into a fast, cost-effective superhuman coder. In the hypothetical situation proposed by the AI Futures Project, Agent-3 can run 200,000 copies in parallel and create a virtual workforce equivalent to tens of thousands of top-tier human programmers-and operating 30x faster. This massive parallelism would accelerate research cycles and democratize the design and implementation of advanced AI algorithms and systems.

这个预测远超当今的实际限制；然而，这是关于未来AI生产力潜力的一个有趣的思想实验。

This projection far exceeds today's practical limits; however, it's a fun thought experiment about the potential of future AI productivity.

加速的研究将允许新想法被快速开发、测试和完善。由此产生的R&D加速将为AI性能的大幅提升铺平道路。

Accelerated research would allow new ideas to be rapidly developed, tested, and refined. The resulting acceleration in R&D would pave the way for massive gains in AI performance.

自我改进的AI很快就会达到一个点，它可以有效地在研究和开发任务中超越人类团队。这些系统持续运行，不知疲倦。它们勤奋地处理大量数据流，并以远超人类能力的速度完善算法。不间断的改进周期意味着每一天都会带来模型准确性和效率的新提升水平。这种自我改进的进度简化了R&D管道，降低了运营成本，并实现了以前无法想象的创新水平。此时，人类团队转变为监督和高级策略的角色，而AI承担繁重工作，并以重新定义技术未来的速度交付突破。

Self-improving AI will soon reach a point where it can effectively surpass human teams in research and development tasks. These systems operate continuously and without rest. They diligently process massive streams of data and refine algorithms at speeds that far exceed human capabilities. Nonstop cycles of improvement mean that every day brings a new level of enhancement to model accuracy and efficiency. This self-improving progress streamlines R&D pipelines, reduces operational costs, and enables a level of innovation that was previously unimaginable. At this point, human teams transition into roles of oversight and high-level strategy, while the AI handles the heavy lifting and delivers breakthroughs at a pace that redefines the future of technology.

Agent-4是一个假设的自我重写和超人研究者。这本质上是AGI情景，其中AI可以重写自己的代码来改进自己。Agent-4建立在其前身的基础上，但以其自我改进和以最大效率优化复杂研究任务的能力而著称。

Agent-4 is a hypothetical self-rewriting and superhuman researcher. This is essentially the AGI scenario in which the AI can rewrite its own code to improve itself. Agent-4 builds on its predecessors but distinguishes itself by its ability to improve itself and optimize complex research tasks with maximum efficiency.

在Agent-4情景中，问题解决被加速。它使用机制可解释性阐明自己的内部决策过程。这有助于理解AI底层算法和推理过程的内部工作机制。

In the Agent-4 scenario, problem solving is accelerated. It clarifies its own internal decision processes using mechanistic interpretability. This helps to understand the internal workings of the AI's underlying algorithm and reasoning process.

实际上，Agent-4的性能使其能够解决科学挑战、生成创新研究设计，并推动生成式AI模型所能实现的边界。它以远超人类能力的速度完成所有这些工作。这将是一个真正的突破，标志着AI研究和开发的一个转折点。它本质上创造了发现和进步的良性循环。

In practical terms, Agent-4's performance allows it to solve scientific challenges, generate innovative research designs, and push the boundaries of what generative AI models can achieve. It does all of this at speeds well beyond human capability. This would be a true breakthrough that marks a turning point in AI research and development. It essentially creates a virtuous cycle of discovery and progress.

AI Futures Project展示了这些代理的演变，包括AI系统基础设施、自动化编码、持续学习和自我改进模型的进步。每一代都增强了研究生产力和创新。这些代理共同强调了AI系统性能和效率对于向AGI和超级智能迈进至关重要。

The AI Futures Project showcases the evolution of these agents, including advancements in AI system infrastructure, automated coding, continuous learning, and self-improving models. Each generation enhances research productivity and innovation. Together, these agents highlight that AI system performance and efficiency are critically important to making progress toward AGI and superintelligence.

智能编译器和自动化代码优化 (Smart Compilers and Automated Code Optimizations)

我们正在进入一个极其智能的编译器和自动化的AI性能工具包时代。性能工程师手工调优每个CUDA内核或拨动每个底层旋钮的日子已经一去不复返了。越来越多的高级工具甚至AI驱动的系统正在承担繁重工作，以榨取最后的性能。

We are entering an era of extremely smart compilers and automation in the AI performance toolkit. Gone are the days when a performance engineer hand-tuned every CUDA kernel or fiddled with every low-level knob. Increasingly, high-level tools and even AI-powered systems are doing the heavy lifting to squeeze out the last bits of performance.

像PyTorch、TensorFlow和JAX这样的AI框架正在快速发展，以使用智能编译器和执行图优化器来利用最新的GPU功能。这些框架可以自动融合操作并利用Tensor Core。它们帮助使用现代GPU功能（如Tensor Memory Accelerator）重叠计算和异步数据移动。

AI frameworks like PyTorch, TensorFlow, and JAX are rapidly evolving to harness the latest GPU capabilities using smart compilers and execution-graph optimizers. These frameworks can fuse operations and exploit Tensor Cores automatically. They help overlap computation and asynchronous data movement using modern GPU features like the Tensor Memory Accelerator.

此外，OpenAI的Triton编译器允许开发者使用其基于Python的语言编写GPU内核。Triton将这些基于Python的内核编译成高效的CUDA内核，但这种复杂性对Triton用户是抽象的。

Additionally, OpenAI's Triton compiler lets developers write GPU kernels using its Python-based language. Triton compiles these Python-based kernels into efficient CUDA kernels under the hood, but this complexity is abstracted away from the Triton user.

这种工具每天都在变得越来越强大。事实上，OpenAI和NVIDIA密切合作，确保Triton完全支持最新的GPU架构--并自动利用其专门功能。

This kind of tooling is becoming more and more powerful by the day. In fact, OpenAI and NVIDIA collaborate closely to make sure Triton fully supports the newest GPU architectures-and automatically takes advantage of their specialized features.

一旦发布新的GPU代，更新的Triton编译器就会暴露GPU的新功能，而无需研究人员或工程师了解底层C++代码或PTX汇编代码。相反，他们编写高级Python代码，编译器为该特定GPU环境生成优化代码。

As soon as a new GPU generation is released, an updated Triton compiler exposes the GPU's new capabilities without the researcher or engineer needing to know the low-level C++ code or PTX assembly code. Instead, they write high-level Python code, and the compiler generates optimized code for that specific GPU environment.

已经，许多过去手工编码的优化正在被编译器自动化，而且这一趋势正在加速。自动内核融合、内核启动参数的自动调优，甚至数值精度决策都可以委托给编译器和AI助手。

Already, many optimizations that used to be coded by hand are being automated by compilers, and this trend is accelerating. Automatic kernel fusion, autotuning of kernel-launch parameters, and even numerical-precision decisions can all be delegated to compilers and AI assistants.

除了内核生成，现代框架在执行图和调度方面也变得更加智能。图执行有助于减少CPU-GPU同步开销，并为整个图的全局优化打开大门。像NVIDIA的CUDA Graphs这样的技术允许将一系列GPU操作--连同它们的依赖关系--捕获为静态图，然后可以使用cudaGraphInstantiate()和cudaGraphLaunch() API以最小的CPU开销实例化和启动，如图20-7所示。

Beyond kernel generation, modern frameworks are getting smarter about execution graphs and scheduling. Graph execution helps to reduce CPU-GPU synchronization overhead and opens the door to global optimizations across the whole graph. Technologies like NVIDIA's CUDA Graphs allow capturing a sequence of GPU operations-along with their dependencies-as a static graph that can then be instantiated and launched with minimal CPU overhead using the cudaGraphInstantiate() and cudaGraphLaunch() APIs, as shown in Figure 20-7.

Figure 20-7. Graph execution in CUDA reduces overhead when launching multiple kernels in a sequence (source: https://oreil.ly/kxSDm)

我们看到AI框架自动将训练循环和其他重复模式捕获到图中以减少开销。即使执行图是动态的而不是静态的，框架也可以跟踪一次，然后重复运行跟踪。

We're seeing AI frameworks automatically capturing training loops and other repetitive patterns into graphs to reduce overhead. Even if the execution graph is dynamic instead of static, the framework can trace it once and then run the trace repeatedly.

此外，通信与计算的重叠将越来越自动化。这过去需要人工安排，但系统可能会分析您的模型并意识到，例如，当GPU 1正在计算第10层时，GPU 2可以并行开始计算第11层--有效地在底层进行流水线并行。

Moreover, overlapping communication with computation will be increasingly automated. This used to require manual effort to arrange, but the system might analyze your model and realize, for example, that while GPU 1 is computing layer 10, GPU 2 could start computing layer 11 in parallel-effectively doing pipeline parallelism under the hood.

截至撰写本文时，完全自动的流水线并行仍然是一个活跃的研究领域。当前的AI框架仍然需要显式的流水线并行实现，尚不能在没有用户指导的情况下透明地将顺序层分布在GPU之间。

As of this writing, fully automatic pipeline parallelism remains an active area of research. Current AI frameworks still require explicit pipeline-parallel implementations and do not yet transparently distribute sequential layers across GPUs without user guidance.

我们已经看到如何实现3D、4D和5D并行（数据、张量、模型、专家和上下文/序列）来在训练和服务大型模型时最大化GPU利用率。像这样的技术是一门艺术和科学，目前涉及大量的人类直觉和经验。

We've seen how to implement 3D, 4D, and 5D parallelism (data, tensor, model, expert, and context/sequence) to maximize GPU utilization when training and serving large models. Techniques like these are an art and science that currently involve a lot of human intuition and experience.

虽然这些技术目前在Hugging Face的Ultra-Scale Playbook等专家指南中描述，但希望它们很快就会融入编译器、库和框架中。

While these techniques are currently described in expert guides like Hugging Face's Ultra-Scale Playbook, the hope is that they'll be baked into compilers, libraries, and frameworks soon.

本质上，AI框架应该理解这些模式并调度工作，使分布式系统的所有部分保持忙碌--无需用户分析、调试和优化每个GPU流、内存传输和网络调用。例如，我们可能有一天会有一个AI顾问，当您定义一个5000亿参数的模型时，它立即建议，"您应该在每个节点上使用八路张量并行，然后跨节点使用四路流水线。顺便说一句，使用这些层分组和块大小以获得最佳效率。"

In essence, the AI framework should understand these patterns and schedule work to keep all parts of a distributed system busy-without the user profiling, debugging, and optimizing every GPU stream, memory transfer, and network call. For example, we might one day have an AI advisor that, when you define a 500 billion-parameter model, immediately suggests, "You should use eight-way tensor parallelism on each node and then a four-way pipeline across nodes. And, by the way, use these layer groupings and chunk sizes for optimal efficiency."

对于性能工程师来说，这将成为一个巨大的生产力提升。您不需要尝试无尽的策略和配置，而是可以从一开始就向AI系统询问接近最优的解决方案。通过将人类洞察与编译器/AI自动化相结合，您可以以比过去更少的努力实现最优结果。这有点像再次从汇编语言转向高级语言，因为我们正在将更多责任委托给工具。对于性能工程师来说，这意味着我们的角色更多地转向指导这些工具--并快速验证它们做得很好--而不是缓慢地实验和手动验证一切。

For performance engineers, this would become a huge productivity boost. Instead of trying endless strategies and configurations, you could ask an AI system for a near-optimal solution from the start. By combining human insight with compiler/AI automation, you can achieve optimal results with less effort than in the past. It's a bit like moving from assembly- to high-level languages all over again as we're delegating more responsibility to the tools. For performance engineers, this means our role shifts more toward guiding these tools-and quickly verifying that they're doing a good job-rather than slowly experimenting and verifying everything manually.

简而言之，AI的软件栈正变得越来越智能和自主。这里的最佳实践是拥抱这些工具而不是与它们对抗。利用像OpenAI的Triton这样的高级编译器，它们了解您硬件的功能和性能选项。并关注新的AI驱动优化服务，因为它们起初可能看起来像黑盒，但它们封装了大量来之不易的性能知识。

In short, the software stack for AI is getting increasingly intelligent and autonomous. The best practice here is to embrace these tools rather than fight them. Leverage high-level compilers like OpenAI's Triton that know about your hardware's capabilities and performance options. And keep an eye on new AI-driven optimization services, as they might seem like black boxes at first, but they encapsulate a lot of hard-won performance knowledge.

AI辅助的实时系统优化和集群运营 (AI-Assisted Real-Time System Optimizations and Cluster Operations)

推动自动化的不仅仅是代码--还有系统和集群运营层面。在未来，AI系统将越来越多地管理和优化自己--特别是在大规模训练和推理集群中，在任何给定时间点都有无数的并发作业和请求在运行--需要复杂的资源共享策略。

The push for automation isn't just in code-it's at the system and cluster operations level as well. In the future, AI systems will increasingly manage and optimize themselves-especially in large-scale training and inference clusters where there are myriad concurrent jobs and requests in flight at any given point in time-requiring complex resource-sharing strategies.

一个即将到来的发展是由AI驱动的自主调度和集群管理。今天的集群编排器（例如Kubernetes、SLURM）仍然依赖静态启发式和简单的资源请求，但向更自适应调度机制的趋势正在上升。但想象一个智能代理观察整个集群的状态，并学习如何调度推理请求和训练作业以获得最大整体吞吐量。

One imminent development is autonomous scheduling and cluster management driven by AI. Today's cluster orchestrators (e.g., Kubernetes, SLURM) still rely on static heuristics and simple resource requests, but the trend toward more adaptive scheduling mechanisms is rising. But imagine a smart agent observing the entire cluster's state and learning how to schedule inference requests and training jobs for maximum overall throughput.

这个调度代理可能会了解到某些请求或作业可以在同一节点上共存而不相互干扰--也许是因为一个是计算密集型的，而另一个是内存带宽密集型的。通过摄取来自Kubernetes集群的遥测数据（pod的GPU利用率、队列等待时间等），AI调度器可以动态重新调度作业或调整pod资源以最大化整体吞吐量并最小化空闲时间。

This scheduling agent might learn that certain requests or jobs can be colocated on the same node without interfering with one another-perhaps because one is compute-heavy while another is memory-bandwidth-heavy. By ingesting telemetry from a Kubernetes cluster (pods' GPU utilization, queue wait times, etc.), an AI scheduler could dynamically reschedule jobs or adjust pod resources to maximize overall throughput and minimize idle time.

在某种意义上，集群开始表现得像自动驾驶汽车，根据实时条件不断调整其驾驶策略（资源分配）--而不是遵循固定路线。对性能工程师的好处是更高的资源利用率和更少的瓶颈。我们的工作将转向为AI调度器设定高级策略和目标，并让它弄清楚细节。

In a sense, the cluster begins to behave like a self-driving car, constantly adjusting its driving strategy (resource allocation) based on real-time conditions-rather than following a fixed route. The benefit to performance engineers is higher resource utilization and fewer bottlenecks. Our job would shift to setting the high-level policies and goals for the AI scheduler and letting it figure out the specifics.

例如，NVIDIA Dynamo的分布式推理框架协调请求调度、KV缓存放置和跨GPU和节点的数据移动。它与Kubernetes集成用于推理和分解。在这种情况下，Dynamo的调度器会将微批次分配给不同的流水线阶段，并通过重新路由请求来处理节点故障。

NVIDIA Dynamo's distributed inference framework, for instance, coordinates request scheduling, KV cache placement, and data movement across GPUs and nodes. It integrates with Kubernetes for inference and disaggregation. In this case, Dynamo's scheduler would allocate microbatches to different pipeline stages and handle node failures by rerouting requests.

通过权重流和激活卸载等技术，模型的层可以仅在需要权重时（例如，在解码期间）按需从主机内存流式传输到GPU。这可以跨许多节点和GPU发生。这允许在更便宜的存储上托管100万亿参数模型的部分。这有助于无缝扩展推理。

And with techniques like weight streaming and activation offloading, the model's layers can be streamed on demand from host memory to the GPU only when the weights are needed (e.g., during decode.) And this can happen across many nodes and GPUs. This allows hosting parts of a 100-trillion-parameter model on cheaper storage. This helps to seamlessly scale inference.

我们还可能看到面向系统操作员的AI性能副驾驶。LLM可以以支持角色成为基础设施的一部分。例如，性能工程师可能有一个AI助手，他们可以问，"我如何加速我的训练作业？"并获得有根据的建议。这听起来很奇特，但当您考虑到这样的助手可以在数千次过去运行、日志和调整的累积知识上进行训练时，这是合理的。

We could also see AI performance copilots for system operators. LLMs can become part of the infrastructure in a support role. For example, a performance engineer might have an AI assistant they can ask, "How can I speed up my training job?" and get informed suggestions. This sounds fanciful, but it's plausible when you consider such an assistant could be trained on the accumulated knowledge of thousands of past runs, logs, and tweaks.

AI性能副驾驶可能还会注意到您的GPU内存使用率很低并建议增加批次大小，或者注意到您的梯度噪声尺度很高并建议学习率计划更改。这个代理将封装人类专家的一些来之不易的经验--使这些知识随时可用。

The AI performance copilot might also recognize that your GPU memory usage is low and suggest increasing batch size, or notice that your gradient noise scale is high and suggest a learning rate schedule change. This agent would encapsulate some of the hard-won experience of human experts-making this knowledge available anytime.

同样，AI助手可以监视训练作业和推理服务器并标记异常。例如，助手可能正在监视训练作业并说，"嘿，损失在训练早期发散；也许检查您的数据输入是否有问题或降低学习率，"如图20-8所示。

Similarly, AI assistants could watch over training jobs and inference servers and flag anomalies. For instance, the assistant could be monitoring a training job and say, "Hey, the loss is diverging early in training; maybe check if your data input has an issue or reduce the learning rate," as shown in Figure 20-8.

Figure 20-8. AI assistant monitoring a long-running training job and suggesting actions to fix an anomaly

像Splunk（现为Cisco）和PagerDuty这样的公司已经在系统日志数据上使用AI模型来预测数据中心中的故障和检测异常。建议您扩展这些概念，以使用AI工作负载特定的遥测数据。

Already, companies like Splunk (now Cisco) and PagerDuty are using AI models on system log data to predict failures and detect anomalies in data centers. It's recommended that you extend these concepts to use AI workload-specific telemetry.

简而言之，AI为每个运行中的作业和每个推理服务器提供了一双永远新鲜的眼睛。它可以监视它们、建议它们并实时调整。传统的利用率指标可能会产生误导。例如，一个100%忙碌于冗余数据传输的GPU并不是高效的。这些AI驱动的调度器旨在最大化有效吞吐量，并确保当GPU忙碌时，它正在做有用的神经计算。这直接提高了成本效率。

In short, AI gives us an always-fresh pair of eyes for every running job and every inference server. It can monitor them, advise them, and adjust in real time. Traditional utilization metrics can be misleading. For instance, a GPU 100% busy on redundant data transfers isn't productive. These AI-driven schedulers instead aim to maximize goodput and make sure that when a GPU is busy, it's doing useful neural compute. This directly improves cost efficiency.

在AI集群中，例如，您可以使用基于Prometheus的指标管道来馈送基于LLM的助手，当GPU内存由于潜在内存泄漏或数据停顿而突然下降时发出警报。它甚至可以识别可能的根本原因。这是AI可以帮助自动化并24/7不间断、不分心地运行的繁琐工作类型。

In an AI cluster, for instance, you can use a metrics pipeline based on Prometheus to feed an LLM-based assistant that alerts when GPU memory suddenly drops due to either a potential memory leak or data stall. It can even identify likely root causes. This is the kind of tedious work that AI can help automate and run 24/7 without interruption and distraction.

AI在AI系统中的另一个强大用途是自动化调试和故障分析。当一个训练作业在其三个月运行的中途失败时，人类必须阅读错误日志、设备统计信息，甚至内存转储来弄清楚出了什么问题。是硬件故障？数值溢出？网络故障？

Another powerful use of AI is in automated debugging and failure analysis for AI systems. When a training job fails halfway through its three-month run, a human has to read through error logs, device statistics, and perhaps even memory dumps to figure out what went wrong. Was it a hardware fault? A numerical overflow? A networking hiccup?

在未来，AI系统可以消化所有这些数据，包括日志、指标和警报，并比今天更快地查明可能的原因。它可能会说，"节点42在作业崩溃前有5个ECC内存错误--可能是GPU上的HBM内存设备或通道问题。"或者，"损失在迭代10,000时变为NaN--可能是不稳定的梯度；考虑梯度裁剪。"

In the future, an AI system could digest all that data, including logs, metrics, and alerts, and pinpoint likely causes much faster than they do today. It could say, "Node 42 had 5 ECC memory errors right before the job crashed-likely an HBM memory device or channel issue on the GPU." Or, "The loss became NaN at iteration 10,000-perhaps an unstable gradient; consider gradient clipping."

通过从许多过去的事件中学习，AI故障排除器可以为工程师节省许多小时的侦探工作。一些大型计算站点已经在他们的事件数据库上训练模型来预测故障并建议修复。

By learning from many past incidents, the AI troubleshooter could save engineers many hours of detective work. Some large computing sites are already training models on their incident databases to predict failures and suggest fixes.

更进一步，RL可以应用于实时控制系统行为，这是固定算法无法轻易匹配的。例如，可以训练一个功率管理RL代理来持续调整频率和核心分配，以在实时系统中最大化每瓦性能。这个代理将通过实时分析系统来学习最优策略。

Taking things a step further, RL can be applied to real-time control of system behavior in ways that fixed algorithms cannot easily match. For example, a power-management RL agent could be trained to continuously tweak frequencies and core allocations to maximize performance per watt in a live system. This agent would learn the optimal policy by analyzing the system in real time.

另一个例子是主动管理AI模型中的内存。AI代理可以学习哪些张量保留在GPU内存中，哪些交换到CPU或NVMe--超越"交换最近最少使用的"等静态规则。通过观察实时访问模式，AI可以更高效地管理缓存。当模式不明显或依赖于工作负载时，这尤其有效。

Another example is actively managing memory in AI models. An AI agent could learn which tensors to keep in GPU memory and which to swap to CPU or NVMe-beyond static rules like "swap least recently used." By observing live access patterns, an AI can manage a cache more efficiently. This is especially effective when patterns are nonobvious or workload-dependent.

最先进的从业者已经在使用RL来优化缓存驱逐、网络拥塞控制等。超大规模系统的复杂性--具有数百个交互组件和资源--使它们成为这种基于学习的控制的理想候选。人类无法及时偶然发现最佳设置的可调旋钮太多了--而且能够以实时适应不同工作负载的方式。

Already, state-of-the-art practitioners are using RL to optimize cache eviction, network congestion control, and more. The complexity of ultrascale systems-with hundreds of interacting components and resources-makes them prime candidates for such learning-based control. There are just too many tunable knobs for a human to stumble upon the best settings in a timely manner-and in a manner that adapts to different workloads in real time.

对于性能工程师来说，AI辅助运营代理的兴起意味着角色将更多地转向编排和监督AI驱动的过程，而不是手动调整每个参数。这在某种程度上类似于飞行员如何管理现代飞机中的自动驾驶仪。他们仍然需要深厚的知识和监督，但毫秒级的控制是自动化的。驾驶特斯拉全自动驾驶（FSD）模式的人也是如此。驾驶员仍然需要知识和直觉来避免困难情况并防止事故，但车辆的控制由FSD软件自动化。

For the performance engineer, the rise of AI-assisted operational agents means the role will become more about orchestrating and supervising AI-driven processes rather than manually tweaking every single parameter. It's somewhat analogous to how pilots manage autopilot in a modern aircraft. They still need deep knowledge and oversight, but much of the millisecond-by-millisecond control is automated. The same with someone driving a Tesla in Full Self-Driving (FSD) mode. The driver still needs knowledge and intuition to avoid difficult situations and prevent accidents, but the vehicle's control is automated by the FSD software.

为了指导AI助手高效管理我们的集群，我们只需设定目标、提供安全和公平的护栏，并处理AI以前未见过的新情况。负载均衡、故障恢复和内存缓冲区调优等常规优化由AI处理。拥抱这一范式对未来至关重要。

To guide the AI assistant to manage our cluster efficiently, we simply set the objectives, provide the safety and fairness guardrails, and handle novel situations that the AI hasn't seen before. Routine optimizations like load balancing, failure recovery, and memory-buffer tuning are handled by the AI. Embracing this paradigm will be important for the future.

那些坚持在如此复杂的AI系统中手工优化一切的人，将被那些拥抱AI辅助和自动调优的人超越。那些对AI自动化友好的人可以将他们的人力集中在新颖创新、复杂优化和创造性解决方案上。这是人类在这个勇敢的新AI世界中可以增加最大价值的地方。让AI处理其余的工作。

Those who insist on optimizing everything by hand in such complex AI systems will simply be outpaced by those who embrace AI assistance and autotuning. Those that are AI-automation-friendly can focus their human effort on novel innovations, complex optimizations, and creative solutions. This is where humans can add the most value in this brave new AI world. Let the AI handle the rest.

向百万GPU集群和100万亿参数模型扩展 (Scaling Toward Multimillion GPU Clusters and 100-Trillion-Parameter Models)

最后，让我们重新审视我们向超大规模、100万亿参数模型的探索。我们已经突破了万亿参数的门槛。现在的问题是如何在未来几年扩展到数十甚至数百万亿参数的模型。这种规模的模型对我们的系统有什么要求，需要什么创新才能使训练如此强大的模型成为可能？这就是我们讨论的所有内容汇聚在一起的地方，包括高效的硬件、智能的软件和巧妙的算法。达到100万亿参数模型将需要使用书中的每一个技巧--以及一些可能尚未被发现的技巧。让我们深入探讨！

Finally, let's revisit our quest toward ultrascale, 100-trillion-parameter models. We've already broken the trillion-parameter threshold. Now the question is how to scale to tens or even hundreds of trillion-parameter models in the coming years. What does that kind of model demand from our systems, and what innovations are needed to make training such a powerful model feasible? This is where everything we've discussed comes together, including efficient hardware, smart software, and clever algorithms. Reaching 100-trillion-parameter models will require using every trick in the book-and then some tricks that may not have been discovered yet. Let's dive in!

在硬件方面，明显的需求是更多的内存和更多的带宽--最好直接在GPU上。如果您有100万亿参数并且想要训练它们，您需要高效地存储和移动大量数据。下一代内存技术将至关重要。

On the hardware front, the obvious need is for more memory and more bandwidth-preferably right on the GPU. If you have 100 trillion parameters and you want to train them, you need to store and move an insane amount of data efficiently. The next generations of memory technology will be critical.

高带宽内存（HBM）继续发展。HBM3e与Blackwell代GPU一起使用，而HBM4在Rubin代GPU中使用。HBM4再次将每个堆栈的带宽翻倍--大约每个堆栈1.6 TB/s。它还将每个堆栈的容量增加到可能每个模块48 GB或64 GB。

High-bandwidth memory (HBM) continues to evolve. HBM3e is used with the Blackwell generation of GPUs, while HBM4 is used in the Rubin generation of GPUs. HBM4 doubles bandwidth per stack again-on the order of 1.6 TB/s per stack. It will also increase capacity per stack to possibly 48 GB or 64 GB per module.

HBM的更高容量和吞吐量意味着未来的GPU可能拥有，比如说，8或16个HBM堆栈，每个64 GB，总计单个板上有512 GB或1,024 GB的超快HBM RAM。这种本地HBM容量直接在每个GPU上保存大量模型参数--显著减少了换入换出数据的需要。

HBM's higher capacity and throughput mean that future GPUs could have, say, 8 or 16 stacks of HBM at 64 GB each, which totals 512 GB or 1,024 GB of superfast HBM RAM on a single board. That kind of local HBM capacity holds a lot of model parameters directly on each GPU-significantly reducing the need to swap data in and out.

不难看出这如何实现更大的模型、更高带宽的训练运行和更低延迟的推理服务器。过去需要跨8个GPU分片的东西可能适合一个。需要100个GPU的可能适合10个，依此类推。

It's not hard to see how this enables larger models, higher-bandwidth training runs, and lower-latency inference servers. What used to require sharding across 8 GPUs might fit in one. What required 100 GPUs might fit in 10, and so on.

除了像Grace Blackwell Superchip这样的多芯片架构外，多机架的NVL72可以链接成一个巨大的集群，创建共享统一快速网络的数百个GPU。本质上，从通信角度来看，您的集群表现得像一个巨型GPU。这对于扩展到100万亿参数很重要，因为这意味着我们可以不断添加GPU以获得更多总内存和计算--而不会遇到通信瓶颈墙。这假设NVLink（或类似技术）继续扩展到那些超大规模尺寸。

In addition to multichip architectures like the Grace Blackwell Superchip, multiple racks of NVL72s each can be linked into one giant cluster to create hundreds of GPUs sharing a unified fast network. Essentially, your cluster behaves like a single mega-GPU from a communication standpoint. This is important for scaling to 100 trillion parameters because it means we can keep adding GPUs to get more total memory and compute-without hitting a communication bottleneck wall. This assumes that NVLink (or similar) continues to scale to those ultrascale sizes.

然而，仅靠硬件无法解决100万亿参数的挑战。软件和算法创新同样重要，甚至更重要。例如，使用朴素数据并行训练这种规模的模型将极其缓慢和昂贵。想象一下优化器每一步必须更新100万亿权重！我们需要严重依赖减少有效计算的技术。

However, hardware alone won't solve the 100-trillion-parameter challenge. Software and algorithmic innovations are equally, if not more, important. Training a model of that size with naive data parallelism, for example, would be incredibly slow and expensive. Imagine the optimizers having to update 100 trillion weights every step! We will need to lean heavily on techniques that reduce the effective computation.

我们探索过的一个大领域是低数值精度。除了FP8和FP4，未来的硬件可能支持网络的某些部分更低（1位）精度。混合方案可能至关重要，对模型的大部分使用较低精度，但对敏感部分使用较高精度。

One big area that we explored is low numerical precision. In addition to FP8 and FP4, future hardware might support even lower (1-bit) precision for some parts of the network. Hybrid schemes will likely be critical to use lower precision for most of the model but higher precision for sensitive parts.

作为性能工程师，我们应该关注这些新功能并准备使用它们。要训练100万亿参数模型，您很可能需要使用低精度以提高效率；否则，工作负载将极其缓慢和昂贵。

As performance engineers, we should watch for these new capabilities and be ready to use them. To train 100-trillion-parameter models, you very likely need to use low precision for efficiency; otherwise, the workload would be prohibitively slow and expensive.

好消息是，硬件和库将使这种过渡相对无缝。我们已经看到CUDA通过NVIDIA的Transformer Engine（TE）和Tensor Core--以及充分利用CUDA的PyTorch和OpenAI的Triton--对低精度算术提供一流支持。

The good news is that hardware and libraries will make this transition relatively seamless. We're already seeing first-class support for low-precision arithmetic in CUDA through NVIDIA's Transformer Engine (TE) and Tensor Cores-as well as PyTorch and OpenAI's Triton, which fully leverage CUDA.

另一个关键方法是稀疏性和条件计算。我们已经在稀疏专家混合（MoE）等模型中使用稀疏激活，其中只有模型参数的一部分对给定输入活跃。这个想法可以推广，这样您不必每次都使用全部100万亿参数。相反，您只使用需要的部分。使用MoE架构的模型正在证明非常强大和高效。到100万亿参数模型到来时，我预计其中许多将需要稀疏激活。

Another critical approach is sparsity and conditional computation. We already use sparse activation in models like sparse mixture of experts (MoE), where only a fraction of the model's parameters are active for a given input. This idea can be generalized so that you don't always use the full 100 trillion parameters every time. Instead, you use just the parts you need. Models using the MoE architecture are proving to be very capable and efficient. By the time 100-trillion-parameter models arrive, I expect a lot of them will need to be sparsely activated.

对于性能工程师来说，这意味着吞吐量不仅关于矩阵乘法速度，还关于MoE条件路由的效率、专家输出的缓存和稀疏数据交换的通信模式。这增加了复杂性，但也带来了机会。如果您能确保正确的专家在正确的时间位于正确的设备上以最小化通信，您可以大幅加速这些大规模模型。

For performance engineers, the implication is that throughput will be about matrix multiplication speed as well as the efficiency of MoE conditional routing, caching of expert outputs, and communication patterns for sparse data exchange. This adds complexity but also opportunity. If you can ensure the right experts are on the right devices at the right time to minimize communication, you can drastically accelerate these massive models.

我们还应该考虑算法效率改进。使用更少内存的优化器可能至关重要。传统的Adam优化器变体通常保留两份额外的权重副本用于动量和方差估计。这实际上使内存使用量增加了两倍。所以如果您有100万亿参数权重，您需要额外的200万亿个值来保存优化器状态！像Adafactor和Shampoo这样的内存高效优化器有助于减少这种开销。

We should also consider algorithmic efficiency improvements. Optimizers that use less memory could be vital. The traditional Adam optimizer variants typically keep two extra copies of weights for momentum and variance estimates. This effectively triples memory usage. So if you have 100-trillion-parameter weights, you need an extra 200 trillion values to hold the optimizer states! Memory-efficient optimizers like Adafactor and Shampoo help to reduce this overhead.

激活检查点等技术通过重新计算激活而不是存储它们来帮助用计算换取内存。在100万亿参数规模下，您几乎肯定会积极使用检查点。一个更激进的想法可能是，我们不在每一步更新所有权重。考虑以轮换方式更新权重子集--类似于一个人可能不是每天给每棵植物浇水，而是轮流浇灌。如果做得明智，模型仍然可以有效学习，但每个参数的更新频率较低。这减少了系统的总计算需求。

Techniques like activation checkpointing help to trade compute for memory by recomputing activations instead of storing them. At a 100-trillion-parameter scale, you'd almost certainly be checkpointing aggressively. An even more radical idea is, perhaps, we don't update all weights on every step. Consider updating subsets of weights in a rotating fashion-similar to how one might not water every plant every day but rotate through them. If done wisely, the model still learns effectively but with less frequent updates per parameter. This reduces the total computational needs of the system.

这些想法模糊了算法设计领域，但性能感知的视角是有用的。我们应该问，"我们真的需要这么频繁或以这种精度做X吗？"对于训练和推理的每个方面。通常答案是我们可以找到一个仍然有效的更便宜的近似。在100万亿参数规模下，这些近似可以节省数月时间或数百万美元。

These kinds of ideas blur into the algorithm design realm, but a performance-aware perspective is useful. We should ask, "Do we really need to do X this often or at this precision?" for every aspect of training and inference. Often the answer is that we can find a cheaper approximation that still works. At a 100-trillion-parameter scale, these approximations can save months of time or millions of dollars.

超大规模训练中经常被忽视的一个方面是基础设施和网络。当您谈论在一个模型上工作的10,000+ GPU集群时，网络结构与GPU本身一样重要。以太网和InfiniBand技术在增加吞吐量和更智能的自适应路由技术等方面不断进步。

An often overlooked aspect of ultrascale training is infrastructure and networking. When you're talking about clusters of 10,000+ GPUs working on one model, the network fabric becomes as important as the GPUs themselves. Ethernet and InfiniBand technologies are advancing in terms of increased throughput and smarter adaptive routing techniques, etc.

NVIDIA的Spectrum-X是一个针对AI优化的基于以太网的结构（例如，RoCE、自适应路由、高双分带宽），可减少大规模训练和推理工作负载中的拥塞。

NVIDIA's Spectrum-X is an Ethernet-based fabric optimized for AI (e.g,. RoCE, adaptive routing, high bisection bandwidth) that reduces congestion in large-scale training and inference workloads.

性能工程师需要深入了解这些层级，并确保数据在正确的时间位于正确的位置。目标将是模拟一个跨越GPU和CPU的巨大内存空间，这样即使模型不适合一台机器，程序员也可以某种程度上透明地处理它。其中一些今天已经可以通过统一内存和使用cudaMemPrefetchAsync()在目标设备上预暂存页面并避免页面错误停顿的按需分页系统实现。但在100万亿参数规模下，这种功能将真正受到考验。

Performance engineers will need to deeply understand these tiers and ensure that data is in the right place at the right time. The goal will be to simulate a huge memory space that spans GPUs and CPUs so that even if a model doesn't fit in one machine, it can be treated somewhat transparently by the programmer. Some of this is already possible today with Unified Memory and on-demand paging systems using cudaMemPrefetchAsync() to to pre-stage pages on the target device and avoid page-fault stalls, for instance. But at a 100-trillion-parameter scale, this functionality will really be put to the test.

毫不奇怪，像xAI、OpenAI和Microsoft这样的前沿研究实验室正在构建1,000,000+ GPU的大型集群。在100万亿参数规模下，您可能有一个跨越整个数据中心硬件的作业。性能工程师必须在数据中心和多数据中心（全球）规模上思考。

It's not surprising that frontier research labs like xAI, OpenAI, and Microsoft are building large clusters of 1,000,000+ GPUs. At a 100-trillion-parameter scale, you might have one job spanning an entire datacenter's worth of hardware. Performance engineers must think at datacenter and multidatacenter (global) scale.

最后，随着模型及其所需计算的扩展，有一个社会技术趋势。任何单个团队--甚至单个公司--单独训练最大的模型可能变得不可行。我们（希望）将看到AI社区中更多的协作和共享来处理这些巨大项目。这将类似于大型科学项目--如粒子物理实验--如何涉及许多机构。类似于现已解散的Open Collective Foundation（一个非营利倡议）的倡议可以帮助汇集AI计算资源来训练100万亿参数模型，然后与世界分享。

Last, there's a socio-technical trend as models-and their required compute-scale up. It may become infeasible for any single team-or even single corporation-to train the biggest models alone. We (hopefully) will see more collaboration and sharing in the AI community to handle these enormous projects. This would be analogous to how big science projects-like particle physics experiments-involve many institutions. Initiatives similar to the now-dissolved Open Collective Foundation, a nonprofit initiative, could help pool AI compute resources to train a 100-trillion-parameter model, which would then be shared with the world.

这将需要标准化检查点格式、共同开发训练代码，并考虑模型的多方所有权。虽然这本身不是性能问题，但它将影响我们如何构建大型AI系统。我们需要使它们更加容错和易于快照共享，以便不同团队可以平滑高效地处理训练和推理工作流程的不同部分。作为工程师，您可能最终不仅优化纯速度，还要优化可重现性和互操作性。这允许不同团队平滑高效地处理训练和推理工作流程的不同部分。

This will require standardizing things like checkpoint formats, codeveloping training code, and thinking about multiparty ownership of models. While this is not a performance issue per se, it will influence how we build large AI systems. We'll need to make them even more fault-tolerant and easily snapshot-able to share partial results. As an engineer, you might end up optimizing for pure speed, as well as reproducibility and interoperability. This allows different teams to work on different parts of the training and inference workflow smoothly and efficiently.

达到100万亿参数模型将需要全栈创新。这个挑战没有单一的解决方案。相反，拼图的每一块都必须改进。硬件需要更快并保存更多数据。软件需要更多地自我优化--并通过编译器、AI助手和实时适应更高效地使用资源。算法需要通过稀疏性、更低精度和更好的优化器来巧妙地避免不必要的工作。

Reaching 100-trillion-parameter models will require holistic, full-stack innovations. There's no single solution to this challenge. Instead, every piece of the puzzle must improve. Hardware needs to be faster and hold more data. Software needs to self-optimize more-and use resources more efficiently through compilers, AI assistants, and real-time adaptation. Algorithms need to be clever about not avoiding unnecessary work through sparsity, lower precision, and better optimizers.

性能工程师的角色将是将所有这些进步整合到一个连贯的工作流程中。这就像组装一辆高性能赛车。发动机、轮胎、空气动力学和驾驶员技能都必须协同工作。如果我们做得对，现在看起来不可能的事情--例如，100万亿参数在不破产的情况下训练--将变得可实现。

The role of the performance engineer will be to integrate all these advancements into a coherent workflow. It's like assembling a high-performance racing car. The engine, tires, aerodynamics, and driver skill all have to work in unison. If we do it right, what seems impossible now-e.g., 100 trillion parameters trained without breaking the bank-will become achievable.

不久前，1万亿参数模型听起来还很疯狂。然而今天，这种规模已经由开源权重模型如Moonshot AI的Kimi K2（1万亿参数MoE，每个token活跃320亿参数）等证明。以这种进步速度，以及AI辅助的人类智慧，我们将在非常短的时间内征服下一个里程碑和数量级。

It wasn't long ago that 1-trillion-parameter models sounded crazy. Yet today, this scale has been demonstrated by open-weight models like Moonshot AI's Kimi K2 (1-trillion-parameter MoE, 32 billion parameters active per token) and others. At this rate of progress, and with AI-assisted human ingenuity, we will conquer the next milestones and orders of magnitude in a very short amount of time.

关键要点 (Key Takeaways)

以下要点总结了本章案例研究和超大规模AI系统性能工程假设未来状态中讨论的最佳实践和新兴趋势：

The following points summarize the best practices and emerging trends discussed in this chapter's case studies and hypothetical future states of ultrascale AI systems performance engineering:

协同设计的硬件和软件优化：LLM的性能改进真正来自于紧密集成的硬件/软件协同设计创新的突破。

Codesigned hardware and software optimizations: Performance improvements in LLMs are truly achieved by breakthroughs coming from tightly integrated hardware/software codesign innovations.

AI辅助编码和性能优化：Google DeepMind、NVIDIA和Predibase已经展示了核心内核（如矩阵乘法和注意力）的AI辅助发现和优化。这些努力表明，AI可以生成、测试和完善底层GPU代码，并在极少人工干预的情况下产生显著加速。

AI-assisted coding and performance optimizations: Google DeepMind, NVIDIA, and Predibase have demonstrated AI-assisted discovery and optimization for core kernels such as matrix multiplication and attention. These efforts show that AI can generate, test, and refine low-level GPU code and produce significant speedups with very little human intervention.

100万亿参数模型的策略：训练具有100万亿参数的模型将需要激进量化、多维并行（数据、流水线、张量、专家和上下文/序列）以及精心编排的机架间通信的融合。这强调了未来AI扩展既取决于硬件能力，也取决于软件级调度的独创性。

Strategies for 100-trillion-parameter models: Training models with 100 trillion parameters will require a blend of aggressive quantization, multidimensional parallelism (data, pipeline, tensor, expert, and context/sequence), and careful orchestration of inter-rack communication. This stresses that future AI scaling depends on both hardware capabilities and the ingenuity of software-level scheduling.

指数级计算基础设施扩展：下一代AI数据中心正在设计中，以提供数量级的计算能力增加。这些设施将训练计算预算远超当今水平的AI模型。这使使用当前系统100到1000倍FLOPS的训练运行成为可能。

Exponential compute infrastructure scaling: Next-generation AI data centers are being designed to provide orders-of-magnitude increases in computational capacity. These facilities will train AI models with compute budgets far beyond today's levels. This enables training runs that use 100 to 1,000 times the FLOPS used in current systems.

不断发展的AI模型和代理：未来模型将是能够生成和优化代码、用新数据持续更新权重、甚至重写自己代码的自我改进系统。这种学习和完善的永久循环将缩短突破之间的时间，并创建在研究和R&D任务中超越人类团队的虚拟劳动力。

Evolving AI models and agents: Future models will be self-improving systems capable of generating and optimizing code, continuously updating their weights with fresh data, and even rewriting their own code. This perpetual cycle of learning and refinement will reduce the time between breakthroughs and create a virtual workforce that outperforms human teams in research and R&D tasks.

AI辅助的实时故障排除：除了调度，AI副驾驶还将监视系统日志和训练/推理工作负载以快速检测异常--包括准确性损失激增或硬件错误数量。这些副驾驶可以帮助自动化调试、执行故障分析，甚至通过强化学习学习最优配置。这些有助于最大化每瓦和每单位时间的性能。

AI-assisted real-time troubleshooting: In addition to scheduling, AI copilots will monitor system logs and training/inference workloads to detect anomalies quickly-including spikes in accuracy loss or number of hardware errors. These copilots can help automate debugging, perform failure analysis, and even learn optimal configurations through reinforcement learning. These help to maximize performance per watt and per unit time.

每瓦性能，一个关键指标：所有这些协同设计努力最终都旨在最大化每单位成本的吞吐量。具体而言，目标是每美元每瓦功率每秒处理和生成更多token。例如，Grace Blackwell NVL72机架系统将每瓦性能比之前的Hopper代提高了25倍。这直接转化为比前一代GPU集群更低的每token成本。

Performance-per-watt, a critical metric: All of these codesign efforts ultimately aim to maximize throughput per unit cost. Specifically, the goal is to process and generate more tokens per second per dollar per watt of power. For example, the Grace Blackwell NVL72 rack system dramatically improves performance-per-watt 25x over the prior Hopper generation. This directly translates to lower cost per token than previous-generation GPU clusters.

结论 (Conclusion)

本书标志着AI系统性能工程领域的一个转折点。NVIDIA将CPU和GPU紧密集成到Grace Hopper和Grace Blackwell（以及即将推出的Vera Rubin和Feynman）等超级芯片模块中，实现了新的计算效率和规模水平。在底层，GPU使用高度优化的Tensor Core--以及为LLM计算基础优化的transformer引擎。

This book marks a turning point for the field of AI systems performance engineering. NVIDIA's tight integration of CPU and GPU into superchip modules like Grace Hopper and Grace Blackwell (and upcoming Vera Rubin and Feynman) has achieved new levels of compute efficiency and scale. Under the hood, the GPUs use highly optimized Tensor Cores-as well as a transformer engine optimized for LLM computation fundamentals.

像NVIDIA GB200/GB300 NVL72这样的超级计算系统使用NVLink、NVSwitch和SHARP等技术，将72个GPU链接成一个处理单元（NVLink域），为机架和数据中心通信奠定了基础。这为万亿参数模型提供低延迟、实时推理。

Supercomputing systems like the NVIDIA GB200/GB300 NVL72, which links 72 GPUs into a single processing unit (NVLink domain), set the rack and data center communication foundation using technologies like NVLink, NVSwitch, and SHARP. These provide low-latency, real-time inference for multitrillion-parameter models.

在软件方面，像vLLM、SGLang、NVIDIA Dynamo和TensorRT-LLM这样的工具改善了大型推理集群中的调度和资源使用。这包括飞行中批处理、分页KV缓存和（将提示预填充阶段与生成解码阶段分离到不同资源池以提高效率）等技术。这些有助于减少尾部延迟并提高每瓦吞吐量。

On the software side, tools like vLLM, SGLang, NVIDIA Dynamo, and TensorRT-LLM improve scheduling and resource usage across large inference clusters. This includes techniques like in-flight batching, paged KV cache, and (separating the prompt prefill stage from the generation decode stage onto different resource pools for efficiency.) These help to reduce tail latency and improve throughput per watt.

这些例子证明了协同设计的力量，其中硬件、软件和算法共同演进。这种植根于机械同情的伙伴关系有助于减少训练时间、提高推理性能并降低运营费用。这对于为当今快速改进和资本密集的AI系统产生可衡量的投资回报是必要的。

These examples prove the power of codesign, in which hardware, software, and algorithms evolve together. This partnership rooted in mechanical sympathy helps to reduce training times, improve inference performance, and lower operational expenses. This is needed to produce measurable returns on investment for today's fast-improving and capital-intensive AI systems.

此外，来自Google DeepMind、NVIDIA和Predibase的AI驱动编码和算法代理展示了AI如何帮助优化AI。随着模型和系统变得过于复杂而无法手动调优，自动化可以处理常规优化，使人类工程师能够专注于更高级的优化和系统设计。

Additionally, AI-driven coding and algorithm agents from Google DeepMind, NVIDIA, and Predibase show how AI can help optimize AI. As models and systems become too complex for manual tuning, automation can handle routine optimizations and free human engineers to focus on higher-level optimizations and system designs.

我们正在从暴力扩展转向智能扩展：每个周期做更多有用的工作，从新硬件功能中榨取每一盎司的性能，让AI助手管理细节。性能工程师将向上移动堆栈，成为平衡效率、可靠性和可持续性的全球计算生态系统的架构师。

We're shifting from brute-force scaling to smart scaling: doing more useful work per cycle, squeezing every ounce of performance from new hardware features, and letting AI assistants manage the details. Performance engineers will move up the stack, becoming architects of global compute ecosystems that balance efficiency, reliability, and sustainability.

我们作为AI系统性能工程师的角色将从单节点内核扩展到系统级和设施级优化。我们将依靠我们的直觉来发现瓶颈--比如缓慢的all-reduce模式--然后指导我们的AI工具来修复它们。同时，我们将继续学习，因为硬件和算法创新的步伐只会加速。

Our role as an AI systems performance engineer will expand beyond single-node kernels to system-wide and facility-wide optimization. We'll rely on our intuition to spot bottlenecks-like a slow all-reduce pattern-and then guide our AI tools to fix them. Meanwhile, we'll keep learning, since the pace of hardware and algorithmic innovation will only accelerate.

总之，为了保持相关性和竞争力，您应该在AI系统性能基础方面建立坚实的基础、保持好奇心、实验新的硬件和软件进步、信任AI建议，并准备好随着景观变化适应量子等领域。试想一下--在民主化研究、一键访问的AI超级计算机和可访问的万亿参数模型时代，您可能是下一个超级智能突破的推动者之一！

In conclusion, to stay relevant and competitive, you should build a strong foundation in AI systems performance fundamentals, stay curious, experiment with new hardware and software advancements, trust AI recommendations, and be ready to adapt as the landscape changes into quantum and beyond. And just think-in an era of democratized research, one-click-accessible AI supercomputers, and accessible multitrillion-parameter models, you could be one of the enablers of the next big superintelligence breakthrough!