One Small Step for (DeepSeek) AI
The use of AI generated code, to further develop itself or for other functions, continues to evolve. One challenging use case has been the code set of instructions for the creation of the now famous NVIDIA GPU chips. Creating optimized GPU kernels for attention is a challenging task that requires significant skill and time — even for experienced software engineers. Up until now, LLMs like DeepSeek-R1 have shown promise in code generation tasks but still face challenges in creating optimized code on the first try.
To address these challenges, NVIDIA engineers developed a new workflow that uses the DeepSeek-R1 model in a closed-loop fashion to generate optimized GPU kernels, producing numerically correct kernels for 100% of Level-1 problems and 96% of Level-2 problems in the Stanford KernelBench benchmark.
The key point here is that, in some cases, these auto-generated kernels outperformed those meticulously crafted by seasoned engineers.
The results demonstrate the potential of using the DeepSeek-R1 model with inference-time scaling to generate optimized GPU kernels. More work is needed to consistently produce better results for a wider variety of problems, but the approach now shows promise for automating GPU kernel generation.
NVIDIA's latest technical blog dives deep into the concept of test-time scaling (or inference-time scaling) — a method that dynamically allocates additional computational resources during inference to evaluate multiple outcomes and select the optimal one. This approach allows AI models to strategize and tackle complex problems more effectively.
- Key Insights: Challenges in Kernel Optimization: Developing these optimized GPU kernels is non-trivial, demanding significant expertise and time. Innovative Closed-Loop Workflow: To overcome these hurdles, NVIDIA engineers devised a closed-loop workflow. It integrates DeepSeek-R1 with a specialized verifier during inference. A manual prompt proceeds to GPU code generation, and the verifier analyzes the output. New prompts are then crafted based on this feedback, iteratively refining the kernel quality. Results and Future Potential: After about 15 minutes of inference-time scaling, the workflow produced significantly improved attention kernels. While further work is needed to generalize this approach across a broader set of problems, these initial results underscore the promise of automating GPU kernel generation with advanced AI models. This innovative approach not only streamlines the process of kernel optimization but also paves the way for more dynamic, self-improving systems in AI. The fusion of inference-time scaling with intelligent code generation could be a game changer for performance-critical applications in deep learning.
- Challenges in Kernel Optimization: Developing these optimized GPU kernels is non-trivial, demanding significant expertise and time.
- Innovative Closed-Loop Workflow: To overcome these hurdles, NVIDIA engineers devised a closed-loop workflow. It integrates DeepSeek-R1 with a specialized verifier during inference. A manual prompt proceeds to GPU code generation, and the verifier analyzes the output. New prompts are then crafted based on this feedback, iteratively refining the kernel quality.
- Results and Future Potential: After about 15 minutes of inference-time scaling, the workflow produced significantly improved attention kernels. While further work is needed to generalize this approach across a broader set of problems, these initial results underscore the promise of automating GPU kernel generation with advanced AI models.
Sources
Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling