CuTe DSL FlashAttention: Optimizing a Triton Kernel

Résumé

Pixel shares a detailed walkthrough of rewriting a Triton FlashAttention-2 kernel in NVIDIA CuTe DSL and adding optimizations one at a time. The article measures performance on an RTX 4070 SUPER, covering shared-memory swizzling, ldmatrix, register storage, cp.async, and tile and stage choices under a 99 KB limit.

DEV Community

Rewriting My Triton FlashAttention Kernel in CuTe DSL, One Optimisation at a Time

Table of Contents What Triton did under the hood The setup CuTe in four ideas Round 0: a straightforward port Round 1: swizzle the shared memory Round 2: ldmatrix Round 3: keep P in registers Round 4: cp.async Round 5: tiles and stages under 99 KB The final profile, and was it a fair fight? Round 6: beyond Triton's tl What CuTe makes you write by hand, as a checklist Future articles In my previous article, I tuned a FlashAttention-2 (FA2) forward kernel in Triton until it matched PyTorch's scaled_dot_product_attention (SDPA) on my RTX 4070 SUPER. Triton automatically handled where each value lives, how data moves into shared memory, and in what order. What are registers, shared memory and global memory? Global memory is the GPU's main memory (DRAM, 12 GB here), reached through two caches. L2 (48 MB) is shared by the whole GPU, and L1 sits inside each SM. Shared memory is a small scratchpad inside each SM (at most 99 KB per thread block here) that a block's threads fill and read by hand. Registers are each thread's own, fastest storage. In this article I write the same kernel in NVIDIA's CuTe DSL, a Python library for writing kernels at the level of CUTLASS, NVIDIA's C++ template library. I begin with a direct porting of code from Triton to CuTe DSL without all the optimisations that Triton provided for under the hood. This gives a baseline measurement of the performance of an unoptimised kernel. I then layer on each optimisation which Triton automatically provided, measuring the performance gain which each optimisation brings. The harness and the test file are copied unchanged, so the numbers compare directly with the previous article's. Here are the results at N = 4096, the median of the harness runs (10 for CuTe v5 and Triton v5, 3 for the other kernels, 35 for SDPA). Triton v5 (previous article, re-measured) CuTe v0 (straight port) CuTe v5 (this article) SDPA NVIDIA's CuTe example Non-causal 1.968 ms, 69.8 TFLOP/s 5.436 ms, 25.3 TFLOP/s 1.991 ms, 69.1 TFLOP/s 2.158 ms, 63.7 TFLOP/s 1.950 ms, 70.5 TFLOP/s Causal 1.079 ms, 63.7 TFLOP/s 2.997 ms, 22.9 TFLOP/s 1.029 ms, 66.8 TFLOP/s 1.118 ms, 61.5 TFLOP/s 1.038 ms, 66.2 TFLOP/s What is TFLOP/s? Trillions of floating-point operations per second. The harness counts 4·B·H·N²·D operations for attention (half that for causal) and divides by the time. B is the batch size (4 here), H the number of attention heads (8), N the sequence length in tokens (512 to 4096), and D the size of each head's query, key and value vectors (64). I also ran the kernels back to back for longer, logging clock speed and power (Was it a fair fight?). CuTe v5 and Triton v5 ran at nearly the same clock (within 1.1%) and hit the power cap equally often. Under those matched conditions, the results were as follows. Causal. CuTe v5 is 4–5% faster for N ≥ 1024. At N = 4096 it is still 4% faster when Triton uses the same tile sizes, so the gap does not come from differences in tile size. Non-causal. CuTe v5 is 4% faster at N = 1024 and 2% faster at N = 2048. At N = 4096 they are level. N = 512. A tie, with or without the causal mask. What are clock, throttling and power cap? The SM clock is how many cycles per second the GPU's cores run, and every kernel's speed scales with it. Throttling is the driver lowering that clock to stay within a limit. The power cap is the limit on board power, 220 W here. Thermal slowdowns are limits on temperature. NVML, the driver's monitoring library, reports which of these reasons is active. Below are the four optimisations which Triton implemented under the hood, and which I have to manually implement in CuTe DSL. a swizzled shared-memory layout. This change alone made the straight port about twice as fast (5.44 → 2.69 ms at N = 4096, non-causal) matrix loads with ldmatrix keeping P in registers asynchronous copies with cp.async. A sweep of tile and pipeline sizes then gained 0–4%. This article assumes you know the Flash Attention 2 (FA2) forward pass and the previous article. T...
CuTe DSL FlashAttention: Optimizing a Triton Kernel
Powered by Readybot.io

Les dernières actus de Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …