When a Faster Kernel Doesn't Speed Up Serving: Profiling FP8 KV Cache on AMD Instinct MI308X
AMD, Wednesday, July 15th, 2026
AMD engineers explain why a faster FP8 KV cache kernel failed to improve end-to-end LLM serving throughput on MI308X.
This post investigates a counterintuitive result in which an optimized FP8 KV cache kernel showed microbenchmark gains but no improvement in production LLM serving throughput.
AMD engineers use profiling to trace the discrepancy to stalls elsewhere in the serving pipeline that dominate total latency. The analysis illustrates how kernel-level benchmarks can mislead when the bottleneck sits in scheduling, memory movement, or batching rather than compute.
The takeaway is a methodology for validating kernel work against end-to-end serving metrics before declaring a win.