How We Investigated Prompt Cache Misses with Datadog
Datadog, Friday, November 6th, 2026
Datadog used Cloud Cost Management and Agent Observability to find why its internal AI apps were missing LLM prompt caches.
Datadog investigated why some internal AI applications were not benefiting from prompt caching despite it appearing enabled, using Cloud Cost Management for provider spend and Agent Observability to inspect individual model requests.
Requests without cache hits each carried more than 180,000 input tokens at $0.74 to $0.82 in input cost, versus $0.02 to $0.14 total per request when caching worked.
Causes included changing values such as timestamps placed early in system prompts, which broke prefix matching, and a Go map that sent tool definitions in inconsistent order.
Fixes included placing stable instructions first and sorting tool names, and Datadog also found some apps enabled caching without sending the information providers needed.