Compare AI Agent Changes with Online Experiments in Agent Observability
Datadog, Tuesday, November 3rd, 2026
Datadog Online Experiments in Agent Observability lets teams test AI agent changes against live production traffic.
Datadog says changing an agent's prompts, models, retrieval or tools can have unclear effects, and offline datasets take time to build and may miss real-world requests.
Online Experiments, available in Agent Observability, compares agent variants using live production traffic and combines agent evaluations with Datadog Experiments.
Teams can assess response quality alongside latency, cost, token usage and errors, and investigate traces to understand behavior differences.
The post also describes how Datadog used the feature internally to evaluate a model change.