Building Strong Safety Evaluations
Anthropic, Friday, November 13th, 2026
Anthropic shares recommendations for designing, running and reporting AI safety evaluations, drawing on psychology's replication reforms.
Anthropic explains that safety evaluations inform decisions such as whether a model is ready to deploy and which safeguards ship with it, so a safety claim is only as sound as the eval behind it.
A safety eval measures a model's propensity to cause or contribute to harm rather than specific capabilities.
Evals can be gamed, may not measure what they claim, and can shift with question format, problems similar to those psychology addressed through preregistration and more rigorous statistics.
The post recommends involving both methods experts and domain experts from the start, and running evals in sandboxes without internet access and with real-time monitoring, noting Anthropic hardened its own environments after incidents.