AWS unifies AI agent evaluation with an open-source framework
AWS has released UAEF, an open-source framework to compare AI agents on a common basis. The tool targets LangGraph, LangChain, Strands Agents, and Bedrock AgentCore, with 36 metrics and reproducible comparisons.
On November 11, 2026, AWS released a technical presentation of the Universal Agent Evaluation Framework, or UAEF, an open-source system licensed under Apache-2.0 designed to evaluate AI agents with 36 metrics across seven dimensions. The challenge is simple to state: to properly compare agents built on different frameworks without breaking scores with every change of architecture, prompt, or tool. In this context, evaluation stops being a local patchwork and becomes a more stable checkpoint.
A common foundation for agents that don’t speak the same language
UAEF targets LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore, environments that record messages, tool calls, and transfers between agents differently. Without standardization, a framework change often requires rewriting the evaluation logic, metrics, and sometimes even the storage of results. AWS states that 12 organizations in telecom, healthcare, and financial services have used UAEF to apply consistent metrics and compare multiple agent versions before production deployment.
The core of the system relies on a canonical trace, called AgentTrace, to which adapters convert each execution. This single trace becomes the calculation base for the evaluation engine. The framework can also reuse a trace from a previous run, a log, or an observability tool, or launch an agent via an Amazon Resource Name or an HTTP endpoint. For in-house frameworks, a generic JSON adapter takes over. Convenient, especially when no one wants to redo plumbing for every new platform.
36 metrics, but not 36 ways to tell the same story
The metrics catalog covers response quality, tool calling, security and responsible AI, performance, multi-turn conversations, multi-agent coordination, and reasoning. Scores are normalized on a scale from 0 to 1, with 1 as the best value. Some measures are deterministic and calculated directly from traces and ground truth; others use a judge model on Amazon Bedrock to rate relevance or safety.
Expected responses are not mandatory in all cases. However, metrics based on reference comparison require them. For tool call evaluation, UAEF can also define expected tools, their order, and arguments. If a metric lacks information, it returns an empty score with a warning, then this result is excluded from dimension averages instead of being counted as zero. This allows evaluation of partially annotated datasets or even production traffic without complete labeling, as long as available inputs suffice.
Comparisons designed to spot regressions
Once persisted, evaluations become experiments. Amazon DynamoDB stores metadata useful for listing and comparison, including the number of evaluations and average scores, while Amazon S3 keeps detailed JSON results, with scores per metric and judge reasoning. Each run can record the framework, model, tools, system prompt, parameters, a configuration snapshot, and the associated Git commit. The decision chain thus remains linked to the code actually tested.
For metrics shared between two runs, the engine calculates absolute and relative variations compared to a reference baseline. By default, a configurable 5% threshold separates improvements, regressions, and unchanged scores. Regressions are further classified as minor, moderate, or severe, then grouped by dimension. A prompt change can therefore degrade tool usage without affecting response quality. This is the kind of detail you prefer to see before production rather than after.
An architecture made for teams, not just notebooks
AWS emphasizes the separation between reproducible statistics and model-generated interpretation. The web interface displays results by test case and dimension. An optional statistical report adds score distributions, confidence intervals, outliers, correlations between metrics, and failing cases, without using an LLM. Separately, an analysis produced by Bedrock can suggest probable causes and priority next steps for a run or comparison.
The Python library and low-code web service rely on the same engine and storage. Teams can therefore work in notebooks, scripts, CI/CD pipelines, or directly in a browser without changing measurement logic. The documentation notes, however, that installation requires Python 3.10 or higher and the uv package manager. And for judgment by Bedrock or persistence in DynamoDB and S3, AWS credentials remain necessary.
Security, isolation, and framework limits
UAEF treats the output of the evaluated agent as an untrusted input for judgment prompts. The system therefore separates the scoring rubric, escapes content, strictly analyzes judge responses, and requires a compliant proof snippet before accepting a score. AWS also specifies that judge scores should not drive security controls. Enforcement of safeguards must go through a separate path via Amazon Bedrock Guardrails.
The repository presents UAEF as a reference architecture designed to operate within a platform that already provides isolation between users and tenants. Direct multi-tenant deployment is not supported without this framework. Each tenant must have a separate deployment, with its own AWS account or, at minimum, its own stack, DynamoDB tables, S3 buckets, and Cognito user pool. In other words, the framework standardizes evaluation, not the internal organization of the company [to be verified].
For companies running multiple agent frameworks side by side, UAEF provides a more stable measurement foundation against changes in prompts, models, tools, or implementation. But continuity does not come only from the library: it also depends on metric choice, dataset quality, and isolation applied around the system. The framework promises cleaner comparison; the rest, as often in AI, is integration work.
Key points
- UAEF includes 36 metrics across seven dimensions.
- 12 pilot organizations have used the framework, according to AWS.
- The default regression threshold is set at 5%.
- The repository targets LangGraph, LangChain, Strands Agents, and Bedrock AgentCore.
- The framework is published under the Apache-2.0 license.
By the numbers
- 36 metrics — catalogued by AWS in UAEF, November 2026.
- 7 dimensions — tool calling, quality, security, performance, multi-turn, multi-agent, reasoning.
- 12 organizations — pilots cited by AWS, in telecom, healthcare, and finance.
- 5% — default threshold to distinguish improvement and regression.
- Python 3.10+ — installation prerequisite mentioned in the documentation.