AWS introduces open-source system to compare agents across frameworks

AWS introduced the Universal Agent Evaluation Framework, an open-source system that lets developers compare AI agents built with different frameworks using consistent metrics. AWS says customers across 12 organizations in telecom, healthcare and financial services used it to compare agent versions before release. The system converts records of agent messages, tool calls and handoffs into a common format, then scores them using 36 built-in metrics across 7 dimensions. Available under the Apache 2.0 license, it runs through a Python library or a deployable AWS service with a web interface. Some metrics use model judges; AWS cautions that their scores should not gate security controls.