Microsoft has introduced a tool intended to make AI-agent testing more systematic, connecting risk discovery, evaluation and runtime controls in one workflow. For companies adopting agents that can take actions, its business significance lies in the evidence a team can demand before expanding access.
The company’s September 24 technical post describes run-assert-eval, highlighted in Microsoft’s news feed on September 25. Microsoft says it combines Clarity, ASSERT and Agent Control Specification to identify risks, measure failures, generate policy and repeat the evaluation. A developer still reviews which risks to prioritize.
Keep the comparison meaningful
Microsoft says the workflow keeps the behavior definition, test cases and evaluator constant when comparing the agent before and after a control is applied. The post demonstrates a billing-support example involving customer verification and access to other accounts. These are vendor-described capabilities and examples, not an independent certification that an agent is safe in every setting.
For a buyer, the useful question is what a reported improvement actually measures. If a supplier changes both the safeguard and the test, a better score may not isolate the effect of the safeguard. A clear comparison should identify what changed, what stayed the same and which failures remain.
Imagine a service agent that can draft a refund but must obtain approval before issuing it. A demonstration showing polite answers is not enough to establish that the approval boundary works. A purchasing team could ask for tests covering an ordinary refund, a mistaken identity, an ambiguous request and a request outside the agent’s permitted scope. Those examples are a suggested evaluation approach, not reported results from Microsoft’s release.
Measure useful work as well as refusals
A control can make an agent refuse more often while also preventing legitimate tasks. Managers therefore need to ask whether the system completes permitted work accurately as well as whether it blocks prohibited actions. A service that avoids mistakes by declining everything has not delivered the business function it was purchased to perform.
Record the practical cost of review too. Who examines a failed case? How long does it take to explain the decision to a customer? Can the team reproduce the issue after a software update? Those questions connect evaluation to staffing and operations without assuming that a single benchmark score predicts the entire workload.
The same caution applies to the word “proof.” A successful rerun can support a conclusion about the tested conditions. It cannot establish the absence of every possible failure in future conversations, new integrations or changed permissions. Decision-makers should ask for the scope of the evidence in ordinary language.
Put the findings into the release decision
A sensible pilot record would include the task, access permissions, test version, observed failures and the person responsible for approving expansion. That gives a team something concrete to revisit when the agent changes.
The issue complements our coverage of Microsoft’s security-operations preview. More capable agents make testing and oversight part of the product decision. The new workflow is worth assessing on how clearly it exposes limitations, alongside the failures it helps a team fix.
Illustrative electronics; not a photograph of the Microsoft evaluation software. Photo: Pixabay / Pexels.