IBM Research, writing on the Hugging Face Blog, published an article on September 15, 2026, detailing a new approach to improving the reliability of AI agents. The article, authored by Evelyn Duesterwald, Lilian Ngweta, Vatche Isahagian, Jayaram Radhakrishnan, Vinod Muthusamy, Gaodan Fang, Ashwath Vaithinathan Aravindan, Punleuk Oum, G Thomas, Merve Unuvar, Ayhan Sebin, and Michał Ulewicz, introduces "consistency guidelines" as part of the ALTK-Evolve system. These guidelines are designed to mitigate the problem of agents failing to repeat tasks they have previously completed successfully, a critical issue for production environments and mission-critical applications.
The authors highlight that traditional benchmarks often obscure this variability by reporting average success rates. For instance, a ReAct agent using GPT-4.1 on AppWorld achieved a Mean@5 (average success over five runs) of 77.4%, but its Pass^5 (succeeding in all five runs) was only 53.0%. This 24.4-point difference is termed the "consistency gap."
Key Points
- IBM Research introduced "consistency guidelines" within its ALTK-Evolve system on September 15, 2026.
- These guidelines address the problem of AI agents failing to consistently repeat tasks they have previously succeeded at.
- A ReAct agent using GPT-4.1 on AppWorld showed a Mean@5 of 77.4% but a Pass^5 of only 53.0%.
- The difference between Mean@k and Pass^k is defined as the "consistency gap," which was 24.4 points in the GPT-4.1 example.
- The problem is attributed to "flat" probability distributions in LLM decisions, where multiple tokens have similar likelihoods, making outcomes vulnerable to minor perturbations.
- The solution involves a two-stage pipeline: a Consistency Analyzer to detect decision steps prone to variability, and the generation of targeted consistency guidelines.
- The Consistency Analyzer uses controlled resampling, replaying each decision step with k=5 completions by default, to identify inconsistent decisions.
Context
According to IBM Research, most agent evaluations report Mean@k, which averages the pass rate over k runs (often k=3 or k=1). This metric answers how good an agent is on average but does not indicate if it will perform consistently when asked the same question again. For that, the authors propose Pass^k, which measures the fraction of tasks where the agent succeeds on all k runs. The article clarifies that Pass^k is distinct from Pass@k, which optimistically asks if at least one of k attempts succeeded. The consistency gap arises because even with a temperature 0.0 setting, slight shifts in hosted endpoint probabilities can reorder near-tied token choices, leading to different outcomes across runs.
Why It Matters
This research highlights a critical reliability challenge for builders deploying AI agents in production, particularly for mission-critical applications. The distinction between average performance and consistent performance reveals that an agent can be capable yet unreliable, necessitating new evaluation metrics and intervention strategies to ensure predictable behavior.
What To Do
- Note the difference between Mean@k and Pass^k when evaluating agent benchmarks.
- Consider the implications of the "consistency gap" for agents deployed in production environments.
- Watch for further details on how "consistency guidelines" are generated and integrated into agent workflows.
- Review the described mechanism of the Consistency Analyzer for detecting decision steps prone to variability.