A paper published on arXiv cs.CY titled "When Is an Agent Evaluation Over? Outcome Finality and Cross-Unit Separation" examines the conditions necessary for interpreting agent evaluation scores as final results. The authors propose that two independent conditions, outcome finality and cross-unit separation, are essential but not inherently established by the endpoint of a stopped run.
Key Points
- Current agent evaluations typically score models based on the state visible at the end of a stopped run, counting it as one trial.
- Interpreting such scores as final results requires both outcome finality and cross-unit separation, which are not guaranteed by the run's endpoint.
- Outcome finality means anything that could change the claimed outcome is resolved, bounded, or retained as uncertainty.
- Cross-unit separation ensures that runs are isolated to prevent state carryover between them.
- A controlled replay experiment demonstrated that endpoint and terminal labels differed for every delayed operation when agent actions were held fixed.
- The experiment also showed that a delayed write altered the next run's score if service state persisted between runs, but not after isolation or verified reset.
- A review of ten public protocols indicated that all protocols identify issues related to these conditions.
Context
The paper develops a completion argument to specify the evidence required for decisions regarding outcome finality and cross-unit separation. According to the authors, a final label is justified only when all potential changes to the claimed outcome are resolved, bounded, or explicitly noted as uncertain. The research includes a controlled replay experiment designed to demonstrate the mechanisms at play. In this experiment, agent actions were held constant, and it was observed that the final state at the run's end often did not match the true terminal label due to delayed operations. Furthermore, the persistence of service state between runs allowed delayed writes to influence subsequent run scores, a problem mitigated by isolating runs or performing verified resets. The paper also includes a review of ten public protocols, finding that all of them exhibit challenges related to these two conditions.
Why It Matters
For builders and researchers, understanding outcome finality and cross-unit separation is crucial for designing robust agent evaluation methodologies. Misinterpreting evaluation scores due to unresolved outcomes or state leakage between trials can lead to inaccurate conclusions about agent performance and reliability, impacting development and deployment decisions.
What To Do
- Review existing agent evaluation protocols to identify how they address or fail to address outcome finality and cross-unit separation.
- Consider implementing mechanisms for verifying outcome finality, such as waiting for all asynchronous operations to complete or explicitly bounding the time for outcomes to settle.
- Ensure strict isolation between evaluation runs to prevent state leakage and carryover effects, using verified resets or dedicated environments for each trial.
- Note the potential for delayed operations to alter final scores and design evaluation systems that account for these temporal discrepancies.