Capability matters when it survives scrutiny.
A system should be judged by what it does across changing conditions, not by how confident it sounds in a demonstration. Good evaluation makes strengths, limits, and failure modes visible.
The central question.
Which tests reveal failures beyond familiar benchmarks?
Questions that follow
- How should we evaluate actions that affect external systems?
- What new risks appear when models gain access to tools?
How we study it
A strong evaluation uses transparent protocols, reproducible baselines, adversarial tests, and careful analysis of uncertainty. Tests should cover normal use, edge cases, and recovery when something goes wrong.
What meaningful progress looks like
Measure usefulness and robustness together: task success, unsafe-action rates, failure detection, recovery, and the value of human intervention.
Explore all research programsRead the work behind these questions.
These independent papers and official guidance offer useful starting points for this field. Follow each link to review the original methods, evidence, and limits in context.