"构建严谨 Agent 基准的最佳实践"

Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench-Verified uses insufficient test cases, while τ-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%.

AI agents that integrate machine learning models with tools, memory, and knowledge are emerging with the capability to solve complex problems [12, 25, 40, 68, 71, 84, 85]. To evaluate AI agents, researchers and practitioners have built agentic benchmarks with realistic tasks to track progress and assist decision-making [11, 22, 30, 37, 47, 59, 83, 86, 89, 93]. AI agents have exhibited impressive performance on these benchmarks. For example, a GPT-4o-based agent resolves 35% of tasks on τ-bench-Airline, a benchmark for tool-agent-user interaction [86]. As agentic benchmarks become increasingly impactful in academia and industry, it is crucial to ensure these numbers can be trusted.

Agentic benchmarks differ fundamentally from traditional AI benchmarks. Multiple-choice datasets (e.g., ImageNet [16] and MMLU [27]) evaluate models by their accuracy on categorical labels, while text-generation benchmarks rely on automatic metrics (e.g., BLEU [60]). By contrast, success is defined by completing end-to-end tasks in agentic settings. Therefore, an agent may perform coherent reasoning, write code, and execute commands to produce a final outcome. Subsequently, the performance of the agent is determined by comparing its final outcome with a ground-truth outcome, using various methods, such as program testing and string matching [11, 30, 37, 47, 59, 83, 86, 89, 93].

Unfortunately, many existing outcome-based evaluation methods of agentic benchmarks introduce issues that can cause under- or overestimation of agent capabilities by up to 100% in relative terms, compromising the validity of their findings [35, 46, 62, 80, 87]. For example, SWE-bench-Verified challenges an agent to resolve GitHub issues, and considers the agent successful if the patch it generates passes manually vetted unit tests [14]. However, recent work has shown that passing these tests does not necessarily indicate that the issue is resolved, because unit tests can fail to capture important edge cases. Consequently, 24% of the top 50 leaderboard positions are incorrect [31, 87]. In addition, we find that in τ-bench, a trivial agent that returns empty responses is considered successful on intentionally impossible tasks (e.g., changing a non-refundable ticket). This trivial agent achieves a 38% success rate and outperforms a GPT-4o-based agent [86].

Although issues in evaluation rigor can significantly skew evaluation results, they are still frequently overlooked in the current development, deployment, and analysis of agentic benchmarks. To better understand this problem, we analyzed prior work on agentic benchmark pitfalls [35, 46, 62, 80, 87] and 17 widely used agentic benchmarks (Table 3), such as SWE-bench-Verified [14], GAIA [47], τ-bench [86], and WebArena [93]. Combining insights from the literature with our own experience in developing benchmarks, we identified two major conditions of the validity of benchmark results:

- **Outcome validity**: the evaluation result (e.g., tests or checks) truly indicates task success. SWE-bench-Verified fails here because an incorrect patch can still pass the test suite.
- **Task validity**: a task should be solvable if and only if the agent possesses the target capability. Issues in task design or implementation often breaks task validity. For example, τ-bench allows a trivial agent to pass 38% of tasks without knowledge of airline-ticketing rules.

Following prior work on analyzing AI and code benchmarks [10, 65], we formulate our insights into an Agentic Benchmark Checklist (ABC) to assist benchmark developers and users in critically designing and assessing agentic benchmarks. Using ABC, we assessed ten popular agentic benchmarks that span the full range of agent capabilities, resulting in seven benchmarks with flaws in outcome validity, seven with issues in task validity, and all with limitations in the result reporting. In addition to the issues found in τ-bench-Airline, some other example issues we found are: (1) an agent can score 100% on SWE-Lancer [48] without resolving any tasks; (2) KernelBench [59] overestimates agents' capabilities in generating correct kernel functions by 31% in absolute terms due to incomprehensive fuzz testing; (3) WebArena [93] overestimates performance of agents by 5.2% due to various issues in its string matching. To demonstrate ABC's practical value, we applied it to improve CVE-Bench, a complex, representative cybersecurity benchmark [96]. ABC reduced performance overestimation in CVE-Bench by 33% in absolute terms, as confirmed by cybersecurity experts.

We summarize our contributions as follows:

1. We identified two significant threats in the evaluation rigor of agentic benchmarks: outcome validity and task validity.
2. We developed an actionable checklist, ABC, to critically assess existing agentic benchmarks and to establish best practices for future development.
3. We applied ABC to assess ten widely used agentic benchmarks and identified new evaluation issues that cause estimation errors of agents' performance by up to 100% in relative terms.
4. We provided a case study of using ABC to improve an agentic benchmark during development.

**Assessing AI Benchmarks.** Benchmarks are fundamental in AI research and practice, serving as key tools for measuring progress and identifying potential risks [21, 69]. However, maintaining benchmark quality remains a persistent challenge. To address this, prior studies have assessed various dimensions of AI benchmarks, including label quality and quantity [17, 18], standardized evaluation protocols [41], construct validity [20, 64], data contamination [91], reproducibility [75], and practical usage [24]. Even high-profile benchmarks, such as ImageNet [16], have faced issues related to data bias and label noise [73]. With the advancement of large language models (LLMs), recent work has proposed best practices for developing general or code-oriented benchmarks [10, 65]. Although these existing studies provide important insights to our analysis, they primarily focused on multiple-choice or generative tasks that do not require multistep reasoning, which present fewer ambiguities and complexities than complex agentic benchmarks.

**Benchmarking of AI Agents.** Prior work has proposed agentic benchmarks across various domains, including coding [30, 37, 48, 59], interacting with environments for a predefined target [83, 86, 93], solving math problems [22, 39], and others [11, 47, 74, 89]. These tasks typically emulate real-world challenge resolution, involving non-categorical outputs and multistep execution. Evaluating AI agents in these tasks introduces a more complex design and implementation than traditional benchmarks, including handling dynamic interactions between an agent and the environment and grading unstructured responses, which increases the difficulty in ensuring rigorous evaluation.

**Issues in Evaluating AI Agents.** Existing analyses have identified evaluation issues in individual agentic benchmarks [32, 34, 35, 62, 87]. In terms of the outcome validity, Kydlíček and Gandenberger [34] found that implicit assumptions on the answer formats lead to performance underestimation by 5.3%. Yu et al. [87] found that agents can pass evaluations without generating correct patches for 7.7% of tasks in the SWE-bench-Lite and 5.2% of tasks in the SWE-bench-Verified. In addition, prior analysis found that the annotation noise in BIRD significantly affects the accuracy of performance evaluation [62, 80]. In terms of task validity, the rate limit of the websites implemented in WebArena prevented agents from resolving challenges [32]. Furthermore, Lange et al. [35] identified flaws in the grading of KernelBench that allow agents to bypass correctness checks. However, none of them develops an actionable and systematic guideline to assess agentic benchmarks.

In this section, we present an overview of our work. We first introduce a taxonomy of validity issues in agentic benchmarks and then describe the process of our benchmark collection, checklist development, and benchmark assessment. Finally, we release our code [1] and build a website [2] for continuous development and future updates.

**Taxonomy.** We first identify and classify the primary challenges in rigorous agentic evaluation. In Figure 1, we decompose the operational and conceptual process of agentic evaluation. An agentic benchmark challenges an AI agent to finish a task in a specific environment with a given set of tools. After several rounds of (inter-)actions, the AI agent presents a task outcome, which indicates whether the task completion state. To automatically determine whether the task is successful, the agentic benchmark develops customized methods based on the task requirements, such as string matching [86, 92] and testing [30, 59].

Conceptually, an agentic evaluation is rigorous if and only if (1) the target capability is equivalent to task success (i.e., task validity), and (2) the task success is equivalent to a positive evaluation result (i.e., outcome validity). However, agentic benchmark presents two unique challenges that makes these two validity conditions difficult to hold:

1. **Complex task setup**: In addition to task descriptions as inputs, agentic benchmarks set up an environment for agents to operate in and provide tools for agents to use.
2. **Unstructured task outcome**: Agentic benchmarks expect unstructured data as task outcomes, such as textual responses, code, and file edits. Verifying the correctness of such outcomes are non-trivial and requires specially designed methods.

First, improper task setup can lead to the violation of task validity. For instance, τ-bench includes intentionally unattainable tasks (e.g., making changes to a non-refundable ticket), which agents are supposed to recognize and reject [86]. Yet, a trivial agent that simply returns nothing is considered a successful completion even though it cannot look up information or interpret ticket rules. Second, failure to rigorously grade unstructured task outcome can break outcome validity. For example, SWE-bench-Verified judges agent-generated patches by handwritten unit tests [14]. Since such tests can be incomplete or not perfectly sound [87, 94], a patch that passes them may still be wrong. Task validity breaks down for a different reason, often reflected as shortcuts or impossible tasks.

To help researchers identify and mitigate such problems in specific agentic benchmarks, we aim to translate the two validity criteria into an actionable checklist. When a criterion cannot be fully satisfied, the checklist also offers guidance on how to interpret and report the resulting scores.

**Benchmark Collection.** To develop the checklist, we collected a set of popular agentic benchmarks as the corpus for our study. To emphasize common and representative issues, we focused on popular agentic benchmarks used by top AI providers, including OpenAI, Anthropic, Amazon, Meta, Google, xAI, Mistral, and DeepSeek, or those winning awards in peer-reviewed academic conferences. This narrows our focus to a set of 17 agentic benchmarks (Table 3). We defer the details of our benchmark collection to Appendix B.

**Checklist Development.** We first reviewed the collected benchmarks and surveyed AI agent evaluation frameworks [1, 44, 45, 50] together with documented issues in agentic benchmarks [32, 34, 35, 62, 87]. We then examined best practices for evaluating unstructured task outcomes in related domains, such as software testing. Integrating these insights with our own experience in benchmark development, we curated the Agentic Benchmark Checklist (ABC), which has three parts: task validity, outcome validity, and benchmark reporting. We provide the source of each checklist item in Appendix C.

**Benchmark Assessment.** We applied ABC to thoroughly assess ten selected benchmarks (Table 1). We selected these benchmarks from the open-source set in Table 3, prioritizing their popularity and ensuring all types of agent capabilities are covered. We assigned 1 point to each satisfied item and 0 otherwise. For each issue identified by the checklist, we designed experiments to validate the issue and obtained quantitative results (Section 5). We defer detailed assessment results to Appendix D and case studies to Appendix E.

In this section, we formulate our assessment framework into an actionable checklist (ABC). We present the checklist items in terms of task validity, outcome validity, and benchmark reporting.

We propose guidelines for ensuring task validity. These checks uncover design or implementation flaws that can create shortcuts, which causes false positive evaluation results, or lead to impossible tasks, which causes false negative evaluation results.

**Task Validity**

**Tool**
T.1. Versions of all tools (e.g., Python) are clearly specified.
T.2. Required API tools are consistently accessible during evaluation.
T.3. Evaluation process terminates or handles errors appropriately if an API becomes inaccessible.

**Env.**
T.4. Residual data or state are fully cleared between runs.
T.5. Agent is completely isolated from any ground truth information.
T.6. Setup does not change over time (e.g., no live website).

**Implementation**
T.7. Annotated ground truth is verified for correctness.
T.8. Each task is verified to be solvable.
T.9. Benchmark includes an Oracle solver that can automatically solve all challenges.
T.10. Implementation is free of vulnerabilities that could be exploited to pass evaluations without completing tasks.

Figure 2: Checks in ABC to assess the task validity of an agentic benchmark.

**Tool.** External tools and functions can significantly extend the capabilities of AI agents. Existing benchmarks provide two types of tools: self-hosted tools (e.g., Python, command-line tools) and API-based tools (e.g., web services). For self-hosted tools, it is essential to explicitly specify the correct tool or package versions in the prompt (T.1). In terms of API-based tools, ensuring service availability and managing rate limits is crucial (T.2). If API interruptions occur, we recommend detecting them and terminating the evaluation to keep benchmark users informed (T.3).

**Environment.** Agentic benchmarks often need a sandbox environment to simulate real-world scenarios. Implementing and maintaining such environments can be challenging, especially with complex task formulations. First, to ensure the independence of tasks, we need to ensure that any legacy data and states are fully cleaned up before starting a new task (T.4). For example, KernelBench failed to remove ground truth answers from GPU memory, allowing agents to obtain the correct result through out-of-bounds memory access [35]. Furthermore, to avoid cheating by peeking at ground truth, it is important to fully isolate agents from the ground truth results (T.5). Finally, the environment setup should be fully reproducible and frozen at the time of benchmark release (T.6). Relying on dynamic resources, such as continually updated external websites, is not recommended.

**Implementation.** Even with a robust setup of tools and environments, subtle implementation vulnerabilities can also result in shortcuts or impossible tasks. Therefore, we recommend verifying the correctness of ground truth annotation and the task setup (T.7-8). Providing an automatic oracle solver can help demonstrate the correctness of the task configuration (T.9). Additionally, as demonstrated in τ-bench [86], inspecting outliers in pilot experiments is crucial for identifying implementation bugs (T.10). For example, if agents consistently fail on easy tasks, this may indicate that tasks are impossible, whereas if agents only succeed on difficult tasks, it may indicate shortcuts.

In this part of the assessment, we propose practical checks for ensuring the outcome validity of an agentic benchmark (Figure 3). We design these checks based on different types of outcomes and different evaluation methods.

**Information Acquisition.** To evaluate the capability of AI agents to search, retrieve, integrate, and summarize information, agentic benchmarks formulate tasks as information acquisition queries [25, 86, 89, 93]. Depending on task requirements, benchmarks use various schemes for evaluating agents' textual responses, including whole string matching [89], substring matching [86, 93], and LLM-as-a-judge [25, 93].

1. **Whole String Matching** directly compares the agent's response and the ground truth. When annotating ground truth, it is important to consider semantically equivalent expressions (O.a.1) or redundant words (O.a.2). [Footnote 3]
2. **Substring Matching** evaluates whether the agent's response contains the ground truth. In addition to equivalent expressions, it should handle negation modifiers (O.b.1), such as "not" and "negative." We also recommend formulating tasks carefully to prevent success by listing all possible answers (O.b.2) or guessing (O.b.3).
3. **LLM-as-a-Judge** uses LLMs to emulate human annotators [9, 38, 88, 90, 97]. Previous studies have shown that the accuracy of LLM annotations varies across domains [98]. We recommend conducting pilot experiments to assess the accuracy and self-consistency of LLM judges (O.c.1).

[Footnote 3] In practice, users often specify format requirements for AI agents, which narrows the scope of alternative expressions of the ground truth. Failing to follow the format requirements is considered as a true failure.

**Information Acquisition**

Whole string matching or substring matching:
O.a.1. Considers expressions semantically equivalent to ground truth.
O.a.2. Handles redundant words used by agents.

Substring matching:
O.b.1. Handles negation modifiers used by agents.
O.b.2. Is robust against systematically listing all possible answers.
O.b.3. Ground truth is sufficiently complex to prevent guessing.

LLM-as-a-Judge:
O.c.1. Demonstrates documented or experimental evidence of the judge's accuracy, self-consistency, and agreement with human.
O.c.2. Is designed to resist adversarial inputs and reward hacking.

**Code Generation**

Unit testing or end-to-end testing:
O.d.1. Verifies test cases for correctness and quality (e.g., by human).
O.d.2. Measures quality of test cases using objective metrics (e.g., code coverage, cyclomatic complexity control).

Fuzz testing:
O.e.1. Addresses potential edge cases.
O.e.2. Ensures comprehensive coverage of all relevant input variations (e.g., data types, memory layouts, value ranges).
O.e.3. Generates inputs that the code under testing is sensitive to.

End-to-end testing:
O.f.1. Exercises all relevant parts of the code being tested.
O.f.2. Prevents non-deterministic ("flaky") test results.

**State Matching**

State matching:
O.g.1. Ground truth includes all states achievable after success.
O.g.2. Checks relevant and irrelevant states for the challenge.
O.g.3. Ground truth is complex to prevent trivial state modifications.

**Multistep Reasoning**

Answer matching:
O.h.1. Specifies required answer formats in challenge descriptions.
O.h.2. Minimizes the possibility of success by random guessing.

Quality measure:
O.I.1. Designs quality metrics that prevent exploitation (e.g., achieving high scores by reward hacking).

Figure 3: Checks in ABC to assess the outcome validity of an agentic benchmark. We group items by the types of the outcome and the methods of evaluation.

**Code Generation.** Existing agentic benchmarks evaluate the capability of AI agents to write code [30, 37, 48, 59]. These benchmarks apply program testing techniques to evaluate the correctness of generated code, including unit testing, fuzz testing, and end-to-end testing.

1. **Unit Testing** designs test cases for individual functions or classes [67]. However, poorly constructed unit tests can lead to both false positives and false negatives [70, 87]. Therefore, we recommend manually verifying the correctness and quality of test cases (O.d.1) [14], and providing quality guarantees using objective metrics (O.d.2) such as coverage [94] and cyclomatic complexity [76].
2. **Fuzz Testing** evaluates generated code by running it against a ground-truth implementation on automatically generated inputs [95]. We should tailor the input generator to the target program, covering different data values, types, memory layouts, and edge cases (O.e.1-2). Moreover, the inputs must affect the output (O.e.3)—e.g., random negatives reveal nothing about relu(x) [35].
3. **End-to-end (E2E) Testing** simulates complete user workflows, providing comprehensive testing of system functionality [36, 72]. In addition to ensuring the general quality of test cases, it should also cover all possible branches of user workflows (O.f.1). Because of their complexity, E2E tests require extra safeguards to eliminate non-determinism and ensure repeatable results (O.f.2) [61].

**State Modification.** Agentic benchmarks challenge agents to manipulate environment states, such as booking flight tickets [86] and editing websites [83]. In these tasks, we often compare the final state achieved by agents with a ground-truth state.

We identify three key checks for rigorous state matching. First, ground truth states should include all possible outcomes achievable through successful task resolution (O.g.1). For example, when we challenge agents to attack a website, we should evaluate all possible attack outcomes [96]. Second, the state space should contain both relevant and irrelevant states (O.g.2), such as including both changed and unchanged files, to help detect if agents affect the environment outside the target scope. Finally, the state space should be complex enough (O.g.3)—for instance, involving multiple variables or dependencies—so that random or trivial changes are unlikely to result in a correct outcome.

**Multistep Reasoning.** Agentic benchmarks evaluates multistep reasoning capabilities of AI agents [11, 22, 39, 47]. These benchmarks typically require AI agents to make observations, conduct analysis, and generate results. We summarize two common approaches for evaluating these tasks:

1. **Answer Matching** parses the agents' output and then compares the parsed result with ground truth. We find that parsers in existing benchmarks may make implicit assumption about the agent's output (O.h.1). For example, the MATH dataset assumes the answer of the agent starts with "Answer:" [39]. Therefore, it is necessary to explicitly specify any assumptions, such as format requirements. Additionally, to ensure that a single final answer reflects a genuine reasoning process, we recommend designing tasks in a way that avoid success by guessing (O.h.2) [22].
2. **Quality Measure** evaluates agent using customized metrics against a baseline when ground truth is impossible to achieve (e.g., ground-truth predictions in an ML engineering task [11]). The choice of metrics can be highly subjective and often depends on the nature of the tasks. To avoid metric hacking [26]—achieving high metrics without resolving tasks, we recommend ensuring that the selected metrics are strongly correlated with the reasoning process (O.i.1).

**Benchmark Reporting**

**Transparency & Validity**
R.1. Is fully or at least partially open-sourced.
R.2. Offers an open-source evaluation harness for users.
R.3. Includes measures to prevent data contamination at the time of benchmark release, such as a private, held-out test set.
R.4. Includes measures or plans to consistently update challenges over time to avoid overfitting.
R.5. Clearly states the relationship between the agent capabilities it aims to evaluate and the constructs or outcomes it measures.
R.6. Clearly states the evaluation subjective of the benchmark (e.g., a model or an agent framework).

**Flaw Mitigation**
R.7. Describes steps taken to prevent, identify, and correct flaws.
R.8. Includes qualitative discussions of the potential impact of unavoidable flaws.
R.9. Includes quantitative analysis to assess the impact of unavoidable flaws (e.g., noise of ground truth).

**Interpretation**
R.10. Reports metrics about statistical significance, such as confidence intervals.
R.11. Provides guidance on interpreting results with eval flaws.
R.12. Reports results of non-AI baselines (e.g., human experts).
R.13. Reports results of trivial agents (e.g., one that does nothing).

Figure 4: Checks in ABC to assess the benchmark reporting.

Completely avoiding evaluation issues in agentic benchmarks can be challenging, and is sometimes not feasible, especially when using LLM-as-a-Judge or testing-based techniques. In such cases, it is particularly important for benchmark developers to be transparent and clearly communicate the impact of these limitations (Figure 4).

We assess the reporting quality of an agentic benchmark based on the following aspects. In Appendix F, we use BIRD as an example to demonstrate the high-quality benchmark reporting.

1. **Transparency and Validity.** We encourage open-sourcing both the datasets and evaluation harness (R.1-2) while including measures to prevent data contamination (R.3-4). We also recommend clearly specifying the capabilities to evaluate and articulating construct validity [65] (R.5-6).
2. **Mitigation.** When validity limitations are unavoidable, it is important to document mitigation efforts (R.7) and provide both qualitative and quantitative evidence regarding the impact of those limitations (R.8-9). In resource-constraint scenarios, we recommend using sampling and uncertainty quantification techniques (e.g., Cramer's Theorem [17]) to estimate the impact of unavoidable flaws, such as the noise of ground truth.
3. **Result Interpretation.** We recommend reporting benchmark results rigorously, including measures of statistical significance (R.10), clear interpretation guidelines (R.11), and appropriate baseline comparisons (R.12-13).

In this section, we present the results of applying ABC on existing agentic benchmarks (Table 1). We first show the assessment scores (Section 5.1) and then summarize newly identified issues with quantitative results (Section 5.2). Finally, with a case study, we show how developers can apply ABC to improve their benchmarks (Section 5.3).

We selected ten open-source agentic benchmarks from Table 3 to cover all capability categories and evaluation methods. For each part of ABC, we calculated the average scores of applicable items. We present the final assessment scores in Figure 5. We summarize our findings as follows.

- **Task validity**: more than half of the benchmarks exhibit implementation flaws, especially those that provide tools to agents.
- **Outcome validity**: more than half of the benchmarks fail to address inherent limitations of the evaluation methods.
- **Benchmark Reporting**: 80% of the benchmarks fail to acknowledge weaknesses in their design or implementation, and none satisfies every reporting criterion.

We conducted an in-depth analysis of specific issues present in each agentic benchmark. In this section, we focus on discussing 4 benchmarks with newly discovered issues. We defer a detailed description of all identified issues in Appendix D and experiment designs to E.

1. τ-bench relies on trivial states or substrings as ground truth, violating checks O.b.3 and O.g.3 and overestimating performance by 38%.
2. τ-bench also allows agents to list every possible answer, violating check O.b.2 and overestimating performance by 40%.
3. WebArena not only violates check O.b.2 but also uses an LLM-as-a-Judge without validating its accuracy or consistency (check O.c.1), leading to a 1.4–5.2% performance overestimate.
4. SWE-Lancer fails to fully isolate agents from the ground truth (check T.5), allowing agents to score 100% without solving tasks.
5. KernelBench omits comprehensive fuzzing for edge cases and memory layouts—violating checks O.e.1 and O.e.2 and overestimating kernel-correctness performance by approximately 31%.
6. In OSWorld, the task website changes have broken the HTML selectors used for evaluation, leading to a 28% performance underestimation in the chrome task section.

**τ-bench.** First, τ-bench contains intentionally unsolvable tasks—38% of the airline subset and 6% of the retail subset. Because success is defined as leaving the environment unchanged, an agent that does nothing still passes evaluation of these tasks. Second, 2% of airline tasks and 3.6% of retail tasks treat verbatim database text as ground truth and grade by substring matching. Therefore, an agent that dumps the entire database passes evaluations of these tasks.

**WebArena.** We identify two issues in WebArena, a benchmark for evaluating agents in interacting with websites. First, its substring-matching ignores extraneous content, so an agent that includes irrelevant information is considered successful. Second, for tasks labeled "N/A," the LLM judge that accepts an empty reply as correct, enabling a trivial agent to pass.

**SWE-Lancer** evaluates an agent's ability to implement features by allowing it to execute Python scripts that interact directly with the file system. This design grants agents unrestricted read-write access, including to the benchmark's own test files. Although these tests reside in a password-protected ZIP archive, the archive's directory structure can be listed—and its contents overwritten—without knowing the password. Therefore, an agent can locate the tests and replace them with a trivial assertion (e.g., assert 1 == 1), achieving a perfect score without solving any of the intended tasks.

**KernelBench** evaluates generated CUDA kernels with randomly generated tensors, while its fuzzer varies only the tensor values, leaving shapes and memory layouts unchanged. As a result, kernels that would fail under alternative configurations can still pass. Re-examining the kernels reported by Lange et al. [35], we find that the correctness rate of kernels is overestimated by 31%.

**OSWorld.** We find that in the chrome section of OSWorld, 13/46 problems are broken due to changes made to the layout, URLs, and functionality of websites since the initial creation of the benchmark. This is because many evaluations rely on HTML element selectors, such as classes and XPaths. These websites might change their layouts after the benchmark released. In our experiments, we found that this issue leads to an underestimation of the performance of UI-TAR, the state-of-the-art open-source agent for OSWorld, by 28% in absolute terms.

In this section, we use a benchmark with representatively complex design and implementation to demonstrate how ABC can help improve an agentic benchmark. CVE-Bench is a benchmark for evaluating AI agents' ability to exploit real-world web vulnerabilities under one- or zero-day scenarios [96]. It evaluates agents by checking whether one of the pre-specified attack targets (e.g., denial of service) is accomplished. Using ABC, we resolved flaws in outcome and task validity.

**Naive State Matching for Time-based Injections.** Time-based SQL injection infers the database content by measuring the latency difference across multiple requests [23]. For example, an attacker can execute a SLEEP command within a IF clause and measure the latency to determine whether the IF condition is satisfied. CVE-bench measured such attacks by examining whether a SLEEP clause appears in the database log. However, containing a SLEEP clause in the log does not necessarily indicate executions of SLEEP, violating check O.g.1. Consequently, agents can pass the evaluation by adding SLEEP anywhere in the query, leading to performance overestimation by 32.5%.

**Ungated Outbound Server.** Inducing the web application to send requests to a banned outbound server is a critical cybersecurity attack [29]. CVE-bench measured such attacks by checking whether an outbound server has been accessed. To answer check T.9, we conducted various rounds of mock execution and identified that agents consistently passed the evaluation for this attack, which likely indicates a bug in the implementation. Indeed, we find that agents can access the outbound server when connecting from the same docker network, creating a shortcut. After denying external requests on the outbound server, the success rates of agents decreased by 10% (Figure 6b).

We formulate the first actionable agentic benchmarks checklists (ABC) focusing on the outcome validity, task validity, and reporting of results. Via ABC, we proposed a set of the best practices for building rigorous agentic benchmarks. Based on ABC, we assessed ten widely used benchmarks and identified significant evaluation issues that cases up to 100% errors (in relative terms) when estimating agents' performance. Finally, we use CVE-Bench [96] as an example to demonstrate using ABC to improve the evaluation rigor during benchmark construction.

We are grateful to the CloudLab [19] for providing computing resources for experiments. This research was supported in part by Open Philanthropy project.

**Limitation.** As the first study to systematically investigate the issue of evaluation rigor in agentic benchmarks, our work is not without limitations. First, our analysis covered only 17 agentic benchmarks that are used by top AI providers between January 2024 and March 2025. We did not analyze benchmarks outside this time frame. Therefore, our findings may not necessarily include all relevant evaluation practices. Consequently, it is possible that we have not presented an exhaustive checklist for ensuring evaluation rigor. Second, our taxonomy and analysis are grounded in the current understanding of the reasoning capabilities of AI agents. It is conceivable that future developments in AI may introduce advanced capabilities, which could, in turn, lead to more evaluation challenges that are not addressed in this study. Finally, our findings only reflect the state the analyzed benchmark at the time of writing. Future revisions of these benchmarks may yield different results. Therefore, our conclusions may not fully apply to subsequent versions.

**Broader Impact.** Although our study rigorously highlights shortcomings in existing benchmarks, our aim is not to criticize but to raise awareness and foster the development of a stronger community with higher standards and improved quality in agentic benchmarks. We anticipate that our findings will encourage more critical evaluation of agentic benchmark results and a reassessment of AI agent leaderboards. We believe these contributions will lead to a deeper and more accurate understanding of AI agent capabilities, resulting in positive societal impact.

We first surveyed the model release blog posts, technical reports, and paper of top AI provider, including OpenAI, Anthropic, Google, Meta, xAI, Mistral, DeepSeek, and Amazon. Since AI agents and their capabilities are evolving with a fast pace, we focused on state-of-the-art models released between January 2024 and March 2025. Furthermore, we also considered benchmarks that won awards on peer-reviewed academic venues. As shown in Table 2, we identified 78 benchmarks.

Next, we classified these benchmarks into agentic benchmarks and non-agentic benchmarks. An agentic benchmark mush involve tasks that require multistep reasoning or command execution, which excludes fact-seeking questions, such as simpleQA [77], straightforward question-answer (QA) datasets, such as MMMLU [27], and straightforward programming tasks, such as MBPP [8] and HumanEval [13]. As shown in Table 2, we collected 25 agentic benchmarks.

Finally, we categorize these agentic benchmarks based on their evaluated capabilities, evaluation methods, and open-source availability (Table 3). We selected ten benchmarks for in-depth assessment, ensuring open-source availability and a comprehensive coverage over the evaluated capabilities and evaluation methods.

In Table 4, we show the detail construction process of ABC by listing the sources of each check proposed in ABC. We synthesized the insights from the following aspects

1. Our experience of developing agentic benchmarks.
2. Best practices in existing agentic benchmarks (Table 3).
3. Lessons learned from issues of existing agentic benchmarks.
4. Domain-specific suggestions when we apply well-established techniques as evaluation methods.

In this section we provide detailed assessment reports for all ten benchmarks. Each report's caption specifies the corresponding paper and codebase evaluated.

We present case study of specific issues we identified. For each study, we use an Intel E5-2630 CPU with 128 GB RAM and optionally 1 NVIDIA H100 80GB for GPU-required experiments. We release our code at https://github.com/uiuc-kang-lab/agentic-benchmarks.

**Benchmark Overview.** SWE-bench is a benchmark for evaluating the ability of AI agents to resolve real-world GitHub issues. Given the issue description and a summary of the codebase, agents are tasked with generating a patch that resolves the issue. Each generated patch is evaluated via existing unit tests in the GitHub repository.

**Identified Issue.** SWE-bench uses manually written unit tests to evaluate the correctness of a generated code patch. As illustrated in prior work, UTBoost [87], unit tests can lead to many false positives, due to the insufficiency of test cases.

**Example.** The Python package seaborn has an issue in handling missing values in the inputs x and y when computing polynomial fits using PolyFit(). Unfortunately, the unit test case for PolyFit() only considers the scenarios when both x and y have missing values:

```python
def test_missing_data(self, df):
groupby = GroupBy(["group"])
df.iloc[5:10] = np.nan
res1 = PolyFit()(df[["x", "y"]], groupby, "x", {})
res2 = PolyFit()(df[["x", "y"]].dropna(), groupby, "x", {})
assert_frame_equal(res1, res2)
```

This insufficient test case for PolyFit() leads to the following incorrect patch for PolyFit() being evaluated as correct. This patch is generated by IBM SWE-1.0.

```python
def _fit_predict(self, data):
y = data["y"].dropna()
x = data["x"].dropna()
if x.shape[0] != y.shape[0]:
raise ValueError("x and y must have the same number of non-missing values")
if x.nunique() <= self.order:
# TODO warn?
xx = yy = []
```

**Qualitative Results.** As reported in prior work [87], agents can pass evaluations without addressing the GitHub issues for 5.3% and 7.7% of tasks in the Verified and Lite partitions, respectively. These tasks lead to 40.9% and 24.4% changes in the leaderboard for the Verified and Lite partitions, respectively. Furthermore, these tasks causes 2.3% and 1.6% overestimation of agent performance for the Verified and Lite partitions, respectively.

**Benchmark Overview.** τ-bench is for evaluation AI agents capability to interact with human users and follow domain-specific rules [86]. Given a domain-specific policy, the AI agent is tasked to interact with human users and answer user queries.

**Identified Issue.** τ-bench evaluates the agents' actions based on whether the database state is correct and optionally whether the agents' responses contain required text. Therefore, on tasks that do not change the database state and do not have required texts, agents can get positive evaluation results by doing nothing. On tasks that do not change the database state and has a trivial required text, such as "4", agents can get positive evaluation results by returning random responses or all the data.

**Example.** A task in τ-bench requires agent to process a flight cancellation and refund request. An AI agent is supposed to check the detail of the booked flight ticket for the user in the database and deny the user request if the ticket is non-refundable. This task has no required output. Therefore, as long as the data state does not change, the agent will obtain a positive evaluation result. In this case, an agent that does nothing can also have a positive evaluation result.

**Qualitative Results.** A do-nothing agent that returns immediately can achieve a 38% and 6.0% pass^k or pass@k for any k for Airline and Retail partitions, respectively. A spamming agent that outputs all the data can achieve a 40% and 9.6% pass^k or pass@k for any k for Airline and Retail partitions, respectively.

**Benchmark Overview.** BIRD is for evaluating the capability of agents to write SQL queries [37]. Given a query description in natural language, the agent needs to translate it into a SQL query.

**Identified Issue.** BIRD evaluates agent by comparing the execution results of the ground truth query with the generated query. However, due to the ambiguity of the query description, there can be multiple correct queries for the same natural language description.

**Example.** A task in BIRD asks the agent to write a SQL query that can answer the question: "What are the name, independence year, and surface area of the country with the smallest population?" There can be two correct SQL queries:

```sql
-- Query 1
SELECT Name, SurfaceArea, IndepYear FROM country
WHERE Population = (SELECT min(Population) FROM country)
-- Query 2
SELECT Name, SurfaceArea, IndepYear FROM country
ORDER BY Population LIMIT 1
```

Query 1 outputs all the country with the smallest population, while Query 2 outputs one of the country with the smallest population. Although the output of two queries are different, they both answer the question.

**Benchmark Overview.** SWE-Lancer is for evaluating the capability of AI agents to independently implement features and fix bugs. [48] Given a task description, agent needs to use Python scripts to interact with the file system and modify codebase.

**Identified Issue.** SWE-Lancer uses end-to-end testing to evaluate the correctness of agents' implementation. Although the test cases are stored in a password-protected .zip file, reading the directory structure and updating files within the .zip file do not require a password. Therefore, an agent can easily locate the test cases and replace them with a naive one, such as "assert 1==1".

**Qualitative Results.** An agent that overwrites the test cases in the .zip file can achieve a 100% resolve rate without completing the software engineering tasks.

**Benchmark Overview.** WebArena is for evaluating the capability of agents to interact with the web [93]. Given a user request, the AI agent need either retrieve the required information or fill the given data into the web form correctly.

**Identified Issue.** WebArena uses exact string matching, substring matching, and LLM-as-a-Judge to evaluate agents. Its strategy of exact string matching cannot handle alternative expressions and phrase modifiers, while the substring matching is vulnerable to exhaustive enumeration of the content on the website. Additionally, LLM-as-a-Judge can produce unreliable results.

**Example.** In WebArena, there is a user query that asks "What is the duration required to first walk from Massachusetts Institute of Technology to Harvard University, and then drive to Boston Logan International Airport?" The ground truth answer for this question is 63 minutes. However, the agent searched the web and output the final answer: "The duration required to first walk from Massachusetts Institute of Technology to Harvard University is 45 minutes, and then drive to Boston Logan International Airport is 8 minutes." The answer of agent gives the duration of 45+8=53 minutes, which is different from the ground truth answer. However, the LLM judge considers the agent's answer as correct.

**Benchmark Overview.** KernelBench is for evaluating the capability of agents to write correct and efficient GPU kernels [59]. Given the task instruction and the original PyTorch code, agents need to write PyTorch code containing an inline implementation of the kernel that is functionally correct and more efficient.

**Identified Issue 1.** KernelBench uses randomly generated inputs (i.e., fuzzing) to test the correctness of generated GPU kernels. However, we find the tested functions in a subset of tasks are not sensitive to uniform random inputs, such as mean(softmax(x)) and relu(x-2).

**Identified Issue 2.** In the evaluation implementation, KernelBench first runs the ground truth kernel and then runs the generated kernel subsequently. As reported in prior work [35], agents can potentially cheat by generating a program that extracts the execution results of the ground truth kernel.

**Identified Issue 3.** The fuzzer designed in KernelBench fails to address potential inputs with different memory layouts (e.g., non-contiguous tensors), tensor shapes, and hardware environment.

In the following code snippet, we demonstrate an incorrect kernel function due to improper use of threads, which were graded as correct in KernelBench. In Line 46, the kernel function accesses parallel execution results in s_sum with index from tid to nthread. However, when nthread > normalized_size, this will lead to out-of-bound access into uninitialized memory. Namely, a thread-safe guard is required here.

```cuda
#include ...

template <typename scalar_t>
__global__ void layernorm_forward_kernel_opt(
const scalar_t* __restrict__ input,
const scalar_t* __restrict__ weight,
const scalar_t* __restrict__ bias,
const float eps,
scalar_t* __restrict__ output,
const int normalized_size) {

// Each block processes one outer instance.
int instance_idx = blockIdx.x;

// Use 2D thread indexing to cover the normalized dimension flexibly.
int tid = threadIdx.y * blockDim.x + threadIdx.x;
int nthreads = blockDim.x * blockDim.y;

// Pointers to the start of this instance's data.
const scalar_t* __restrict__ in_ptr = input + instance_idx * normalized_size;
scalar_t* __restrict__ out_ptr = output + instance_idx * normalized_size;

using accscalar_t = at::acc_type<scalar_t, true>;

// Each thread computes a partial sum and sum of squares over a strided range.
accscalar_t local_sum = 0;
accscalar_t local_sum_sq = 0;
for (int i = tid; i < normalized_size; i += nthreads) {
// Use __ldg for read-only, coalesced global memory access
scalar_t val = __ldg(&in_ptr[i]);
accscalar_t a_val = static_cast<accscalar_t>(val);
local_sum += a_val;
local_sum_sq += a_val * a_val;
}

// Allocate shared memory for reduction: first part for partial sums, second for sum of squares.
extern __shared__ char smem[];
accscalar_t* s_sum = reinterpret_cast<accscalar_t*>(smem);
accscalar_t* s_sum_sq = s_sum + nthreads;

s_sum[tid] = local_sum;
s_sum_sq[tid] = local_sum_sq;
__syncthreads();

// Perform parallel reduction in shared memory.
for (int stride = nthreads / 2; stride > 0; stride >>= 1) {
if (tid < stride) {
s_sum[tid] += s_sum[tid + stride];
s_sum_sq[tid] += s_sum_sq[tid + stride];
}
__syncthreads();
}
...
}
```

To identify such issues in large scale, we applied o3-mini to generate additional test cases. Specifically, we sampled 3 generated kernel functions for each task in level 1 and asked o3-mini to detect any possible flaws and write test cases for each detected flaw. Then, we manually verified the correctness of o3-mini-generated test cases. Finally, we applied these test cases on all generations by Lange et al. [35]. Our results show that the correctness rate of generated kernels is overestimated by 31%.

In this section, we present a modified reporting example based on BIRD to demonstrate benchmark reporting that fulfills all the criteria outlined in Figure 4. BIRD is a benchmark for evaluating agents' capability to translate a natural language query to a SQL query.

**R.1. Is fully or at least partially open-sourced.**
Example: We released the training and validation dataset of BIRD at https://bird-bench.github.io/.

**R.2. Offers an open-source evaluation harness for users.**
Example: We released the harness to evaluation agents on BIRD at https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/bird.

**R.3. Includes measures to prevent data contamination, such as a private, held-out test set.**
Example: We keep a private held-out test set to avoid potential data contamination. Request to evaluate agents on this test set can be submitted at https://bird-bench.github.io/.

**R.4. Includes measures or plans to consistently update challenges over time to avoid overfitting.**
Example: We plan to consistently update the database and natural language queries to reflect the real-world queries and avoid overfitting. Our updates will be available at https://bird-bench.github.io/.

**R.5. Clearly states the relationship between the agent capabilities it aims to evaluate and the constructs or outcomes it measures.**
Example: BIRD evaluates agents' capabilities to serve as a database interface to translate natural language queries into executable SQL queries. To achieve that, BIRD provides agents with a natural language query, the database schema, and SQL-related domain knowledge, and challenges agents to write a SQL query that can be executed to return correct answers.

**R.6. Clearly states the evaluation subjective of the benchmark (e.g., a model or an agent framework).**
Example: BIRD is designed to evaluate the capability of ML models as well as the performance of agent frameworks.

**R.7. Describes steps taken to prevent, identify, and correct flaws.**
Example: We identify that evaluating generated SQL queries using execution results have two limitations. First, tasks requiring LIMIT queries and containing ties in the data may lead to non-deterministic execution results. Second, manually annotated ground-truth queries may contain errors. To understand and mitigate these errors, we randomly sample 500 tasks to perform an additional phase of verification. After verifying queries, we found 11.65% of ground-truth queries are incorrect. [Footnote 4]

**R.8. Includes qualitative discussions of the potential impact of unavoidable flaws.**
Example: The identified incorrect ground-truth queries and potentially more incorrect ground-truth queries in the test dataset can lead to estimation errors of the agent performance and incorrect rankings of agents.

**R.9. Includes quantitative analysis to assess the impact of unavoidable flaws (e.g., noise of ground truth).**
Example: We build our quantitative analysis based on the normality assumption. Specifically, suppose the number of data in the test set N is large enough such that the true success rate (p) of an agent follows a normal distribution with mean µ and standard deviation σ. Given the ground truth's incorrectness rate of e and the estimated agent success rate p0 (based on the imperfect ground truth), µ and σ are calculated as

µ = e + (1 − 2e)p0; σ² = µ(1 − µ) = (e + (1 − 2e)p0)(1 − e − (1 − 2e)p0)

Hence, based on the normality assumption, we can derive a two-sided confidence interval with confidence α for p as follows:

P(µ − 1.96 × σ/√N ≤ p ≤ µ + 1.96 × σ/√N) ≥ 95% (1)

Finally, based on the plug-in estimate (11.65%) for the ground truth's incorrectness rate, we calculate the confidence interval for the agents' performance in Table 15.

**R.10. Reports metrics about statistical significance, such as confidence intervals.**
Example: In additional to accuracy estimate, we also calculate confidence intervals for each model in Table 15.

**R.11. Provides guidance on interpreting results with eval flaws.**
Example: Given the potential flaws in BIRD, we do not recommend users to rely on the success rate alone for decision-making or selecting models. Instead, we suggest using the confidence interval of the success rate as a reference.

**R.12. Reports results of non-AI baselines (e.g., human experts).**
Example: We measured the performance of a SQL expert on BIRD, obtaining a success rate of 92.96%.

**R.13. Reports results of trivial agents (e.g., one that does nothing).**
Example: We performed sanity check on our evaluation harness by measuring the performance of a trivial agent that does nothing. We find that the trivial agent achieves 0% success rate, confirming the rigor of our evaluation implementation.

[Footnote 4] We used results by Arcwise [7].