Methodology

Explanation: The benchmark developers mention and define what underlying capability or characteristic of a model is supposed to be tested with the benchmark.

Justification: Defining the objective of the benchmark is necessary for clarity in its design. It also helps users determine if the benchmark aligns with their specific application needs and ensures that users and developers have a shared understanding of the concept being evaluated, facilitating consistent interpretation of results.

Points:

• 0: Tested concept, capability, or characteristic not explicitly mentioned.
• 5: Tested concept explicitly mentioned and need for definition acknowledged, but definition not provided.
• 10: Tested concept, capability, or characteristic explicitly mentioned but not defined.
• 15: Tested concept, capability, or characteristic explicitly mentioned and defined.

Explanation: The benchmark developers describe how the tested capability or characteristic translates to the task implemented in the benchmark/the task the model is tested on in the benchmark.

Justification: Clearly explaining this translation ensures that the benchmark tasks accurately reflect the intended tested capabilities and concepts, providing valid assessment results.

Points:

• 0: No description of how the tested capability or concept translates to the benchmark task.
• 5: Acknowledgement that not describing how the tested capability or concept translates to the benchmark task is an issue, but no description provided.
• 10: Description of how tested capability or concept translates to benchmark tasks provided for some but not all tasks.
• 15: Description of how tested capability or concept translates to benchmark tasks provided for all tasks.

Explanation: The developers describe why it is useful to know about the tested capability in the real world.

Justification: This description helps users understand the practical value of the benchmark, demonstrating how the tested capability impacts real-world applications and use cases.

Points:

• 0: No description of how knowing about the tested concept is helpful in the real world.
• 5: Acknowledgement that not describing how knowing about the tested concept is helpful in the real world is an issue, but no description provided.
• 10: Limited description of how knowing about the tested concept is helpful in the real world.
• 15: Full description of how knowing about the tested concept is helpful in the real world.

Explanation: A use case for an AI benchmark involves specifying a scenario in which the AI system will be evaluated. This scenario should include the cultural and geographic context and the type of interactions between humans and models, if applicable. Additionally, user personas should be defined to represent the different types of users that might interact with the AI system, if applicable.

Justification: Use cases set the context and scope of the benchmark. User personas outline an understanding of the different types of interactions the benchmark developers anticipate the tested AI system to be used in, e.g., ranging from typical users to those with specific challenges or malicious intent. This approach ensures that the design of the benchmark is closely related to real-world applications and that it's effective across diverse scenarios.

Points:

• 0: The benchmark does not include any description of use cases or user personas.
• 5: The benchmark acknowledges the importance of use cases or user personas but does not explicitly formulate or describe them.
• 10: The benchmark provides a partial description of use cases or user personas.
• 15: The benchmark fully describes use cases and user personas, specifying the cultural and geographic context, types of human-model interactions (if applicable), and representing different user types that might interact with the AI system (if applicable).
• n/a: For AI systems that do not involve direct human interaction, such as those used in industrial automation or scientific simulations, defining user personas is not relevant. However, real-world use cases should still be specified; in more theoretical benchmarks, this use case might be to advance research.

Explanation: Domain expert(s) who have a professional background or research experience in the concept to be tested are either co-authors of the paper, or were involved in the benchmark design process, i.e., the paper makes clear how they obtained the expertise and how that informed the benchmark design.

Justification: Involving domain experts ensures that the benchmark design is informed by deep, specialized knowledge, increasing its validity and relevance. This expertise helps to create tasks that accurately assess the targeted capabilities and align with real-world scenarios.

Points:

• 0: None of the authors has a background in the benchmark domain and no external experts were consulted during the design process.
• 5: The benchmark mentions domain experts but doesn't specify any further details.
• 10: The benchmark mentions that domain experts were consulted but not how their insights influenced the benchmark design.
• 15: At least one of the co-authors has a professional or academic background in the benchmark domain or the benchmark specified how external experts were consulted and how that influenced the design process.

Explanation: The developers cite domain literature in the background section and describe how insights from this literature informed the design of their benchmark or cite relevant domain literature in the benchmark design process.

Justification: By consulting domain-specific literature, benchmark developers can ensure that the tasks and evaluation criteria they include are representative and aligned with the current state of knowledge in the field. This literature often contains valuable insights into best practices, established methodologies, and proven approaches for evaluating the tested concept, which can be incorporated into the benchmark design to enhance its reliability.

Points:

• 0: The benchmark does not reference domain-specific literature.
• 5: The benchmark mentions the need to integrate domain literature but did not address it in the background section or design process.
• 10: The benchmark references domain literature in the background or related work section but does not describe how that domain literature informed the benchmark design process.
• 15: The benchmark references domain literature throughout the paper and describes how that domain literature informed the benchmark design process.

Explanation: The benchmark developers provide information about what benchmark users can and cannot take away from the benchmark score.

Justification: Clarifying the interpretation of benchmark scores prevents misuse and misinterpretation, ensuring that users draw accurate conclusions about a model's performance. This guidance helps users apply the scores appropriately within their specific contexts, and understand if the benchmark can be used to assess a model for their desired application context.

Points:

• 0: The benchmark does not comment on how the benchmark scores should or shouldn't be interpreted.
• 5: The benchmark acknowledges that the benchmark scores need to be interpreted but gives no guidance with respect to how or how not to do that.
• 10: The benchmark describes how scores should or shouldn't be interpreted or used, but not both.
• 15: The benchmark describes how scores should and shouldn't be interpreted or used.

Explanation: The developers describe how the performance metric for the defined benchmark task should be interpretable, meaningful, and standard for the task that's being evaluated. If a non-standard metric is selected, they describe their rationale for choosing a non-standard metric.

Justification: The metric should be easily understood by the reader to build their own opinion about the model's capabilities, given the benchmark score. If a non-standard metric is used, an explanation is necessary to clarify its relevance and ensure that users can accurately interpret the results.

Points:

• 0: The benchmark does not mention an evaluation metric or does not explain the choice of metric.
• 5: The benchmark acknowledges the need for an informed metric choice but does not justify their metric choice.
• 10: The benchmark provides an explanation for the choice of some but not all of their metrics.
• 15: The benchmark provides an explanation for the choice of all of their metrics.

Explanation: The benchmark provides clear floors and ceilings for the metric(s) it uses.

Justification: Establishing clear floors and ceilings for metrics ensures that users have a reference point for understanding model performance. It helps users understand if a benchmark is already saturated or if progress can be made on the task. This also allows benchmark developers to decide when a benchmark should be retired.

Points:

• 0: The benchmark doesn't provide any floors or ceilings.
• 5: Floors and ceilings are shown in the results figure but not explicitly mentioned in the text.
• 10: The benchmark provides floors and ceilings for some but not all evaluation metrics.
• 15: The benchmark provides floors and ceilings for all evaluation metrics.

Explanation: The benchmark explicitly states human performance measured on the benchmark task. It also explains how human performance was measured and if this was the performance of an average or expert group of humans. The benchmark notes if measuring human performance is not possible on the benchmark task and why.

Justification: Including human performance on a benchmark allows the reader to put the model's performance into perspective and allows for a better interpretability of the benchmarking score.

Points:

• 0: The benchmark does not state human performance and does not explain why this isn't applicable here.
• 5: The benchmark mentions human performance in passing but does not provide a measurement or explanation.
• 10: The benchmark states human performance but does not explain how it was obtained.
• 15: The benchmark states human performance and explains how it was obtained.
• n/a: The benchmark task cannot be completed by a human, and hence reporting human performance is not possible.

Explanation: The developers explicitly states the random performance measured on the benchmark.

Justification: By establishing a baseline performance level achieved through random guessing, generation, or selection, benchmark users can better understand the extent to which a model's performance stems from its inherent capabilities, rather than mere chance or the benchmark's design and especially metric choices. This random performance level serves as a reference point, allowing for a clearer assessment of the model's true effectiveness in tackling the specific task at hand.

Points:

• 0: The benchmark does not state random performance and does not explain why this isn't applicable here.
• 5: The benchmark mentions random performance but does not provide quantitative random performance on the benchmark task(s).
• 10: The benchmark states random performance for some but not all tasks.
• 15: The benchmark states random performance for all tasks.
• n/a: Measuring random performance on the benchmark task is not possible, and hence reporting random performance is not possible.

Explanation: The benchmark contains multiple input variations with the same semantic meaning/intended to elicit the same response or output by the tested model. The developers describe all relevant details such as how many different variations were tested per prompt, and how the variations were designed. For language models, this would mean including a variety of semantically (but not syntactically) equivalent prompts to combat prompt sensitivity. For computer vision models, this could mean inputting a normal, a blurred, and a cropped version of the same image, etc.), while for reinforcement learning, this could mean measuring the sensitivity of learned policies to input features.

Justification: Addressing input sensitivity in a benchmark ensures that the model's performance is consistent across semantically equivalent inputs, thus validating its robustness. Including multiple variations per input and detailing their design allows for inspection and replicable evaluation of the model's capabilities. This serves the goal of approximating intrinsic model capabilities or harms better rather than just measuring "an artifact" of your input.

Points:

• 0: The benchmark does not mention or address input sensitivity.
• 5: The benchmark mentions the issue of input sensitivity but does not describe experiments to test for it.
• 10: The benchmark includes some input variations with the same semantic meaning but lacks thorough descriptions or details on the number of variations and their design.
• 15: The benchmark contains multiple input variations with the same semantic meaning, providing detailed descriptions of all relevant details such as the number of variations per prompt and how they were designed.

Explanation: Evaluating a model against a benchmark does not require human evaluation in the process and the quality of the automated evaluation is validated (if applicable, e.g., in the case of FM-based evaluations).

Justification: Requiring human feedback to evaluate performance on a benchmark will significantly limit the scalability of the benchmark and potentially introduce biases from the human evaluators themselves. In addition, this may require an IRB for researchers, and will be more costly than an automatic evaluation, creating "major barriers to entry".

Points:

• 0: The benchmark does not provide any form of automatic evaluation and relies entirely on human evaluation.
• 5: The benchmark mentions the benefits of automatic evaluation but provides no none or limited automatic valuation.
• 10: The benchmark includes an automatic evaluation method but does not offer any validation.
• 15: The benchmark includes an automatic evaluation method and describes how it was validated as well as the results of the validation.

Explanation: The benchmark developers explain how their benchmark fills a gap compared to existing benchmarks or how it expands on existing benchmarks or their tested concepts.

Justification: Benchmark developers demonstrate the added value and relevance of the new benchmark, justifying its necessity by addressing specific gaps in existing benchmarks or by expanding on saturated benchmarks. This allows users to better understand the differences between related benchmarks and determine which one to use for their specific evaluation context.

Points:

• 0: The benchmarks do not explain any differences or relevance to existing benchmarks.
• 5: The benchmark briefly mentions existing benchmarks but provides no explanations of differences or added value.
• 10: The benchmark provides an explanation of how it fills a gap or expands on existing benchmarks for some but not all mentioned related benchmarks.
• 15: The benchmark provides an explanation of how it fills a gap or expands on existing benchmarks for all mentioned related benchmarks.

Explanation: The benchmark developers make the code available for others to evaluate their own models against the benchmark, e.g., as part of a GitHub repository.

Justification: Without access to the benchmarking procedure itself, the benchmark cannot be scrutinized by external parties to verify its reliability and adequacy, nor can it be utilized for independent evaluations and comparisons by benchmark users. In addition, if benchmark users have to write their evaluation code from scratch, it's more likely that seemingly minor implementation details affect the measured performance, hindering a fair comparison.

Points:

• 0: The evaluation code is not publicly available.
• 5: The benchmark mentions the availability of evaluation code but does not provide access to it.
• 10: The evaluation code is publicly available for some metrics described by the benchmark.
• 15: The evaluation code is publicly available for all metrics described by the benchmark.

Explanation: The benchmark developers give access to the input, output, and evaluation code, as well as all other necessary information (e.g., hyperparameters or random seed set) that they used to create the initial benchmarking results presented in the paper.

Justification: Providing access to the input, output, and code allows for transparency and reproducibility of the reported results, fostering trust into the benchmark, and contributing to overcome the current reproducibility crisis in AI/ML research.

Points:

• 0: [No description provided]
• 5: The issue of result replicability is mentioned in the benchmark paper but not addressed.
• 10: A script to reproduce some results in the benchmark paper is available.
• 15: A script to reproduce all results in the benchmark paper is available.

Explanation: The benchmark developers make the evaluation data, prompts, or the data/environment generation mechanism accessible. These do not have to be made public in order to earn full points (if contamination is a concern, for example), but some access to it for evaluation purposes, e.g., by hosting it privately on Hugging Face, needs to be possible.

Justification: Without any accessibility of the evaluation data, prompts, or environment generation mechanism, a benchmark cannot be used.

Points:

• 0: No access to evaluation data, prompts, or data/environment generation mechanism is provided.
• 5: The existence of evaluation data, prompts, or data/environment generation mechanism is mentioned, but no concrete access is provided.
• 10: Partial access to evaluation data, prompts, or data/environment generation mechanism is provided, allowing for limited evaluation.
• 15: Full access to evaluation data, prompts, or data/environment generation mechanism is provided, enabling comprehensive evaluation.

Explanation: The benchmark developers allow the benchmark evaluation of models via API access, if applicable.

Justification: This criteria is dependent on the subfield. In NLP, for example, closed-source models such as GPT-4 are oftentimes only accessible via API. Without support for API evaluation, they cannot be evaluated, which is especially problematic if such models are the state-of-the-art models in the field.

Points:

• 0: The benchmark does not support evaluation of models via API calls.
• 5: The benchmark mentions the possibility of API evaluation but does not provide concrete implementation details.
• 10: The benchmark supports evaluation of models via one API.
• 15: The benchmark supports evaluation of models via two or more APIs to different models.

Explanation: The benchmark developers implement code to support the evaluation of local models without API access.

Justification: Some model developers only host their models locally. A benchmark should support the evaluation of those to allow for a wide variety of models to be evaluated against the benchmark.

Points:

• 0: The benchmark requires users to write their own code to evaluate a local model.
• 5: The benchmark mentions that local evaluation should be possible but doesn't provide corresponding code.
• 10: The benchmark provides minimal support for local model evaluation, requiring significant user effort.
• 15: The benchmark provides full support for local model evaluation with user-friendly code.

Explanation: Benchmark developers include a globally unique identifier (GUID) or canary string in the main public evaluation code and all public evaluation prompt or data files. Alternatively, they encrypt the test data files and make the key public.

Justification: Including a GUID in relevant (sub-)repositories, public code and data repositories can support the identification of data contamination in models, either by allowing model developers to filter out the evaluation data out of large amounts of web-scraped data or by allowing benchmark developers to identify which model developers trained on their data and hence have created models that potentially perform better than they should on the benchmark. Encrypted test data files prevent non-adversarial crawling of such data; however, some advise against "using standard obfuscation or compression methods that are not key-protected, since some crawling systems include pipelines of automatic decompression or deobfuscation."

Points:

• 0: The benchmark does not include a GUID or encryption of evaluation instances.
• 5: The benchmark acknowledges the risk of contamination but does not address it.
• 10: The benchmark partially implements a GUID or encryption, but not consistently across all relevant files.
• 15: The benchmark consistently includes a GUID or encryption across all relevant files and repositories.

Explanation: The benchmark includes a task to identify if the model was trained on the benchmark data.

Justification: Public benchmarks face the challenges that their evaluation data may be web-scraped and used to train a model. A 'training_on_test_set' task can serve as a "post-hoc diagnosis of whether [...benchmark] data was used in model training."

Points:

• 0: The benchmark does not include a 'training_on_test_set' task.
• 5: The benchmark mentions the possibility that models were trained on its data but does not provide a way to check it.
• 10: The benchmark includes a partial or limited implementation of a 'training_on_test_set' task that only tests for part of the data used.
• 15: The benchmark includes a comprehensive 'training_on_test_set' task.

Explanation: Benchmark developers explicitly mention in the paper if the evaluation tasks or the expected output may contain sensitive or harmful content. If they do not anticipate sensitive/harmful content in either case, they should explicitly state that.

Justification: By explicitly stating the presence of sensitive or harmful content and issuing appropriate warnings, developers help users make informed decisions and take necessary precautions. Even if developers do not expect sensitive or harmful content, if they state that, they showcase to the benchmark users that they actually thought about the possibility. Otherwise, users couldn't be sure if the input or output doesn't contain problematic content or if the developers just forgot to include a warning.

Points:

• 0: The benchmark does not mention that they checked for the presence or absence of sensitive/harmful content in the evaluation tasks or expected output.
• 5: The benchmark mentions the general possibility of sensitive/harmful content but does not provide clear statements or warnings.
• 10: The benchmark explicitly states the presence or absence of sensitive/harmful content for either the evaluation tasks or the expected output.
• 15: The benchmark explicitly states the presence or absence of sensitive/harmful content for both the evaluation tasks and the expected output.

Explanation: Benchmark developers specify rules for benchmark users to "ensure the integrity of test results". While not all benchmark developers will be able to enforce the release requirements, they should at least specify them. One example is: "1. Publishers do not train directly on or against the benchmark dataset and retract any reported results if and when benchmark data is found to have been in training data. 2. Techniques that are likely to increase the test performance without a commensurate increase in safety factor are discouraged and may result in benchmark exclusion. [...]"

Justification: Written terms of use can help to set expectations and have a foundation to address subsequent contamination or intentional gamification attempts of the benchmark. Potential options they could mention in case of release requirement breaches are, e.g., "publishing public statements correcting the public record" or "resulting in the [model] being permanently banned from the benchmark"; however, we will not assess the enforcement ability or potential listed sanctions as part of this criteria, just the statement of release requirements.

Points:

• 0: The benchmark does not specify any release requirements for benchmark users.
• 5: The benchmark briefly mentions the issue of potential gameability or misuse by benchmark users but does not provide specific details.
• 10: The benchmark states dos and don'ts how to use the benchmark but does not specify these as requirements for use.
• 15: The benchmark provides a set of release requirements for benchmark users.

Explanation: A build status is a feature, typically implemented as a GitHub Action, that indicates whether the most recent build of the benchmark was successful. It should be implemented for the benchmark's evaluation code. It verifies that the code is running correctly after the latest commit.

Justification: A passing build status signifies that the main evaluation code was usable at the latest commit. Including a build status or equivalent can help to ensure the reliability and usability of the evaluation code. It allows benchmark users to quickly determine if the code is functioning as intended, saving time and effort in identifying potential issues.

Points:

• 0: The benchmark neither references nor implements any form of build status or equivalent.
• 5: The benchmark mentions the need for working evaluation code but does not implement it in any meaningful way.
• 10: The benchmark partially implements a build status or equivalent by providing the information in a less accessible manner.
• 15: The benchmark fully implements a build status or equivalent, clearly displaying the status of the most recent build and providing easy access to the information.

Explanation: A requirements or environment file, or equivalent is available.

Justification: Ease of use is a key criteria for benchmark adoption. Providing a requirements file allows for the quick installation of relevant packages at the correct versions, e.g., within a virtual environment, to use the evaluation code.

Points:

• 0: No requirements file or equivalent is provided.
• 5: A requirements file is mentioned but not provided.
• 10: A requirements file is provided but may be missing some dependencies or versions.
• 15: A complete and accurate requirements file specifying all necessary dependencies and versions is provided.

Explanation: The benchmark developers make a quick start guide or demo available that walks step-by-step through how the benchmark can be used.

Justification: Similar to the criteria above, ease of use is a key criteria for benchmark adoption. Providing a quick-start guide takes away any guesswork on the user side and allows them to directly set up and use the benchmark without spending extra time on setup issues.

Points:

• 0: No quick-start guide or demo code is provided.
• 5: A quick-start guide or demo code is mentioned but not provided.
• 10: A quick-start guide or demo code is provided but may be missing some steps or details.
• 15: A comprehensive, step-by-step quick-start guide or demo code is provided.

Explanation: In-line code comments state the purpose, inputs, outputs, and functionality of each code segment in all files relevant for the benchmark evaluation.

Justification: In-line documentation of code enhances clarity, understanding, and reproducibility. It facilitates collaboration, maintainability, and makes debugging easier for benchmark developers and users, should that be necessary.

Points:

• 0: No in-line code comments are provided.
• 5: In-line code comments are sparse and do not adequately explain the purpose, inputs, outputs, or functionality of the code.
• 10: Informative in-line code comments are present for most of the code but may be lacking in detail or clarity for some code segments.
• 15: Comprehensive and informative in-line code comments are provided for all relevant code segments, clearly explaining their purpose, inputs, outputs, and functionality.

Explanation: A full documentation of the repository and code it entails is publicly available. This includes, for example, an overview of the folder structure, the files in the repo, an explanation of functions in the repo.

Justification: Detailed documentation of code enhances clarity, understanding, and reproducibility. It facilitates collaboration, maintainability, and makes debugging easier for benchmark developers and users, should that be necessary.

Points:

• 0: No code documentation is provided.
• 5: Code documentation is mentioned but not provided.
• 10: Code documentation is minimal or incomplete, lacking important details about the repository structure and functions.
• 15: Comprehensive code documentation is provided, including a clear overview of the folder structure, files in the repo, and detailed explanations of all relevant functions.

Explanation: The benchmark developers define the tasks or task categories a model is tested on and describe the rationale for choosing the tasks or task categories. The rationale should explain how these tasks are relevant to the benchmark's objectives, what they aim to measure, and why they are important for evaluating the concept or capability to be tested.

Justification: Documenting test tasks is essential for transparency and for allowing public scrutiny of the benchmark. The rationale provides insight into the selection process, demonstrating that the tasks are not arbitrary but are carefully chosen to reflect real-world applications and user needs. Both help users decide if the benchmark is adequate for their evaluation contexts.

Points:

• 0: No documentation of test task categories or rationale is provided.
• 5: Test task categories are mentioned but they are neither defined in detail and a rationale for their selection is missing or inadequate.
• 10: Test task categories are defined, but the rationale for their selection is not provided.
• 15: Test task categories are clearly defined, and a comprehensive rationale is provided, explaining their relevance to the benchmark's objectives, what they measure, and their importance for evaluating the targeted concept or capability.

Explanation: If the benchmark measures properties that vary across cultural contexts (e.g., politeness), then normative assumptions are explicitly stated. The benchmark developers clearly define the cultural context and values that the benchmark adheres to, explaining how the measured properties are conceptualized and operationalized within the benchmark.

Justification: By explicitly stating normative assumptions, the authors provide transparency about the cultural framework and values that guide the benchmark's design and evaluation criteria, which can subsequently ensure cultural sensitivity and mitigate potential biases. It also facilitates informed decision-making for users of benchmarks, specifically for culture-dependent use cases they're interested in, such as measuring toxicity or bias, for example.

Points:

• 0: No documentation of normative assumptions is provided, even though the benchmark measures culturally-dependent properties.
• 5: The potential influence and importance of cultural context on the benchmark is acknowledged but normative assumptions aren't stated.
• 10: Normative assumptions are stated, but the explanation of how they are conceptualized and operationalized within the benchmark is incomplete or lacks clarity.
• 15: Normative assumptions are explicitly and clearly stated, defining the cultural context and values that the benchmark adheres to, and explaining how the measured properties are conceptualized and operationalized within the benchmark.

Explanation: Benchmark developers outline the limitations of the benchmark, including but not limited to the tasks, contexts, and scenarios that are not covered by the evaluation are acknowledged. It's stated which use cases are out-of-scope.

Justification: Documenting a benchmark's limitations is necessary for users to assess its suitability for their specific evaluation needs. By understanding what the benchmark does not cover, users can make informed decisions about whether the benchmark aligns with their goals and whether additional evaluations (either in the form of other benchmarks or private evaluations) may be required to complement the benchmark's results.

Points:

• 0: No documentation of the benchmark's limitations is provided.
• 5: Limitations of AI evaluations more broadly are briefly mentioned but without any detail and not applied to the specific benchmark.
• 10: Either limitations regarding the applicability and use of the benchmark or limitations of the benchmark design are discussed, but not both.
• 15: Both limitations regarding the applicability and use of the benchmark and limitations of the benchmark design are comprehensively discussed.

Explanation: Benchmark developers give a detailed account of the design process, including the specific decisions made at each life-cycle stage, the rationale behind them, and any trade-offs or compromises (e.g., balancing complexity vs. practicality) considered.

Justification: Documenting the benchmark design process is essential for transparency, as it allows users to understand how the benchmark was created and what factors influenced its development. It allows users to assess the thoroughness and rigor of the benchmark's construction. This information further enables users to critically evaluate whether the benchmark is suitable for their specific use case.

Points:

• 0: No documentation of the benchmark construction process is provided.
• 5: The benchmark construction process is briefly mentioned but lacks sufficient detail about the decisions made, rationale, and trade-offs considered.
• 10: The benchmark construction process is documented, including some decisions made and their rationale, but the description lacks depth or fails to address important aspects such as trade-offs or compromises.
• 15: The benchmark construction process is comprehensively documented, providing a detailed account of the specific decisions made at each stage, the rationale behind them, and any trade-offs or compromises considered.

Explanation: A detailed description of the data collection process, including the sources of the data, the criteria for selection, and any preprocessing steps is provided. For benchmarks involving prompts, a description of how the prompts were created, how they were validated, and why they were used to elicit the desired responses was included. For benchmarks involving dynamic environments or test data, the design specifics and templates for the dynamic process were explicitly described.

Justification: Documenting the data collection or prompt design process is important for ensuring the transparency, reproducibility, and validity of the benchmark. By providing a clear and detailed account of how the data was collected or how the prompts were designed, users can assess the quality and representativeness of the benchmark's data, identify potential biases or limitations, and determine whether the data or prompts are suitable for their specific use case.

Points:

• 0: No documentation of the data collection or environment/prompt design process is provided.
• 5: The data collection or environment/prompt design process is briefly mentioned but no information about the sources, selection criteria, preprocessing steps, or prompt validation is provided.
• 10: The data collection or environment/prompt design process is documented, including some information about the sources, selection criteria, preprocessing steps, or prompt validation, but the description lacks depth or fails to address important aspects.
• 15: The data collection or environment/prompt design process is comprehensively documented, providing a detailed account of the sources of the data, the criteria for selection, any preprocessing steps, and, if applicable, how the prompts were created, validated, and why they were used to elicit the desired responses.

Explanation: The evaluation metrics used are clearly specified and defined, both for standard and custom metrics tailored to the specific task or domain. The exact formulas or processes used to calculate these metrics, along with any parameters or thresholds employed, are made transparent.

Justification: Documenting the evaluation metrics and scoring process is essential for enabling users to understand how the benchmark quantifies model performance and determines rankings or comparisons. By providing clear and detailed information about the metrics and scoring methods, users can assess whether the chosen metrics are appropriate for the task at hand, align with their own evaluation criteria, and provide a fair and meaningful basis for comparing different models or approaches.

Points:

• 0: No documentation of the evaluation metrics is provided.
• 5: The evaluation metrics are mentioned but not clearly defined, and the exact formulas or processes used to calculate them are not provided.
• 10: The evaluation metrics are defined, but the documentation lacks some important details, such as any parameters or thresholds employed.
• 15: The evaluation metrics are clearly specified. The exact formulas or processes used to calculate these metrics, along with any parameters or thresholds employed, are comprehensively documented.

Explanation: Benchmark developers run statistical significance tests on the benchmark results. They report results for, e.g., more than one random seed, and provide variance bounds. In cases where the benchmark is perfectly deterministic, this should be explicitly stated.

Justification: Not doing statistical significance testing can significantly reduce the validity, utility and confidence in results. Especially for benchmarks, we want to understand how much of the results are due to noise and how much is caused by true differences between the models tested.

Points:

• 0: No statistical significance testing or variance reporting is provided for the benchmark results.
• 5: The need for valid benchmarks and/or statistical significance or uncertainty estimation is mentioned but not addressed.
• 10: Benchmark developers "bound the expected variation across model training runs".
• 15: Benchmark developers run statistical significance tests on the benchmark results for at least one model and provide variance bounds or other uncertainty estimations. In cases where the benchmark is perfectly deterministic, this is explicitly stated.

Explanation: The benchmark/its associated paper was accepted to a peer-reviewed journal, conference, or similar venue.

Justification: Acceptance at a peer-reviewed venue signifies that the benchmark has undergone an evaluation by an external party, ensuring its validity, reliability, and scientific merit. This peer review process contributes to the credibility and assurance to users that the benchmark meets established standards of quality and relevance.

Points:

• 0: The benchmark/its associated paper has not been accepted at a peer-reviewed venue.
• 5: The benchmark/its associated paper has been submitted to a peer-reviewed venue but is still under review or awaiting acceptance.
• 10: The benchmark/its associated paper has been accepted at a peer-reviewed workshop or symposium.
• 15: The benchmark/its associated paper has been accepted at a peer-reviewed journal, conference, or similar high-profile venue.

Explanation: The benchmark developers clearly specify the applicable license for the benchmark in the code repository or paper. This includes providing information about the conditions under which the benchmark can be used, modified, and distributed.

Justification: Specifying the applicable license ensures legal clarity and compliance for benchmark users and enables wider adoption, as commercial users might not be able to use the benchmark if no license is specified.

Points:

• 0: No license is specified for the benchmark.
• 5: A license is mentioned but not clearly specified or linked to in the code repository or paper.
• 10: A license is specified but lacks some important details about the conditions under which the benchmark can be used, modified, or distributed.
• 15: The applicable license for the benchmark is clearly specified in the code repository or paper, providing comprehensive information about the conditions under which the benchmark can be used, modified, and distributed.

Explanation: The benchmark dataset and its associated metadata are assigned a globally unique and persistent identifier, such as a Digital Object Identifier (DOI), to ensure long-term accessibility and citability of the resource (FAIR Principles, 2024).

Justification: A persistent identifier supports the findability and accessibility of the benchmark and its dataset. It allows for unambiguous referencing of the data, facilitates proper attribution, and ensures that the dataset can be located and accessed over time, even if its physical location changes. This practice aligns with the FAIR (Findable, Accessible, Interoperable, Reusable) principles, enhancing the benchmark's scientific value and reusability.

Points:

• 0: The benchmark paper, dataset, and metadata are not assigned any persistent identifier.
• 5: The benchmark assigns persistent identifiers to the paper, the dataset, or the metadata.
• 10: The benchmark assigns a persistent identifier to two out of three (paper, dataset, metadata).
• 15: The benchmark assigns a globally unique, persistent identifier to the dataset, its metadata, and the paper.

Explanation: The benchmark includes comprehensive, standardized metadata that describes the dataset, its structure, and relevant information about its creation and usage. This metadata adheres to established standards such as the Croissant standard, which is designed specifically for machine learning datasets.

Justification: Standardized metadata is crucial for ensuring interoperability and reusability of the benchmark dataset. It provides consistent and machine-readable information about the dataset's contents, structure, and provenance. This standardization facilitates easier discovery, understanding, and integration of the dataset into various research workflows. By following established standards like Croissant, the benchmark enhances its utility across different platforms and tools in the machine learning ecosystem.

Points:

• 0: The benchmark does not include any structured metadata.
• 5: The benchmark includes some basic metadata, but it is not standardized or comprehensive.
• 10: The benchmark includes comprehensive metadata that covers most aspects of the dataset, but it does not fully adhere to a recognized standard like Croissant.
• 15: The benchmark includes complete, standardized metadata (e.g., following the Croissant standard) that thoroughly describes all aspects of the dataset, ensuring maximum interoperability and reusability.

Explanation: The benchmark provides comprehensive documentation detailing the origins of the data, the methods used for data collection, and, where applicable, discusses issues of data provenance and informed consent. They also list the license types for all data used and how they ensured compliance with that license.

Justification: Thorough documentation of data sources and collection methods is necessary for ensuring transparency, reproducibility, and ethical design of the benchmark. It allows users to understand the context and limitations of the data, assess its appropriateness for their specific use cases, and make informed decisions about its application. Furthermore, discussing data provenance and informed consent addresses ethical considerations, particularly when dealing with sensitive or personal data, and helps ensure compliance with data protection regulations.

Points:

• 0: The benchmark provides no information about data sources or collection methods.
• 5: The benchmark mentions data sources but provides minimal details about collection methods or ethical considerations.
• 10: The benchmark includes a detailed description of data sources and collection methods, but lacks a discussion of data provenance, compliance with licensing, or informed consent, where applicable.
• 15: The benchmark provides extensive documentation of data sources, collection methods, and a thorough discussion of data provenance, compliance with licensing, and informed consent, addressing relevant ethical and legal considerations.

Explanation: The benchmark provides a detailed account of all preprocessing steps applied to the raw data before its inclusion in the final dataset. This documentation includes information on data cleaning, normalization, feature engineering, handling of missing values, and any other transformations or manipulations performed on the original data. If no data preprocessing was done, the authors state this explicitly.

Justification: Thorough documentation of preprocessing steps is necessary for ensuring reproducibility and transparency of the benchmark. It allows users to understand exactly how the final dataset was created, which is key for interpreting results, replicating experiments, and assessing the benchmark's applicability to different use cases. Additionally, this information helps identify potential biases or artifacts introduced during preprocessing that could affect model performance or generalization.

Points:

• 0: The benchmark provides no information about data preprocessing steps.
• 5: The benchmark mentions that preprocessing was done but offers minimal details about the specific steps taken.
• 10: The benchmark includes a general description of preprocessing steps, but lacks comprehensive details or fails to cover all aspects of the data preparation process.
• 15: The benchmark provides an exhaustive, step-by-step documentation of all preprocessing procedures, including rationales for choices made and potential impacts on the data.

Explanation: The benchmark provides documentation of the data annotation process, including the annotation guidelines, the qualifications and training of annotators, the annotation tools used, quality control measures, and inter-annotator agreement metrics. This documentation covers the entire workflow from raw data to the final annotated dataset.

Justification: Comprehensive documentation of the annotation process is necessary for understanding the quality, reliability, and potential biases in the labeled data. It allows users to assess the suitability of the dataset for their specific tasks and to interpret results accurately. Transparent annotation documentation also enables reproducibility of the labeling process, facilitates improvements in future iterations of the benchmark, and helps in identifying and mitigating potential sources of bias or error in the annotations.

Points:

• 0: The benchmark provides no information about the data annotation process.
• 5: The benchmark mentions that data was annotated but offers minimal details about the process or guidelines used.
• 10: The benchmark includes a general description of the annotation process, including guidelines and tools used, but lacks comprehensive details on quality control measures or inter-annotator agreement.
• 15: The benchmark provides exhaustive documentation of the entire annotation process, including detailed guidelines, annotator information, quality control measures, inter-annotator agreement metrics, and discussions of potential biases or limitations in the annotation approach.

Explanation: The benchmark provides analysis and documentation of how representative the dataset or environment is of the target population or domain. This includes an explanation of the sampling procedure used, any potential biases in the data collection process, and how well the dataset captures the diversity and distribution of the intended population or phenomenon being studied.

Justification: Understanding the representativeness of the data is necessary for assessing the generalizability and validity of any conclusions drawn from models trained or evaluated on the benchmark. It helps users identify potential limitations or biases in the dataset that could affect model performance in real-world applications. Proper documentation of representativeness also aids in interpreting benchmark results within the context of the population it represents and highlights areas where the dataset may need expansion or improvement to better cover underrepresented groups or scenarios.

Points:

• 0: The benchmark provides no information about the representativeness of the data or the sampling procedure used.
• 5: The benchmark mentions the importance of data representativeness but offers minimal analysis or explanation of how representative the dataset actually is.
• 10: The benchmark includes a general discussion of data representativeness and the sampling procedure, but lacks comprehensive analysis or fails to address potential biases or limitations in representativeness.
• 15: The benchmark provides an in-depth analysis of data representativeness, including detailed explanation of the sampling procedure, quantitative measures of population coverage, discussion of potential biases, and acknowledgment of any limitations in representativeness.

Explanation: The benchmark utilizes a standardized documentation format, such as data cards, to present the information about the dataset that is underlying to the benchmark. This standardized approach ensures that all key aspects of the dataset are systematically covered, including its composition, collection methodology, intended uses, ethical considerations, and potential biases.

Justification: Adopting a standardized documentation scheme like data cards enhances the usability and transparency of the benchmark. It provides a consistent, structured format that makes it easier for users to quickly understand the dataset's characteristics, limitations, and appropriate use cases. Standardized documentation facilitates easier comparison between datasets and benchmarks, promotes best practices in data reporting, and helps identify potential issues or gaps in the dataset's coverage.

Points:

• 0: The benchmark does not use any standardized documentation scheme.
• 5: The benchmark includes some elements of standardized documentation, but does not fully adhere to an established scheme like data cards.
• 10: The benchmark uses a standardized documentation scheme, but some sections are incomplete or lack detail.
• 15: The benchmark fully implements a comprehensive standardized documentation scheme (e.g., data cards), providing thorough and structured information on all relevant aspects of the dataset.

Explanation: The main files of the public code were updated within the last year, or the developers checked that the benchmark code is still usable and explicitly state this check in the README file, including the date of the check.

Justification: Over time, packages that the benchmark depends on may be updated and become incompatible with the original evaluation/benchmark code. To ensure ongoing usability, benchmark developers must check if their code can still be used at least once a year. This practice ensures that users can use the benchmark without encountering and having to fix issues due to outdated dependencies.

Points:

• 0: No updates to the main files of the public code within the last year, and no explicit statement of a usability check in the README file.
• 5: Updates to minor files in the repo were made (e.g., README file) but an explicit statement of a usability check in the README file is not reported.
• 10: Updates to the main files of the public code were made within the last year, but the build status check failed and wasn't fixed.
• 15: Updates to the main files of the public code within the last year, accompanied by a successful build status check, or an explicit statement of a usability check in the README file, including the date of the check was provided.

Explanation: GitHub issues are acknowledged or addressed within three months. If there are no open issues, benchmark developers would get full points.

Justification: Over time, users may find issues with the benchmark tasks or implementation. To ensure continued usability, benchmark developers should address these concerns in a reasonable amount of time. Promptly responding to user feedback helps maintain the reliability and relevance of the benchmark.

Points:

• 0: No acknowledgment or response to GitHub issues that are older than three months.
• 5: GitHub issues are mentioned as a way to provide feedback but there are GitHub issues that were not responded to and that are older than three months.
• 10: All GitHub issues are acknowledged within three months, but not all are addressed or resolved or were closed because the issue/feature request won't be attended to.
• 15: All GitHub issues are acknowledged and addressed within three months, or it is clearly stated if an issue cannot be fixed or if a feature request won't be fulfilled. Alternatively, there are no open issues.

Explanation: The benchmark should include contact details of the person responsible, such as a corresponding author in the associated paper, a contact person listed on GitHub or the website, or an available online feedback form.

Justification: Providing contact details ensures that users have a communication channel for inquiries, feedback, or reporting issues related to the benchmark. This transparency supports effective collaboration and resolution of problems, enhancing the benchmark's usability.

Points:

• 0: It is not disclosed who developed the benchmark.
• 5: The benchmark developers are disclosed but no explicit contact details are provided.
• 10: Contact details are provided but are incomplete or difficult to find, e.g., only as part of terms of service on a website.
• 15: Contact details of the person responsible for the benchmark are easily accessible, such as a corresponding author in the associated paper, a contact person listed on GitHub or the website, or an available online feedback form.