BrowseComp-Plus Leaderboard

Metrics Details

For each query, human labelers annotated:

• Evidence documents: documents that are needed to answer the query.
• Gold documents: documents that are needed to answer the query, and semantically contains the final answer.

For the agents leaderboard:

• Accuracy: Comparing the generated answer to the ground-truth answer. We follow BrowseComp to evaluate with LLM-as-a-judge. However, due to concerns about the long-term availability of GPT-4.1, instead of using GPT-4.1, we evaluate with Qwen3-32B, an open-weight model. Legacy GPT-4.1 judged results are kept to showcase that the two judges differ by < 1%, but we will prioritize the Qwen3-32B judge for future submissions. The LLM-judged results have been human verified to be reliable.
• Recall: The recall of all documents ever retrieved by the agent, against labeled evidence documents.
• Search Calls: The number of search calls made by the agent.
• Calibration Error: The calibration error of the agent, following BrowseComp.
• Custom Tool Accuracy: Some models were trained to use tools beyond simple search, such as grepping within documents or reading a page with another LLM-summarizer. For fair comparison between various models, BrowseComp-Plus evaluates with just a simple search tool as the main metric. However, we recognize that it is also important to showcase models' upper-bound performance with custom tools. Thus, we include the "Custom Tool Accuracy" column to evaluate models with their native toolset, and we welcome submissions with custom tool accuracy results.

For the retrieval leaderboard:

• Evidence metrics: standard retrieval metrics against evidence documents.
• Gold metrics: standard retrieval metrics against gold documents.

Submitting to BrowseComp-Plus

To submit to the agents leaderboard, please prepare your results as a json file. It should contain the following fields:

{
    "LLM": str, # The name of the LLM agent
    "Retriever": str, # The name of the retriever
    "Accuracy (%)": float, # Average accuracy of the LLM agent, between 0 and 100
    "Recall (%)": float, # Average recall of the LLM agent, between 0 and 100
    "Search Calls": float, # Average number of search calls made by the LLM agent
    "Calibration Error (%)": float, # Average calibration error of the LLM agent, between 0 and 100
    "Link": str, # The link to the model card
    "Evaluation Date": str, # The date of the evaluation, in the format of "YYYY-MM-DD"
    "per_query_metrics": [
        {
            "query_id": str, # The query id
            "correct": bool, # Whether the agent answered correctly
            "recall": float, # The recall of the LLM agent for this query, between 0 and 100
        },
        ...
    ]
}

The correctness of each query (Accuracy) should be judged using LLM as judge. You may find the evaluation script and more details about the metrics in the Github repository.

Once you have prepared the results, please email them to s42chen@uwaterloo.ca.

To submit to the retrieval leaderboard, please prepare a json file with the following fields:

{
    "Retriever": str, # The name of the retriever
    "Evidence nDCG@10": float,
    "Evidence Recall@5": float,
    "Evidence Recall@100": float,
    "Evidence Recall@1000": float,
    "Gold nDCG@10": float,
    "Gold Recall@5": float,
    "Gold Recall@100": float,
    "Gold Recall@1000": float,
    "Link": str, # The link to the model card
    "Evaluation Date": str, # The date of the evaluation, in the format of "YYYY-MM-DD"
}

where the labels for evidence and gold documents, along with evaluation scripts, are provided in the Github repository.

Note that though the baselines provided by BrowseComp-Plus are retriever-only systems, you are welcome to submit results with reranking, or any other ranking mechanism in your system to the retrieval leaderboard.

Once you have prepared the results, please email them to s42chen@uwaterloo.ca.

Are you (not) a human?

For fair comparison on the leaderboard, we enforce that the results submitted must be entirely generated from a language model, with no further human editing. To ensure this and speed up the verification process, we encourage you to include instructions / code to reproduce your results, or a short description / report of your models and approaches.

More questions?

You can file an issue on GitHub, or contact:

• Zijian Chen: s42chen@uwaterloo.ca
• Xueguang Ma: x93ma@uwaterloo.ca
• Shengyao Zhuang: s.zhuang@uq.edu.au