BrowseComp-Plus Leaderboard

Metrics Details

For each query, human labelers annotated:

• Evidence documents: documents that are needed to answer the query.
• Gold documents: documents that are needed to answer the query, and semantically contains the final answer.

For the agents leaderboard:

• Accuracy: Comparing the generated answer to the ground-truth answer. We follow BrowseComp to judge using GPT-4.1; however, due to concerns about the long-term availability of GPT-4.1, we also provide a judge using Qwen3-32B, an open-weight model. We will prioritize the Qwen3-32B judge for future submissions.
• Recall: The recall of all documents ever retrieved by the agent, against labeled evidence documents.
• Search Calls: The number of search calls made by the agent.
• Calibration Error: The calibration error of the agent, following BrowseComp.

For the retrieval leaderboard:

• Evidence metrics: standard retrieval metrics against evidence documents.
• Gold metrics: standard retrieval metrics against gold documents.

Submitting to BrowseComp-Plus

To submit to the agents leaderboard, please prepare your results as a json file. It should contain the following fields:

{
    "LLM": str, # The name of the LLM agent
    "Retriever": str, # The name of the retriever
    "Accuracy (%)": float, # Average accuracy of the LLM agent, between 0 and 100
    "Recall (%)": float, # Average recall of the LLM agent, between 0 and 100
    "Search Calls": float, # Average number of search calls made by the LLM agent
    "Calibration Error (%)": float, # Average calibration error of the LLM agent, between 0 and 100
    "Link": str, # The link to the model card
    "Evaluation Date": str, # The date of the evaluation, in the format of "YYYY-MM-DD"
    "per_query_metrics": [
        {
            "query_id": str, # The query id
            "correct": bool, # Whether the agent answered correctly
            "recall": float, # The recall of the LLM agent for this query, between 0 and 100
        },
        ...
    ]
}

The correctness of each query (Accuracy) should be judged using LLM as judge. You may find the evaluation script and more details about the metrics in the Github repository.

Once you have prepared the results, please email them to s42chen@uwaterloo.ca.

To submit to the retrieval leaderboard, please prepare a json file with the following fields:

{
    "Retriever": str, # The name of the retriever
    "Evidence nDCG@10": float,
    "Evidence Recall@5": float,
    "Evidence Recall@100": float,
    "Evidence Recall@1000": float,
    "Gold nDCG@10": float,
    "Gold Recall@5": float,
    "Gold Recall@100": float,
    "Gold Recall@1000": float,
    "Link": str, # The link to the model card
    "Evaluation Date": str, # The date of the evaluation, in the format of "YYYY-MM-DD"
}

where the labels for evidence and gold documents, along with evaluation scripts, are provided in the Github repository.

Note that though the baselines provided by BrowseComp-Plus are retriever-only systems, you are welcome to submit results with reranking, or any other ranking mechanism in your system to the retrieval leaderboard.

Once you have prepared the results, please email them to s42chen@uwaterloo.ca.

Are you (not) a human?

For fair comparison on the leaderboard, we enforce that the results submitted must be entirely generated from a language model, with no further human editing. To ensure this and speed up the verification process, we encourage you to include instructions / code to reproduce your results, or a short description / report of your models and approaches.

More questions?

You can file an issue on GitHub, or contact:

• Zijian Chen: s42chen@uwaterloo.ca
• Xueguang Ma: x93ma@uwaterloo.ca
• Shengyao Zhuang: s.zhuang@uq.edu.au