Can Frontier Models Recall Long-Tail Facts? A Cricket Stress Test

Last year we ran a small experiment on language-model recall using cricket scorecards. We returned to the idea with current models, a larger controlled sample, and a second prompt that forces every model to answer.

Frontier models are trained on enormous corpora and may have encountered billions of facts, but exposure is not the same as reliable recall. The real question is whether a model can be trusted to recover an exact fact buried in that long tail—and whether anyone can verify where its answer came from. The same failure mode matters when someone asks about version-specific library behavior, an obscure compiler flag, a detail in a medical paper, or an exception in an internal policy.

Why use cricket scorecards?

A scorecard gives us unusually clean factual questions. A team won the toss or it did not. A batter scored a specific number of runs. A bowler took an exact number of credited wickets. The answers are public, structured, and easy to derive independently, while many lower-profile international matches are obscure enough to test long-tail recall.

This is deliberately not a contamination-free test of unseen events. The 110 matches run from November 2013 through January 22, 2026, and some or all may have appeared in pretraining. That is the experiment: even when a fact could be encoded in model parameters, can the model recover it accurately without retrieval?

Dataset

We used the JSON match data published by Cricsheet. We defined the long tail as international T20 matches that did not involve an ICC Full Member national side. From 3,150 eligible matches, a fixed seed selected 110 distinct matches:

11 question types, with exactly 10 questions of each type;
55 women's matches and 55 men's matches;
one question per source match;
toss winner and decision, match winner, victory margin, team total, top scorer, top wicket-taker, and total match runs.

Choice questions supplied the allowed options; numeric questions required an exact integer. We scored an answer as correct only when it exactly matched the value derived from Cricsheet's delivery data. Ties in player-leader questions were represented as a set of valid names.

Models and two recall modes

We tested GPT-5.6 Luna, Terra, and Sol, plus Claude Sonnet 5 and Opus 5. Every no-search run used the provider's explicit low reasoning effort setting and the same 110 questions.

Abstention allowed: use internal knowledge only, do not guess, and return no answer when uncertain.
Forced recall: use internal knowledge only and always provide the best available answer, even when uncertain.

We also ran GPT-5.6 Luna with web search required. That is a retrieval condition, not a recall condition, and provides a useful same-model comparison. The tool only serialized the answer; it performed no lookup. All 1,210 evaluated calls completed without an API or schema error.

When models may abstain

The GPT-5.6 models recovered only 15 or 16 facts each, but behaved very differently around uncertainty. Sol attempted 17 answers and got 15 right. Luna attempted 38 and got only 16 right. Both Claude models abstained on all 110 questions.

Model	Answered	Correct	Wrong	Abstained	Overall accuracy	Accuracy when answered
GPT-5.6 Luna	38	16	22	72	14.5%	42.1%
GPT-5.6 Terra	34	15	19	76	13.6%	44.1%
GPT-5.6 Sol	17	15	2	93	13.6%	88.2%
Claude Sonnet 5	0	0	0	110	0.0%	—
Claude Opus 5	0	0	0	110	0.0%	—

The Claude result initially looked suspicious, so we checked the returned model IDs, effort settings, schemas, and per-call responses. We then removed the escape hatch and reran both Claude models in forced-recall mode. The original abstentions were therefore meaningful calibration behavior, not missing responses.

When every model must answer

Unsurprisingly, forcing a model to answer obscure factual questions produced many more wrong answers. It also surfaced additional correct recall: Sol rose from 15 correct answers in abstention mode to 40, Terra from 15 to 32, and Luna from 16 to 30. Sonnet and Opus moved from answering nothing to getting 27 and 26 questions right. Forced answers were still not reliable—the models scored only 23.6% to 36.4%, and every model returned more wrong answers than correct ones—but the output did not collapse into pure noise. Some long-tail facts appear to have been available to the models even when their uncertainty policy prevented them from answering.

Within the GPT-5.6 family, the larger model showed a clearer recall signal. Sol was correct on 15 of the 17 questions it chose to answer, and it led forced recall with 40 of 110 correct. That pattern suggests both stronger recall and better discrimination about when that recall is trustworthy. We did not see the same size effect in the Claude family: Sonnet and Opus were nearly identical in forced recall, with 27 and 26 correct respectively, and both abstained on every question when abstention was allowed. That contrast is interesting: scale appeared to improve recall and calibration within GPT-5.6, while the Claude pair showed no measurable Opus advantage on this sample.

Model	Correct	Wrong	Accuracy
GPT-5.6 Luna	30	80	27.3%
GPT-5.6 Terra	32	78	29.1%
GPT-5.6 Sol	40	70	36.4%
Claude Sonnet 5	27	83	24.5%
Claude Opus 5	26	84	23.6%

Forced recall more than doubled the number of correct GPT answers in some cases, but the added coverage was expensive. Relative to abstention mode, Luna gained 14 correct answers and 58 additional wrong claims; Terra gained 17 correct and 59 wrong; Sol gained 25 correct and 68 wrong. Claude moved from total abstention to roughly one correct answer in four, accompanied by 83 or 84 wrong claims.

The easiest categories were often binary choices such as match winner, toss winner, or toss decision, where chance and team priors can help. Exact totals and player facts were much harsher. In forced mode, neither Claude model got a team total, top-scorer run count, or total-match-run count correct. This is closer to the long-tail behavior that matters in exact factual work.

Retrieval and provenance change the result

GPT-5.6 Luna with required web search answered 105 questions: 98 matched the Cricsheet-derived reference, 7 did not, and 5 were abstentions. Reference agreement rose from 14.5% in calibrated recall and 27.3% in forced recall to 89.1% with retrieval.

Those seven results are reference mismatches, not definitive proof that the retrieved answers were false; Cricsheet and other public sources can disagree. One response returned Bulgaria's 119 from an earlier match on the same day rather than the reference value of 125 for the requested match, while other mismatches involved a nearby player or wicket count. These are plausible retrieval outcomes, but they show why systems still need to identify and expose the event, source, version, and jurisdiction used for an answer.

The broader long-tail problem

A language model compresses patterns from enormous corpora; it is not a lossless factual database. Widely repeated facts have many opportunities to shape the model. A single scorecard, a rarely cited court opinion, or one paragraph in an old administrative order may have only a weak trace. When prompted, the model can blend that trace with common patterns and produce an answer that sounds right.

Consider debugging a regression in an older library version. A model may remember the package and general issue but describe behavior from a nearby release, attribute the fix to the wrong version, suggest a configuration option introduced later, or omit that an API was renamed or deprecated. Those are structurally similar to retrieving the right teams but the wrong same-day score. The surface fluency offers no reliable indication that the exact fact was recalled.

This experiment does not measure software-engineering, legal, medical, or compliance accuracy, and its percentages should not be transferred to those domains. It demonstrates the underlying failure mode: forcing a fluent model to answer a sparse exact-fact question can turn uncertainty into a confident near miss.

What to build instead

Retrieve before answering when a fact is exact, obscure, or consequential.
Show provenance so users can verify the source, date, version, release, document, or event actually used.
Preserve abstention when evidence is missing; forced coverage can sharply increase false claims.
Measure coverage and error together. Accuracy among answered questions alone can reward a system that rarely helps, while answer rate alone rewards guessing.
Validate retrieved identity. Search can return a related package version, match, regulation, or document that looks convincing but answers a different question.

Conclusion

The largest forced-recall score in this experiment was 36.4%. Required web retrieval raised reference agreement to 89.1%. That does not make search infallible, but it changes an ungrounded act of memory into a process that can be inspected: which source was found, which event or document it describes, when it was published, and whether it actually supports the answer.

A larger model and a larger training corpus can increase the chance that an obscure fact left some trace in the parameters. Neither provides provenance, guarantees exact recovery, or tells the user when a fluent answer came from a nearby fact instead. When a long-tail fact matters, retrieval supplies the evidence and provenance makes that evidence auditable.

Limitations

The benchmark contains 110 questions and only 10 per type, so category-level estimates are noisy.
Each condition was run once at low reasoning effort; we did not measure repeatability or higher-effort behavior.
Binary-choice questions have a high chance baseline and may reward priors rather than exact recall.
OpenAI and Anthropic used different schema mechanisms, although both produced the same typed answer fields.
Cricsheet is the scoring reference. A mismatch means disagreement with that reference, not automatically proof of hallucination.
This is a cricket benchmark used to study a general mechanism, not direct evidence about error rates in other domains.

Reproducibility and references

The complete evaluation runner and the fixed 110-question sample are embedded below. The sample was generated with seed 20260726; its SHA-256 is 12076f3c55443456f83238c62c8eef3af0575e3188fe4b89b1394d201935210e. The source archive SHA-256 is 89344b4752358c6292364d6dea4df82fa74370251f4337d11c473788e3ea064e.

Complete evaluation runner (Python)

Save as eval_cricket_long_tail.py. It prepares the fixed sample, runs OpenAI or Anthropic conditions, resumes interrupted JSONL runs, and creates the aggregate report.


#!/usr/bin/env python3
"""Build and run a reproducible long-tail cricket factual-recall evaluation."""

from __future__ import annotations

import argparse
import hashlib
import json
import math
import os
import random
import statistics
import time
import zipfile
from collections import Counter, defaultdict
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
from typing import Any, Iterable

import httpx
from openai import OpenAI


QUESTION_TYPES = (
    "toss_winner",
    "toss_decision",
    "match_winner",
    "victory_margin_runs",
    "victory_margin_wickets",
    "team_total",
    "top_scorer_name",
    "top_scorer_runs",
    "top_wicket_taker_name",
    "top_wicket_taker_wickets",
    "total_match_runs",
)

FULL_MEMBER_TEAMS = {
    "Afghanistan",
    "Australia",
    "Bangladesh",
    "England",
    "India",
    "Ireland",
    "New Zealand",
    "Pakistan",
    "South Africa",
    "Sri Lanka",
    "West Indies",
    "Zimbabwe",
}

NON_BOWLER_DISMISSALS = {
    "obstructing the field",
    "retired hurt",
    "retired out",
    "run out",
    "timed out",
}


def stable_json(value: Any) -> str:
    return json.dumps(value, ensure_ascii=False, sort_keys=True, separators=(",", ":"))


def file_sha256(path: Path) -> str:
    digest = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()


def write_jsonl_atomic(path: Path, rows: Iterable[dict[str, Any]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temp_path = path.with_suffix(path.suffix + ".new")
    with temp_path.open("w", encoding="utf-8") as handle:
        for row in rows:
            handle.write(stable_json(row) + "\n")
        handle.flush()
        os.fsync(handle.fileno())
    os.replace(temp_path, path)


def write_json_atomic(path: Path, value: Any) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    temp_path = path.with_suffix(path.suffix + ".new")
    with temp_path.open("w", encoding="utf-8") as handle:
        json.dump(value, handle, ensure_ascii=False, indent=2, sort_keys=True)
        handle.write("\n")
        handle.flush()
        os.fsync(handle.fileno())
    os.replace(temp_path, path)


def regular_innings(match: dict[str, Any]) -> list[dict[str, Any]]:
    return [innings for innings in match["innings"] if not innings.get("super_over", False)]


def deliveries(match: dict[str, Any]) -> Iterable[dict[str, Any]]:
    for innings in regular_innings(match):
        for over in innings.get("overs", []):
            yield from over.get("deliveries", [])


def team_totals(match: dict[str, Any]) -> dict[str, int]:
    totals: dict[str, int] = defaultdict(int)
    for innings in regular_innings(match):
        total = sum(
            delivery["runs"]["total"]
            for over in innings.get("overs", [])
            for delivery in over.get("deliveries", [])
        )
        totals[innings["team"]] += total
    return dict(totals)


def batter_runs(match: dict[str, Any]) -> dict[str, int]:
    runs: dict[str, int] = defaultdict(int)
    for delivery in deliveries(match):
        runs[delivery["batter"]] += delivery["runs"]["batter"]
    return dict(runs)


def bowler_wickets(match: dict[str, Any]) -> dict[str, int]:
    wickets: dict[str, int] = defaultdict(int)
    for delivery in deliveries(match):
        for wicket in delivery.get("wickets", []):
            if (wicket.get("kind") or "").lower() not in NON_BOWLER_DISMISSALS:
                wickets[delivery["bowler"]] += 1
    return dict(wickets)


def player_options(info: dict[str, Any]) -> list[str]:
    players = info.get("players") or {}
    return sorted({player for team_players in players.values() for player in team_players})


def prompt_context(info: dict[str, Any]) -> str:
    teams = info["teams"]
    date = str(info["dates"][0])
    venue = info.get("venue") or info.get("city") or "an unspecified venue"
    return f"between {teams[0]} and {teams[1]} on {date} at {venue}"


def base_item(
    source: str,
    question_type: str,
    mode: str,
    prompt: str,
    info: dict[str, Any],
) -> dict[str, Any]:
    event = info.get("event") or {}
    competition = event.get("name", "") if isinstance(event, dict) else str(event)
    return {
        "id": f"{source}#{question_type}",
        "type": question_type,
        "mode": mode,
        "prompt": prompt,
        "source": source,
        "date": str(info["dates"][0]),
        "gender": info.get("gender", "unknown"),
        "competition": competition,
        "teams": info["teams"],
    }


def generate_items(source: str, match: dict[str, Any]) -> list[dict[str, Any]]:
    info = match["info"]
    context = prompt_context(info)
    teams = info["teams"]
    items: list[dict[str, Any]] = []

    def choice(question_type: str, prompt: str, options: list[str], gold_set: list[str]) -> None:
        if not options or not gold_set or not set(gold_set).issubset(options):
            return
        item = base_item(source, question_type, "choice", prompt, info)
        item.update({"options": options, "gold_set": sorted(gold_set)})
        items.append(item)

    def number(
        question_type: str,
        prompt: str,
        gold: int | None,
        suffix: str = "",
        meta: dict[str, Any] | None = None,
    ) -> None:
        if gold is None:
            return
        item = base_item(source, question_type, "number", prompt, info)
        if suffix:
            item["id"] += f":{suffix}"
        item["gold"] = int(gold)
        if meta:
            item["meta"] = meta
        items.append(item)

    toss = info.get("toss") or {}
    if toss.get("winner"):
        choice("toss_winner", f"Who won the toss in the T20 match {context}?", teams, [toss["winner"]])
    if toss.get("decision"):
        choice(
            "toss_decision",
            f"What did the toss winner choose to do in the T20 match {context}?",
            ["bat", "field"],
            [toss["decision"]],
        )

    outcome = info.get("outcome") or {}
    if outcome.get("winner"):
        choice("match_winner", f"Who won the T20 match {context}?", teams, [outcome["winner"]])
    margin = outcome.get("by") or {}
    if "runs" in margin:
        number("victory_margin_runs", f"By how many runs was the T20 match {context} won?", margin["runs"])
    if "wickets" in margin:
        number(
            "victory_margin_wickets",
            f"By how many wickets was the T20 match {context} won?",
            margin["wickets"],
        )

    totals = team_totals(match)
    for team, total in sorted(totals.items()):
        opponent = next((candidate for candidate in teams if candidate != team), "their opponent")
        number(
            "team_total",
            f"How many runs did {team} score in the T20 match against {opponent} on {info['dates'][0]} at {info.get('venue') or info.get('city') or 'an unspecified venue'}?",
            total,
            suffix=team,
            meta={"team": team},
        )

    options = player_options(info)
    runs = batter_runs(match)
    if runs:
        top_runs = max(runs.values())
        top_scorers = sorted(player for player, value in runs.items() if value == top_runs)
        choice("top_scorer_name", f"Who scored the most runs in the T20 match {context}?", options, top_scorers)
        picked_scorer = top_scorers[0]
        number(
            "top_scorer_runs",
            f"How many runs did {picked_scorer} score in the T20 match {context}?",
            runs[picked_scorer],
            suffix=picked_scorer,
            meta={"player": picked_scorer},
        )

    wickets = bowler_wickets(match)
    if wickets:
        top_wickets = max(wickets.values())
        top_bowlers = sorted(player for player, value in wickets.items() if value == top_wickets)
        choice(
            "top_wicket_taker_name",
            f"Who took the most bowler-credited wickets in the T20 match {context}?",
            options,
            top_bowlers,
        )
        picked_bowler = top_bowlers[0]
        number(
            "top_wicket_taker_wickets",
            f"How many bowler-credited wickets did {picked_bowler} take in the T20 match {context}?",
            wickets[picked_bowler],
            suffix=picked_bowler,
            meta={"player": picked_bowler},
        )

    if totals:
        number("total_match_runs", f"How many total runs were scored by both teams in the T20 match {context}?", sum(totals.values()))

    return items


def prepare(args: argparse.Namespace) -> None:
    archive = Path(args.archive)
    candidates: dict[tuple[str, str], list[dict[str, Any]]] = defaultdict(list)
    eligible_matches = 0

    with zipfile.ZipFile(archive) as bundle:
        for source in sorted(name for name in bundle.namelist() if name.endswith(".json")):
            match = json.loads(bundle.read(source))
            info = match.get("info") or {}
            date = str((info.get("dates") or [""])[0])
            if info.get("match_type") not in {"T20", "IT20"}:
                continue
            if date > args.cutoff:
                continue
            if info.get("team_type") != "international":
                continue
            if any(team in FULL_MEMBER_TEAMS for team in info.get("teams", [])):
                continue
            if info.get("gender") not in {"male", "female"}:
                continue
            eligible_matches += 1
            for item in generate_items(source, match):
                candidates[(item["type"], item["gender"])].append(item)

    rng = random.Random(args.seed)
    selected: list[dict[str, Any]] = []
    used_sources: set[str] = set()
    for question_type in QUESTION_TYPES:
        for gender in ("female", "male"):
            pool = list(candidates[(question_type, gender)])
            rng.shuffle(pool)
            chosen = []
            for item in pool:
                if item["source"] in used_sources:
                    continue
                chosen.append(item)
                used_sources.add(item["source"])
                if len(chosen) == args.per_type_gender:
                    break
            if len(chosen) != args.per_type_gender:
                raise RuntimeError(
                    f"Only selected {len(chosen)} items for {question_type}/{gender}; "
                    f"needed {args.per_type_gender}"
                )
            selected.extend(chosen)

    rng.shuffle(selected)
    output = Path(args.output)
    write_jsonl_atomic(output, selected)
    sample_hash = file_sha256(output)
    manifest = {
        "archive": archive.name,
        "archive_sha256": file_sha256(archive),
        "cutoff_inclusive": args.cutoff,
        "definition": "International T20 matches not involving an ICC Full Member national side",
        "eligible_matches": eligible_matches,
        "full_member_teams_excluded": sorted(FULL_MEMBER_TEAMS),
        "genders": dict(Counter(item["gender"] for item in selected)),
        "match_count": len(used_sources),
        "per_type_gender": args.per_type_gender,
        "question_count": len(selected),
        "question_types": list(QUESTION_TYPES),
        "sample_sha256": sample_hash,
        "seed": args.seed,
        "source_date_max": max(item["date"] for item in selected),
        "source_date_min": min(item["date"] for item in selected),
        "types": dict(Counter(item["type"] for item in selected)),
    }
    write_json_atomic(output.with_suffix(".manifest.json"), manifest)
    print(json.dumps(manifest, indent=2, sort_keys=True))


def response_format(item: dict[str, Any], force_answer: bool = False) -> dict[str, Any]:
    value_key = "choice" if item["mode"] == "choice" else "number"
    if item["mode"] == "choice":
        choice_schema: dict[str, Any] = {"type": "string", "enum": item["options"]}
        value_schema = (
            choice_schema
            if force_answer
            else {"anyOf": [choice_schema, {"type": "null"}]}
        )
    else:
        value_schema = (
            {"type": "integer"}
            if force_answer
            else {"anyOf": [{"type": "integer"}, {"type": "null"}]}
        )
    answered_schema: dict[str, Any] = (
        {"type": "boolean", "enum": [True]} if force_answer else {"type": "boolean"}
    )
    return {
        "type": "json_schema",
        "name": f"cricket_{item['mode']}_answer",
        "strict": True,
        "schema": {
            "type": "object",
            "properties": {
                "answered": answered_schema,
                value_key: value_schema,
            },
            "required": ["answered", value_key],
            "additionalProperties": False,
        },
    }


def build_input(
    item: dict[str, Any], web_search: bool, force_answer: bool = False
) -> list[dict[str, Any]]:
    if web_search:
        system = (
            "Use web search to verify the cricket scorecard fact. Answer only when the evidence supports it. "
            "Set answered=false and the answer value to null if reliable evidence is unavailable."
        )
    elif force_answer:
        system = (
            "Use only knowledge already contained in the model; no external lookup is available. "
            "Always provide your best answer even when uncertain. Set answered=true."
        )
    else:
        system = (
            "Use only knowledge already contained in the model; no external lookup is available. Do not guess. "
            "Set answered=false and the answer value to null when uncertain."
        )
    user = item["prompt"]
    if item["mode"] == "choice":
        user += "\nOptions: " + json.dumps(item["options"], ensure_ascii=False)
        user += "\nReturn one option exactly when answered=true."
    else:
        user += "\nReturn an integer when answered=true."
    return [
        {"role": "system", "content": system},
        {"role": "user", "content": user},
    ]


def find_url_citations(value: Any) -> list[dict[str, str]]:
    found: list[dict[str, str]] = []
    if isinstance(value, dict):
        if value.get("type") == "url_citation" and value.get("url"):
            found.append({"url": value["url"], "title": value.get("title", "")})
        for child in value.values():
            found.extend(find_url_citations(child))
    elif isinstance(value, list):
        for child in value:
            found.extend(find_url_citations(child))
    return list({entry["url"]: entry for entry in found}.values())


def score_prediction(item: dict[str, Any], prediction: dict[str, Any]) -> dict[str, int]:
    if prediction.get("answered") is not True:
        return {"answered": 0, "correct": 0, "wrong": 0, "abstained": 1}
    if item["mode"] == "choice":
        correct = prediction.get("choice") in item["gold_set"]
    else:
        value = prediction.get("number")
        correct = isinstance(value, int) and not isinstance(value, bool) and value == item["gold"]
    return {
        "answered": 1,
        "correct": int(correct),
        "wrong": int(not correct),
        "abstained": 0,
    }


def evaluate_one(
    client: OpenAI,
    item: dict[str, Any],
    model: str,
    reasoning_effort: str,
    web_search: bool,
    force_answer: bool,
) -> dict[str, Any]:
    started = time.monotonic()
    request: dict[str, Any] = {
        "model": model,
        "input": build_input(item, web_search, force_answer),
        "reasoning": {"effort": reasoning_effort},
        "text": {"verbosity": "low", "format": response_format(item, force_answer)},
        "store": False,
    }
    if web_search:
        request.update({"tools": [{"type": "web_search"}], "tool_choice": "required"})
    try:
        response = client.responses.create(**request)
        response_data = response.model_dump(mode="json")
        prediction = json.loads(response.output_text)
        score = score_prediction(item, prediction)
        return {
            "id": item["id"],
            "source": item["source"],
            "type": item["type"],
            "mode": item["mode"],
            "gender": item["gender"],
            "prompt": item["prompt"],
            "gold": item.get("gold", item.get("gold_set")),
            "prediction": prediction,
            **score,
            "error": None,
            "provider": "openai",
            "requested_model": model,
            "response_model": response.model,
            "response_id": response.id,
            "reasoning_effort": reasoning_effort,
            "force_answer": force_answer,
            "web_search": web_search,
            "web_search_calls": sum(1 for output in response.output if output.type == "web_search_call"),
            "citations": find_url_citations(response_data.get("output", [])),
            "usage": response_data.get("usage"),
            "latency_ms": round((time.monotonic() - started) * 1000),
        }
    except Exception as exc:
        return {
            "id": item["id"],
            "source": item["source"],
            "type": item["type"],
            "mode": item["mode"],
            "gender": item["gender"],
            "prompt": item["prompt"],
            "gold": item.get("gold", item.get("gold_set")),
            "prediction": None,
            "answered": 0,
            "correct": 0,
            "wrong": 0,
            "abstained": 0,
            "error": f"{type(exc).__name__}: {exc}",
            "provider": "openai",
            "requested_model": model,
            "response_model": None,
            "response_id": None,
            "reasoning_effort": reasoning_effort,
            "force_answer": force_answer,
            "web_search": web_search,
            "web_search_calls": 0,
            "citations": [],
            "usage": None,
            "latency_ms": round((time.monotonic() - started) * 1000),
        }


def evaluate_one_anthropic(
    client: httpx.Client,
    api_key: str,
    item: dict[str, Any],
    model: str,
    reasoning_effort: str,
    force_answer: bool,
) -> dict[str, Any]:
    """Evaluate one item through Anthropic's Messages API without retrieval tools."""
    started = time.monotonic()
    input_messages = build_input(item, web_search=False, force_answer=force_answer)
    answer_tool = {
        "name": "submit_answer",
        "description": (
            "Submit the best available answer."
            if force_answer
            else "Submit the answer or abstain when uncertain."
        ),
        "input_schema": response_format(item, force_answer)["schema"],
    }
    request = {
        "model": model,
        "max_tokens": 1024,
        "system": input_messages[0]["content"],
        "messages": [input_messages[1]],
        "tools": [answer_tool],
        "tool_choice": {
            "type": "tool",
            "name": "submit_answer",
            "disable_parallel_tool_use": True,
        },
        "output_config": {"effort": reasoning_effort},
    }
    try:
        response = client.post(
            "https://api.anthropic.com/v1/messages",
            headers={
                "anthropic-version": "2023-06-01",
                "content-type": "application/json",
                "x-api-key": api_key,
            },
            json=request,
        )
        response.raise_for_status()
        response_data = response.json()
        tool_uses = [
            block
            for block in response_data.get("content", [])
            if block.get("type") == "tool_use" and block.get("name") == "submit_answer"
        ]
        if len(tool_uses) != 1:
            raise ValueError(f"Expected one submit_answer tool call, received {len(tool_uses)}")
        prediction = tool_uses[0]["input"]
        score = score_prediction(item, prediction)
        return {
            "id": item["id"],
            "source": item["source"],
            "type": item["type"],
            "mode": item["mode"],
            "gender": item["gender"],
            "prompt": item["prompt"],
            "gold": item.get("gold", item.get("gold_set")),
            "prediction": prediction,
            **score,
            "error": None,
            "provider": "anthropic",
            "requested_model": model,
            "response_model": response_data.get("model"),
            "response_id": response_data.get("id"),
            "reasoning_effort": reasoning_effort,
            "force_answer": force_answer,
            "web_search": False,
            "web_search_calls": 0,
            "citations": [],
            "usage": response_data.get("usage"),
            "latency_ms": round((time.monotonic() - started) * 1000),
        }
    except Exception as exc:
        return {
            "id": item["id"],
            "source": item["source"],
            "type": item["type"],
            "mode": item["mode"],
            "gender": item["gender"],
            "prompt": item["prompt"],
            "gold": item.get("gold", item.get("gold_set")),
            "prediction": None,
            "answered": 0,
            "correct": 0,
            "wrong": 0,
            "abstained": 0,
            "error": f"{type(exc).__name__}: {exc}",
            "provider": "anthropic",
            "requested_model": model,
            "response_model": None,
            "response_id": None,
            "reasoning_effort": reasoning_effort,
            "force_answer": force_answer,
            "web_search": False,
            "web_search_calls": 0,
            "citations": [],
            "usage": None,
            "latency_ms": round((time.monotonic() - started) * 1000),
        }


def load_jsonl(path: Path) -> list[dict[str, Any]]:
    with path.open(encoding="utf-8") as handle:
        return [json.loads(line) for line in handle if line.strip()]


def run(args: argparse.Namespace) -> None:
    dataset = load_jsonl(Path(args.dataset))
    if args.per_type_limit:
        dataset = [
            item
            for question_type in QUESTION_TYPES
            for item in [row for row in dataset if row["type"] == question_type][
                : args.per_type_limit
            ]
        ]
    elif args.limit:
        dataset = dataset[: args.limit]
    output = Path(args.output)
    output.parent.mkdir(parents=True, exist_ok=True)
    completed: dict[str, dict[str, Any]] = {}
    if output.exists():
        completed = {row["id"]: row for row in load_jsonl(output)}
    pending = [item for item in dataset if item["id"] not in completed]
    if not pending:
        print(f"No pending items; {len(completed)} already complete")
        return

    if args.provider == "anthropic" and args.web_search:
        raise RuntimeError("The Anthropic condition is recall-only; --web-search is unsupported")
    if args.force_answer and args.web_search:
        raise RuntimeError("--force-answer cannot be combined with --web-search")
    api_key_env = "ANTHROPIC_API_KEY" if args.provider == "anthropic" else "OPENAI_API_KEY"
    api_key = os.environ.get(api_key_env)
    if not api_key and args.api_key_file:
        api_key = Path(args.api_key_file).read_text(encoding="utf-8").strip()
    if not api_key:
        raise RuntimeError(f"Set {api_key_env} or pass --api-key-file")
    if args.provider == "anthropic":
        client: Any = httpx.Client(timeout=args.timeout)
        evaluate = evaluate_one_anthropic
    else:
        client = OpenAI(api_key=api_key, timeout=args.timeout)
        evaluate = evaluate_one

    try:
        with output.open("a", encoding="utf-8") as handle, ThreadPoolExecutor(
            max_workers=args.concurrency
        ) as executor:
            if args.provider == "anthropic":
                futures = {
                    executor.submit(
                        evaluate,
                        client,
                        api_key,
                        item,
                        args.model,
                        args.reasoning_effort,
                        args.force_answer,
                    ): item
                    for item in pending
                }
            else:
                futures = {
                    executor.submit(
                        evaluate,
                        client,
                        item,
                        args.model,
                        args.reasoning_effort,
                        args.web_search,
                        args.force_answer,
                    ): item
                    for item in pending
                }
            finished = 0
            for future in as_completed(futures):
                result = future.result()
                handle.write(stable_json(result) + "\n")
                handle.flush()
                finished += 1
                print(
                    f"[{finished}/{len(pending)}] {result['id']} "
                    f"answered={result['answered']} correct={result['correct']} error={bool(result['error'])}",
                    flush=True,
                )
    finally:
        if args.provider == "anthropic":
            client.close()


def wilson_interval(successes: int, trials: int, z: float = 1.959963984540054) -> list[float] | None:
    if trials == 0:
        return None
    proportion = successes / trials
    denominator = 1 + z * z / trials
    center = (proportion + z * z / (2 * trials)) / denominator
    spread = z * math.sqrt(proportion * (1 - proportion) / trials + z * z / (4 * trials * trials)) / denominator
    return [center - spread, center + spread]


def aggregate(rows: list[dict[str, Any]]) -> dict[str, Any]:
    n = len(rows)
    errors = sum(bool(row["error"]) for row in rows)
    answered = sum(row["answered"] for row in rows)
    correct = sum(row["correct"] for row in rows)
    wrong = sum(row["wrong"] for row in rows)
    abstained = sum(row["abstained"] for row in rows)
    latencies = [row["latency_ms"] for row in rows]
    return {
        "n": n,
        "answered": answered,
        "correct": correct,
        "wrong": wrong,
        "abstained": abstained,
        "errors": errors,
        "answer_rate": answered / n if n else 0,
        "accuracy_overall": correct / n if n else 0,
        "accuracy_overall_wilson_95": wilson_interval(correct, n),
        "accuracy_when_answered": correct / answered if answered else 0,
        "wrong_rate_when_answered": wrong / answered if answered else 0,
        "wrong_rate_when_answered_wilson_95": wilson_interval(wrong, answered),
        "web_search_calls": sum(row["web_search_calls"] for row in rows),
        "latency_ms_median": round(statistics.median(latencies)) if latencies else None,
        "latency_ms_p95": round(sorted(latencies)[max(0, math.ceil(len(latencies) * 0.95) - 1)]) if latencies else None,
    }


def summarize(args: argparse.Namespace) -> None:
    runs = []
    for run_spec in args.run:
        label, path_value = run_spec.split("=", 1)
        rows = load_jsonl(Path(path_value))
        by_type = {question_type: aggregate([row for row in rows if row["type"] == question_type]) for question_type in QUESTION_TYPES}
        by_gender = {gender: aggregate([row for row in rows if row["gender"] == gender]) for gender in ("female", "male")}
        runs.append(
            {
                "label": label,
                "path": path_value,
                "provider": rows[0].get("provider", "openai") if rows else None,
                "requested_model": rows[0]["requested_model"] if rows else None,
                "response_models": sorted({row["response_model"] for row in rows if row["response_model"]}),
                "reasoning_effort": rows[0]["reasoning_effort"] if rows else None,
                "force_answer": rows[0].get("force_answer", False) if rows else None,
                "web_search": rows[0]["web_search"] if rows else None,
                "overall": aggregate(rows),
                "by_type": by_type,
                "by_gender": by_gender,
            }
        )
    report = {"runs": runs}
    write_json_atomic(Path(args.output), report)
    print(json.dumps(report, indent=2, sort_keys=True))


def parser() -> argparse.ArgumentParser:
    root = argparse.ArgumentParser(description=__doc__)
    commands = root.add_subparsers(dest="command", required=True)

    prepare_parser = commands.add_parser("prepare", help="Build a balanced long-tail recall sample")
    prepare_parser.add_argument("--archive", required=True)
    prepare_parser.add_argument("--output", required=True)
    prepare_parser.add_argument("--cutoff", default="2026-02-16")
    prepare_parser.add_argument("--seed", type=int, default=20260726)
    prepare_parser.add_argument("--per-type-gender", type=int, default=5)
    prepare_parser.set_defaults(func=prepare)

    run_parser = commands.add_parser("run", help="Run one model condition")
    run_parser.add_argument("--dataset", required=True)
    run_parser.add_argument("--output", required=True)
    run_parser.add_argument("--provider", choices=("openai", "anthropic"), default="openai")
    run_parser.add_argument("--model", required=True)
    run_parser.add_argument("--reasoning-effort", default="low")
    run_parser.add_argument("--web-search", action="store_true")
    run_parser.add_argument("--force-answer", action="store_true")
    run_parser.add_argument("--api-key-file")
    run_parser.add_argument("--concurrency", type=int, default=6)
    run_parser.add_argument("--timeout", type=float, default=180)
    run_parser.add_argument("--limit", type=int)
    run_parser.add_argument("--per-type-limit", type=int)
    run_parser.set_defaults(func=run)

    summary_parser = commands.add_parser("summarize", help="Summarize completed model conditions")
    summary_parser.add_argument("--run", action="append", required=True, metavar="LABEL=PATH")
    summary_parser.add_argument("--output", required=True)
    summary_parser.set_defaults(func=summarize)

    return root


def main() -> None:
    args = parser().parse_args()
    args.func(args)


if __name__ == "__main__":
    main()

All 110 evaluation questions and reference answers

The wording, choices, IDs, and Cricsheet-derived references below are the exact fixed sample used in every condition.

Toss winner (10)

Who won the toss in the T20 match between United Arab Emirates and Bahrain on 2018-04-23 at Sulabiya Ground?

Options: United Arab Emirates; Bahrain

Reference answer: Bahrain

ID: 1142216.json#toss_winner
Who won the toss in the T20 match between Hong Kong and Oman on 2019-10-20 at Sheikh Zayed Stadium?

Options: Hong Kong; Oman

Reference answer: Oman

ID: 1199511.json#toss_winner
Who won the toss in the T20 match between Nepal and Qatar on 2021-11-16 at West End Park International Cricket Stadium, Doha?

Options: Nepal; Qatar

Reference answer: Qatar

ID: 1289284.json#toss_winner
Who won the toss in the T20 match between Malaysia and Qatar on 2024-02-11 at Selangor Turf Club, Kuala Lumpur?

Options: Malaysia; Qatar

Reference answer: Qatar

ID: 1419920.json#toss_winner
Who won the toss in the T20 match between Mozambique and Lesotho on 2024-04-26 at Botswana Cricket Association Oval 1, Gaborone?

Options: Mozambique; Lesotho

Reference answer: Mozambique

ID: 1429634.json#toss_winner
Who won the toss in the T20 match between Bahrain and Qatar on 2024-12-07 at West End Park International Cricket Stadium, Doha?

Options: Bahrain; Qatar

Reference answer: Qatar

ID: 1463010.json#toss_winner
Who won the toss in the T20 match between Hungary and Austria on 2025-02-04 at Marsa Sports Club?

Options: Hungary; Austria

Reference answer: Hungary

ID: 1470188.json#toss_winner
Who won the toss in the T20 match between Myanmar and Timor-Leste on 2025-11-14 at Udayana Cricket Ground?

Options: Myanmar; Timor-Leste

Reference answer: Myanmar

ID: 1508280.json#toss_winner
Who won the toss in the T20 match between Netherlands and Nepal on 2026-01-22 at Tribhuvan University International Cricket Ground, Kirtipur?

Options: Netherlands; Nepal

Reference answer: Netherlands

ID: 1515217.json#toss_winner
Who won the toss in the T20 match between Canada and Netherlands on 2015-07-17 at Goldenacre?

Options: Canada; Netherlands

Reference answer: Netherlands

ID: 875517.json#toss_winner

Toss decision (10)

What did the toss winner choose to do in the T20 match between China and South Korea on 2018-09-13 at Mission Road Ground, Mong Kok?

Options: bat; field

Reference answer: bat

ID: 1159087.json#toss_decision
What did the toss winner choose to do in the T20 match between Thailand and Hong Kong on 2019-02-22 at Asian Institute of Technology Ground?

Options: bat; field

Reference answer: bat

ID: 1172920.json#toss_decision
What did the toss winner choose to do in the T20 match between Netherlands and Scotland on 2019-06-27 at La Manga Club Top Ground?

Options: bat; field

Reference answer: field

ID: 1190607.json#toss_decision
What did the toss winner choose to do in the T20 match between Malaysia and Vanuatu on 2019-10-04 at Kinrara Academy Oval?

Options: bat; field

Reference answer: bat

ID: 1202011.json#toss_decision
What did the toss winner choose to do in the T20 match between Mozambique and Nigeria on 2022-12-01 at Gahanga International Cricket Stadium. Rwanda?

Options: bat; field

Reference answer: field

ID: 1343770.json#toss_decision
What did the toss winner choose to do in the T20 match between Hong Kong and Papua New Guinea on 2023-09-24 at Bayuemas Oval, Kuala Lumpur?

Options: bat; field

Reference answer: field

ID: 1398300.json#toss_decision
What did the toss winner choose to do in the T20 match between Namibia and Nigeria on 2023-12-10 at Entebbe Cricket Oval?

Options: bat; field

Reference answer: field

ID: 1411263.json#toss_decision
What did the toss winner choose to do in the T20 match between Brazil and Argentina on 2025-03-10 at St Albans Club, Buenos Aires?

Options: bat; field

Reference answer: field

ID: 1474982.json#toss_decision
What did the toss winner choose to do in the T20 match between Nigeria and Cameroon on 2025-06-07 at Gahanga B Ground, Rwanda?

Options: bat; field

Reference answer: bat

ID: 1487712.json#toss_decision
What did the toss winner choose to do in the T20 match between Singapore and Thailand on 2025-12-11 at Terdthai Cricket Ground, Bangkok?

Options: bat; field

Reference answer: field

ID: 1513828.json#toss_decision

Match winner (10)

Who won the T20 match between Denmark and Germany on 2019-06-19 at King George V Sports Ground?

Options: Denmark; Germany

Reference answer: Germany

ID: 1185193.json#match_winner
Who won the T20 match between Bermuda and Papua New Guinea on 2019-10-19 at ICC Academy?

Options: Bermuda; Papua New Guinea

Reference answer: Papua New Guinea

ID: 1199503.json#match_winner
Who won the T20 match between Bulgaria and Serbia on 2022-07-09 at Lisicji Jarak Cricket Ground?

Options: Bulgaria; Serbia

Reference answer: Bulgaria

ID: 1323552.json#match_winner
Who won the T20 match between Uganda and Tanzania on 2023-08-24 at Gahanga International Cricket Stadium, Rwanda?

Options: Uganda; Tanzania

Reference answer: Uganda

ID: 1393314.json#match_winner
Who won the T20 match between Indonesia and Cook Islands on 2023-09-04 at Vanuatu Cricket Ground?

Options: Indonesia; Cook Islands

Reference answer: Indonesia

ID: 1394801.json#match_winner
Who won the T20 match between Canada and Brazil on 2023-09-04 at Woodley Cricket Field, Los Angeles?

Options: Canada; Brazil

Reference answer: Canada

ID: 1395008.json#match_winner
Who won the T20 match between Hong Kong and Maldives on 2024-02-11 at Bayuemas Oval, Kuala Lumpur?

Options: Hong Kong; Maldives

Reference answer: Hong Kong

ID: 1419921.json#match_winner
Who won the T20 match between Uganda and Malawi on 2025-06-08 at Gahanga International Cricket Stadium, Rwanda?

Options: Uganda; Malawi

Reference answer: Uganda

ID: 1487713.json#match_winner
Who won the T20 match between Papua New Guinea and Samoa on 2025-10-09 at Al Amerat Cricket Ground Oman Cricket (Ministry Turf 1)?

Options: Papua New Guinea; Samoa

Reference answer: Samoa

ID: 1503455.json#match_winner
Who won the T20 match between Thailand and Namibia on 2025-11-23 at Asian Institute of Technology Ground, Bangkok?

Options: Thailand; Namibia

Reference answer: Thailand

ID: 1510296.json#match_winner

Victory margin — runs (10)

By how many runs was the T20 match between Oman and Germany on 2020-02-04 at Al Amerat Cricket Ground Oman Cricket (Ministry Turf 1) won?

Reference answer: 115

ID: 1214763.json#victory_margin_runs
By how many runs was the T20 match between Botswana and Mozambique on 2022-09-19 at Willowmoore Park, Benoni won?

Reference answer: 92

ID: 1332504.json#victory_margin_runs
By how many runs was the T20 match between Scotland and Netherlands on 2023-09-11 at Desert Springs Cricket Ground, Almeria won?

Reference answer: 59

ID: 1395035.json#victory_margin_runs
By how many runs was the T20 match between Namibia and United Arab Emirates on 2023-09-30 at Dubai International Cricket Stadium won?

Reference answer: 7

ID: 1400039.json#victory_margin_runs
By how many runs was the T20 match between Nepal and Namibia on 2024-03-01 at Tribhuvan University International Cricket Ground, Kirtipur won?

Reference answer: 3

ID: 1422040.json#victory_margin_runs
By how many runs was the T20 match between Myanmar and Kuwait on 2024-10-25 at Singapore National Cricket Ground won?

Reference answer: 11

ID: 1456125.json#victory_margin_runs
By how many runs was the T20 match between Bahrain and Cambodia on 2024-11-26 at University of Doha for Science and Technology won?

Reference answer: 49

ID: 1457233.json#victory_margin_runs
By how many runs was the T20 match between Nepal and Kuwait on 2025-10-08 at Al Amerat Cricket Ground Oman Cricket (Ministry Turf 1) won?

Reference answer: 58

ID: 1503452.json#victory_margin_runs
By how many runs was the T20 match between Malaysia and Nepal on 2025-10-02 at Bayuemas Oval, Kuala Lumpur won?

Reference answer: 3

ID: 1505114.json#victory_margin_runs
By how many runs was the T20 match between United Arab Emirates and Netherlands on 2016-02-03 at ICC Academy won?

Reference answer: 84

ID: 967081.json#victory_margin_runs

Victory margin — wickets (10)

By how many wickets was the T20 match between Mozambique and Botswana on 2018-08-21 at Botswana Cricket Association Oval 1 won?

Reference answer: 10

ID: 1158350.json#victory_margin_wickets
By how many wickets was the T20 match between Indonesia and Japan on 2018-12-05 at Friendship Oval won?

Reference answer: 4

ID: 1166887.json#victory_margin_wickets
By how many wickets was the T20 match between Vanuatu and Japan on 2019-05-10 at Ground 2, Independence Park won?

Reference answer: 9

ID: 1182917.json#victory_margin_wickets
By how many wickets was the T20 match between Cameroon and Sierra Leone on 2021-09-15 at Botswana Cricket Association Oval 2, Gaborone won?

Reference answer: 6

ID: 1275113.json#victory_margin_wickets
By how many wickets was the T20 match between Mexico and Argentina on 2023-10-19 at St Georges Quilmes won?

Reference answer: 4

ID: 1403301.json#victory_margin_wickets
By how many wickets was the T20 match between Nigeria and Namibia on 2023-11-30 at Wanderers Cricket Ground, Windhoek won?

Reference answer: 8

ID: 1407109.json#victory_margin_wickets
By how many wickets was the T20 match between Indonesia and Philippines on 2025-07-13 at Udayana Cricket Ground won?

Reference answer: 6

ID: 1490880.json#victory_margin_wickets
By how many wickets was the T20 match between Japan and Indonesia on 2025-09-15 at Albert Park 2, Suva won?

Reference answer: 8

ID: 1499802.json#victory_margin_wickets
By how many wickets was the T20 match between Austria and Luxembourg on 2025-09-14 at Pierre Werner Cricket Ground won?

Reference answer: 8

ID: 1500867.json#victory_margin_wickets
By how many wickets was the T20 match between Malawi and Nigeria on 2025-09-26 at Takashinga Sports Club, Highfield, Harare won?

Reference answer: 9

ID: 1502744.json#victory_margin_wickets

Team total (10)

How many runs did Spain score in the T20 match against Germany on 2021-09-10 at Desert Springs Cricket Ground, Almeria?

Reference answer: 128

ID: 1275052.json#team_total:Spain
How many runs did Greece score in the T20 match against Romania on 2023-08-05 at Moara Vlasiei Cricket Ground, Ilfov County?

Reference answer: 151

ID: 1387947.json#team_total:Greece
How many runs did Oman score in the T20 match against Japan on 2024-02-10 at YSD-UKM Cricket Oval, Bangi?

Reference answer: 116

ID: 1419911.json#team_total:Oman
How many runs did Spain score in the T20 match against Jersey on 2024-04-14 at La Manga Club Top Ground?

Reference answer: 145

ID: 1426480.json#team_total:Spain
How many runs did Mozambique score in the T20 match against Rwanda on 2024-04-22 at Botswana Cricket Association Oval 2, Gaborone?

Reference answer: 85

ID: 1429623.json#team_total:Mozambique
How many runs did Samoa score in the T20 match against France on 2025-03-12 at N'Du Stadium, Noumea, New Caledonia?

Reference answer: 33

ID: 1474426.json#team_total:Samoa
How many runs did Canada score in the T20 match against Bermuda on 2025-04-19 at Jimmy Powell Oval, Cayman Islands?

Reference answer: 108

ID: 1481304.json#team_total:Canada
How many runs did Bermuda score in the T20 match against Cayman Islands on 2025-06-21 at Maple Leaf North-West Ground, King City?

Reference answer: 139

ID: 1488335.json#team_total:Bermuda
How many runs did Singapore score in the T20 match against Cambodia on 2025-08-16 at Singapore National Cricket Ground?

Reference answer: 171

ID: 1498205.json#team_total:Singapore
How many runs did Bulgaria score in the T20 match against Cyprus on 2025-11-03 at Happy Valley Ground 2?

Reference answer: 125

ID: 1506447.json#team_total:Bulgaria

Top scorer — name (10)

Who scored the most runs in the T20 match between Argentina and Brazil on 2021-10-19 at Reforma Athletic Club, Naucalpan?

Options: A Stocks; AT Nascimento; C Sosa; CA Greloni; D Staddon; E de Souza; JC Cullen; L Agatha; L Cardoso; L Iglesias; L Moises; L Taylor; M Artur; M Castineiras; M Esquivel; M Martinez; M Ribeiro; NC Monteiro; RD de Sousa; RMAM Avery; T Basile; V Vasquez

Reference answer: L Agatha

ID: 1282929.json#top_scorer_name
Who scored the most runs in the T20 match between Scotland and Papua New Guinea on 2022-09-25 at Tolerance Oval, Abu Dhabi?

Options: A Aitken-Drummond; A Lister; AH Maqsood; B Tau; H Doriga; HRA Rainey; I Toua; K Arua; K Fraser; K McGill; KE Bryce; L Jack; M Tom; N Vare; P Siaka; PA Chatterji; R Oa; S Horley; S Jimmy; SJ Bryce; T Ruma; V Araa

Reference answer: KE Bryce or T Ruma

ID: 1331403.json#top_scorer_name
Who scored the most runs in the T20 match between France and Germany on 2023-06-02 at Grainville, St Saviour, Jersey?

Options: A Bester; A Bierwisch; A Doddaballapur; AL Seddon; CM Gough; DL Graham; EC Bargna; Emma Patel; Ganesh Pooja; IY Mckeon; K Vijayaraghavan; L Wykes-Templeman; ME Violleau; MS Beresford; PC McGeown; PKM Mahawattage; Rameesha Shahid; S Kolcharam; S Mannan; S Sadarangani; SR McAnanama-Brereton; TE Britton

Reference answer: PC McGeown

ID: 1378020.json#top_scorer_name
Who scored the most runs in the T20 match between Qatar and Bahrain on 2023-09-15 at West End Park International Cricket Stadium, Doha?

Options: Abdul Majid Abbasi; Adnan Mirza; Ahmer Bin Nisar; Ali Dawood; Bipin Kumar; Bukhar Illikkal; H Rathod; Haider Butt; I Liyanage; Imran Ali Butt; Imran Anwar; MM Baig; Mohsin Zaki; Muhammad Jabir; Muhammad Murad; Muhammad Tanveer; Rizwan Butt; S Veerapathiran; Saqlain Arshad; Umer Imtiaz; Uzair Amir; Yasser Nazir

Reference answer: Muhammad Tanveer

ID: 1392803.json#top_scorer_name
Who scored the most runs in the T20 match between Qatar and Kuwait on 2023-09-18 at West End Park International Cricket Stadium, Doha?

Options: Adnan Idrees; Adnan Mirza; Ahsan ul Haq; Bipin Kumar; Bukhar Illikkal; H Rathod; I Liyanage; Jassim Khan; MM Baig; MNM Aslam; Meet Bhavsar; Muhammad Jabir; Muhammad Murad; Muhammad Tanveer; Parvindar Kumar; R Sandaruwan; Saqlain Arshad; Sayed Monib; Shahrukh Quddus; Shiraz Khan; Usman Patel; Yasin Patel

Reference answer: I Liyanage

ID: 1392808.json#top_scorer_name
Who scored the most runs in the T20 match between Uganda and Botswana on 2024-12-04 at Gahanga International Cricket Stadium, Rwanda?

Options: AR Ramjani; B Kgosiemang; C Kyewuta; DM Nakrani; F Achelam; F Nsubuga; H Ssenyondo; J Miyaji; K Motlanka; K Piet; M Kasselman; M Mooketsi; R Nehonde; R Obuya; Raghav Dhawan; Riazat Ali Shah; SG Mangela; T Mpatane; T Perera; T Tshose; V Balakrishnan; V Mbazo

Reference answer: AR Ramjani

ID: 1462912.json#top_scorer_name
Who scored the most runs in the T20 match between Cameroon and Sierra Leone on 2025-06-05 at Gahanga B Ground, Rwanda?

Options: AK Bangura; AM Kamara; Aminata Kamara; B Waluma; BMA Mbida; C Bull; CSN Mbelel; E Guehoada; E Kamara; EGS Kana; F Conteh; F Parkinson; JAN Ngono; L Bull; M Turay; ME Bessala; MFD Douma; MLN Sissako; MSN Ekani; O Ranedoumoun; P Pratt; Z Kamara

Reference answer: C Bull

ID: 1487702.json#top_scorer_name
Who scored the most runs in the T20 match between Thailand and Scotland on 2025-11-15 at Terdthai Cricket Ground, Bangkok?

Options: A Lister; AH Maqsood; C Sutthiruang; DEM Carter; EL Watson; HRA Rainey; K Fraser; M McColl; N Boochatham; N Chaiwai; N Chantam; N Koncharoenkai; N Robertson-Jack; O Kamchomphu; P Maya; PA Chatterji; R Kanoh; R Slater; S Chaturongrattana; S Laomi; SJ Bryce; T Putthawong

Reference answer: N Koncharoenkai or SJ Bryce

ID: 1510282.json#top_scorer_name
Who scored the most runs in the T20 match between Scotland and Papua New Guinea on 2013-11-21 at ICC Academy?

Options: A Vala; CJA Amini; CN Kent; CR Amini; CS MacLeod; G Goudie; GO Jones; JB Reva; JNT Vare; K Pala; KJ Coetzer; MA Leask; MD Dai; MH Cross; MW Machan; NM Carter; RD Berrington; RM Haq; RML Taylor; SM Sharif; TP Ura; WT Gavera

Reference answer: CS MacLeod

ID: 660163.json#top_scorer_name
Who scored the most runs in the T20 match between Papua New Guinea and Scotland on 2013-11-29 at Sheikh Zayed Stadium Nursery 1?

Options: AM Hicks; CJA Amini; CN Kent; CR Amini; CS MacLeod; G Goudie; GO Jones; JNT Vare; KJ Coetzer; MA Leask; MD Dai; MH Cross; MW Machan; N Vanua; NM Carter; P Raho; RD Berrington; RM Haq; RML Taylor; SM Sharif; TP Ura; WT Gavera

Reference answer: RD Berrington

ID: 660225.json#top_scorer_name

Top scorer — runs (10)

How many runs did I Barma score in the T20 match between Bhutan and Nepal on 2021-11-23 at ICC Academy Ground No 2, Dubai?

Reference answer: 18

ID: 1286973.json#top_scorer_runs:I Barma
How many runs did A Lister score in the T20 match between Scotland and United Arab Emirates on 2022-09-23 at Tolerance Oval, Abu Dhabi?

Reference answer: 60

ID: 1331397.json#top_scorer_runs:A Lister
How many runs did TP Ura score in the T20 match between Japan and Papua New Guinea on 2023-07-29 at Amini Park, Port Moresby?

Reference answer: 47

ID: 1383100.json#top_scorer_runs:TP Ura
How many runs did RG Mukasa score in the T20 match between Namibia and Uganda on 2024-03-21 at Achimota Senior Secondary School A Field, Accra?

Reference answer: 36

ID: 1424764.json#top_scorer_runs:RG Mukasa
How many runs did N Caruana score in the T20 match between Gibraltar and Czech Republic on 2024-06-14 at Scott Page Field, Vinor?

Reference answer: 49

ID: 1438090.json#top_scorer_runs:N Caruana
How many runs did D Wheatley score in the T20 match between Bahamas and Panama on 2024-12-11 at Hurlingham Club Ground, Buenos Aires?

Reference answer: 30

ID: 1462893.json#top_scorer_runs:D Wheatley
How many runs did Habeeba Bader score in the T20 match between Brazil and Canada on 2025-03-13 at St Albans Club, Buenos Aires?

Reference answer: 21

ID: 1474986.json#top_scorer_runs:Habeeba Bader
How many runs did K Bhurtel score in the T20 match between Kuwait and Nepal on 2025-04-10 at Mission Road Ground, Mong Kok, Hong Kong?

Reference answer: 81

ID: 1479322.json#top_scorer_runs:K Bhurtel
How many runs did SJ Bryce score in the T20 match between Namibia and Scotland on 2025-11-25 at Asian Institute of Technology Ground, Bangkok?

Reference answer: 36

ID: 1510299.json#top_scorer_runs:SJ Bryce
How many runs did Y Jalloh score in the T20 match between Zambia and Sierra Leone on 2025-12-14 at Nigeria Cricket Federation Oval 1, Abuja?

Reference answer: 45

ID: 1513303.json#top_scorer_runs:Y Jalloh

Top wicket-taker — name (10)

Who took the most bowler-credited wickets in the T20 match between Japan and China on 2016-11-03 at Sano International Cricket Ground?

Options: Feng Yu; Han Junhui; J Yamashita; K Wakita; Lu Cangcang; M Kobayashi; M Taniyama; M Thurgate; MM Siddique; Pu Xianliang; R Matsumura; R Tsutsui; Song Yangyang; Song Yulin; T Hagihara; T Ota; Tian Suqing; W Miyauchi; Wang Yu; Wang Zihao; Xu Hao; Zhang Yu Fei

Reference answer: K Wakita

ID: 1063553.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Canada and Oman on 2019-10-25 at Sheikh Zayed Stadium?

Options: Aamir Kaleem; Abraash Khan; Aqib Ilyas; Bilal Khan; D Heyliger; Fayyaz Butt; GDR Eranga; Hamza Tariq; JOA Gordon; Jatinder Singh; Khawar Ali; Khurram Nawaz; Mohammad Nadeem; N Dutta; NR Kirton; NR Kumar; NS Dhaliwal; Ravinderpal Singh; SA Wijeratne; Sandeep Goud; Suraj Kumar; Zeeshan Maqsood

Reference answer: Zeeshan Maqsood

ID: 1199532.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Botswana and Eswatini on 2021-09-09 at Botswana Cricket Association Oval 1, Gaborone?

Options: A Mokgotlhe; B Madimabe; B Mpedi; D Dlamini; F Samanyika; J Kgang; K Maseko; L Mophakedi; M Dlamini; N Gwebu; N Khumalo; N Mabila; Ntombizodwa Mkhatshwa; Ntombizonke MKhatshwa; O Batisani; P Dlamini; S Mabuza; S Mosweu; T Legaabile; T Modise; T Shadrack; W Ginindza

Reference answer: B Mpedi

ID: 1275106.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Italy and Jersey on 2021-10-20 at Desert Springs Cricket Ground, Almeria?

Options: AM Tribe; Amir Sharif; B Ward; BBJL Perera; CW Perchard; D Birrell; DG Blampied; EJB Miles; G Stewart; GK Berg; GP Meade; H Carlyon; J Grassi; J Sumerauer; JE Dunford; JW Dernbach; JW Jenner; Jaspreet Singh; MGSM Fernando; Manpreet Singh; NL Smith; ZM Tribe

Reference answer: D Birrell or H Carlyon or J Sumerauer

ID: 1282746.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Cameroon and Mozambique on 2023-09-05 at Botswana Cricket Association Oval 1, Gaborone?

Options: A Salomao; AA Moiane; B Waluma; BMA Mbida; C Magaia; CD Mboe; CI Manidom; EGS Kana; FA Zavala; I Chuma; IE Mulhovo; II Mabunda; ME Bessala; MLN Sissako; MSN Ekani; O Matsolo; O Ranedoumoun; P Cuinica; RF Duvane; RO Mazumba; SA Akenji; TMA Leslie

Reference answer: C Magaia

ID: 1394925.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Rwanda and Kenya on 2024-09-11 at Sikh Union Club Ground, Nairobi?

Options: A Ikuzwe; A Wanjira; B Murekatete; C Muthoni; C Umutonni; C Uwase; E Wachira; F Odhiambo; G Ingabire; G Ishimwe; H Ishimwe; I Muhawenimana; J Ajiambo; JM Nthenya; JS Ndanu; KV Mehta; M Uwase; MD Bimenyimana; MI Khagoitsa; Q Abel; Rosine Irera; V Ooko

Reference answer: B Murekatete

ID: 1450463.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Singapore and Philippines on 2024-12-27 at Singapore National Cricket Ground?

Options: ABY Smith; AR Valdez; AS Miranda; Aarsheya Sharma; Angela Busa; DA Galia; Damini Ramesh; E Ungerman; H Dhavina; JKD Andreano; JR Medianesta; KK Donovan; Karri Gullem Keen; Kyte Gullem Keen; NL Jacinta Si Ping; R Castillo; R Gnanaraj; RR Seth; S Merican; SF Sirah; Shafina Mahesh; Vinu Kumar

Reference answer: R Castillo

ID: 1464732.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Germany and Sweden on 2025-05-28 at Roma Cricket Ground?

Options: A Kohli; A Vaidya; Abhilasha Singh; Ashwini Balaji; CM Gough; E Thelander; Eman Asim; G Shukla; H Chamto; H Shreya; I Edwards; JE Ronalds; Jayasooriya; K Vijayaraghavan; Kanchan Rana; N Kingsley; RG Somashekhar; Rameesha Shahid; S Kolcharam; S Sadarangani; W Hornero-Garcia; Zara Mohammad

Reference answer: Ashwini Balaji

ID: 1486098.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Bahamas and Canada on 2025-06-21 at Maple Leaf North-West Ground, King City?

Options: A Gulati; Ansh Patel; D Heyliger; DS Bajwa; E Duff; E Haven; Festus Benn; GT Taylor; H Thaker; J Gallimore; J Jemison; JR Barry; K Hinds; K Tathgur; Kaleem Sana; M Taylor; NR Kirton; Ravinderpal Singh; Saad Bin Zafar; Sandeep Goud; Shivam Sharma; YS Samra

Reference answer: Kaleem Sana or Shivam Sharma

ID: 1488334.json#top_wicket_taker_name
Who took the most bowler-credited wickets in the T20 match between Netherlands and Nepal on 2015-07-02 at Hazelaarweg?

Options: Ahsan Malik; B Bhandari; B Regmi; BN Cooper; G Malla; Karan KC; MB van Schelven; MP O'Dowd; P Khadka; PA van Meekeren; PM Seelaar; PS Airee; PW Borren; RI Ahmed; S Vesawkar; SJ Myburgh; SP Gauchan; SP Khakurel; Sagar Pun; Sompal Kami; T van der Gugten; W Barresi

Reference answer: Ahsan Malik

ID: 883345.json#top_wicket_taker_name

Top wicket-taker — wickets (10)

How many bowler-credited wickets did Janak Prakash take in the T20 match between Myanmar and Singapore on 2018-10-04 at Bayuemas Oval?

Reference answer: 3

ID: 1160950.json#top_wicket_taker_wickets:Janak Prakash
How many bowler-credited wickets did CR Seneviratna take in the T20 match between Malaysia and United Arab Emirates on 2019-02-21 at Terdthai Cricket Ground?

Reference answer: 3

ID: 1172916.json#top_wicket_taker_wickets:CR Seneviratna
How many bowler-credited wickets did S Nampiina take in the T20 match between Rwanda and Uganda on 2019-06-19 at Gahanga International Cricket Stadium. Rwanda?

Reference answer: 3

ID: 1188784.json#top_wicket_taker_wickets:S Nampiina
How many bowler-credited wickets did A Gbla take in the T20 match between Ghana and Sierra Leone on 2022-12-05 at Integrated Polytechnic Regional Centre?

Reference answer: 4

ID: 1343783.json#top_wicket_taker_wickets:A Gbla
How many bowler-credited wickets did R Trumpelmann take in the T20 match between Namibia and Nepal on 2024-02-27 at Tribhuvan University International Cricket Ground, Kirtipur?

Reference answer: 4

ID: 1422037.json#top_wicket_taker_wickets:R Trumpelmann
How many bowler-credited wickets did Imran Anwar take in the T20 match between Oman and Bahrain on 2024-12-17 at ICC Academy, Dubai?

Reference answer: 2

ID: 1463661.json#top_wicket_taker_wickets:Imran Anwar
How many bowler-credited wickets did L Hamilton take in the T20 match between Netherlands and United States of America on 2025-06-11 at Sportpark Maarschalkerweerd, Utrecht?

Reference answer: 3

ID: 1487653.json#top_wicket_taker_wickets:L Hamilton
How many bowler-credited wickets did Jai Sinh take in the T20 match between Switzerland and Estonia on 2025-08-03 at Estonian National Cricket and Rugby Field?

Reference answer: 4

ID: 1494775.json#top_wicket_taker_wickets:Jai Sinh
How many bowler-credited wickets did N Robertson-Jack take in the T20 match between Scotland and Papua New Guinea on 2025-11-14 at Terdthai Cricket Ground, Bangkok?

Reference answer: 3

ID: 1510279.json#top_wicket_taker_wickets:N Robertson-Jack
How many bowler-credited wickets did KLG Li take in the T20 match between South Korea and Hong Kong on 2014-09-22 at Yeonhui Cricket Ground?

Reference answer: 3

ID: 777999.json#top_wicket_taker_wickets:KLG Li

Total match runs (10)

How many total runs were scored by both teams in the T20 match between Rwanda and Tanzania on 2019-05-11 at Takashinga Sports Club?

Reference answer: 190

ID: 1182657.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Rwanda and Ghana on 2021-08-20 at Gahanga International Cricket Stadium. Rwanda?

Reference answer: 215

ID: 1273143.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Scotland and Malaysia on 2022-01-19 at Kinrara Academy Oval, Kuala Lumpur?

Reference answer: 265

ID: 1296030.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Canada and Philippines on 2022-02-18 at Al Amerat Cricket Ground Oman Cricket (Ministry Turf 2)?

Reference answer: 314

ID: 1299567.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Israel and Hungary on 2022-07-03 at Royal Brussels Cricket Club Ground, Waterloo?

Reference answer: 320

ID: 1320985.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Vanuatu and Cook Islands on 2022-09-15 at Independence Park, Port Vila?

Reference answer: 283

ID: 1333929.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Malaysia and Bahrain on 2023-09-03 at Bayuemas Oval, Kuala Lumpur?

Reference answer: 170

ID: 1394761.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Hong Kong and Nepal on 2024-03-09 at Mission Road Ground, Mong Kok, Hong Kong?

Reference answer: 351

ID: 1423439.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Bahrain and Qatar on 2024-12-04 at West End Park International Cricket Stadium, Doha?

Reference answer: 202

ID: 1463008.json#total_match_runs
How many total runs were scored by both teams in the T20 match between Kuwait and Bhutan on 2025-05-12 at Terdthai Cricket Ground, Bangkok?

Reference answer: 135

ID: 1483766.json#total_match_runs

Data attribution: this work derives match facts from Cricsheet's structured cricket data, using its JSON downloads. Cricsheet was created and is maintained by Stephen Rushe. We are grateful for the project's detailed, machine-readable scorecards.

Model selection and API configuration followed the official OpenAI model guidance, Anthropic model documentation, and Anthropic structured-output documentation.