Code-QA-Bench

A fully automated repository-level code understanding benchmark. Closed-book, code-only, and documented conditions separate genuine code reading from documentation recall and pretraining memorization. The paper reports 528 code-derivable and 100 doc-dependent tasks across ten pinned Python repositories.

Release status: Released
Domain: Code & Repo Agents
Total tasks: 628
Models evaluated: 4
Primary metric: Normalized mean / 100

Leaderboard

The default view is the paper's primary metric: normalized mean score on 528 code-derivable tasks in the code-only condition. Every value shown comes from paper v1.

Metric

Rank	Model selection	Score	95% confidence interval	Tasks	Evidence
1	DeepSeek-V4-Prodeepseek-v4-0324	89.2	88.4–89.9	528	Partial ↗
2	Claude Opus 4.6claude-opus-4-6	89.1	88.5–89.7	528	Partial ↗
3	Kimi-K2.6kimi-k2.6-0528	87.3	86.7–88.0	528	Partial ↗
4	Gemini-3.1-Progemini-3.1-pro-preview	77.2	76.1–78.4	528	Partial ↗

Tasks & Verification

This section answers two questions: what the model receives, and how its answer is judged. The leaderboard above aggregates this Task × Verify contract.

628 public tasks

Code-derivable

528 tasks (84%)

Doc-dependent

100 tasks (16%)

01 / TASK 628 repository tasks

528 code-derivable primary tasks and 100 doc-dependent diagnostics across ten pinned Python repository commits.

02 / VERIFY GPT-5.4 · three-axis judge

Each answer is compared with the gold answer and rubric, then scored 0–5 on Accuracy, Completeness, and Specificity.

03 / SCORE 89.2 / 100 top score

The primary leaderboard averages code-only scores over 528 code-derivable tasks; 4 models currently span 77.2–89.2.

Back to leaderboard

Representative taskFrom public task files

Task inputPublic example

Where does `ipartfrac` get called, and how does its data flow through `migcdex`, `igcdex`, and `gcdext`?

Verify mechanism · Gold rubric

1Identify that ipartfrac is called in cos._eval_rewrite_as_sqrt, not directly in the sin implementation.
2Explain that ipartfrac uses functools.reduce with a multiplication function to compute the denominator product.
3Explain that complement values are constructed with denom // x and passed to migcdex.

The judge compares each model answer with the gold answer and rubric, scores Accuracy, Completeness, and Specificity from 0–5, then normalizes the result into the leaderboard above.

Analysis

Use score distribution and evidence coverage to show what current results support, and state what they cannot establish yet.

Three experimental conditions

Code-derivable · 528 tasks

Three experimental conditions

Doc-dependent · 100 tasks

Primary-set category scores

Code-derivable · 528 tasks

What current evidence supports

On the primary code-only metric, DeepSeek-V4-Pro (89.2) and Claude Opus 4.6 (89.1) are effectively tied, while Gemini-3.1-Pro scores 77.2.
Code access adds about 23 points over closed-book on average, indicating that performance mainly comes from active repository reading rather than memorization.
On doc-dependent tasks, documented beats code-only by 7.1 points on average, with the same direction across all four models.
On code-derivable tasks, documented and code-only differ by only about 0.6 points on average, matching the intended code-recoverability property.
Specificity is near ceiling with repository access; accuracy and completeness carry most of the remaining discrimination.

Limits and boundaries

Version 1 uses a single GPT-5.4 judge and reports no inter-judge agreement or human-correlation study.
Claude Opus 4.6 generated the tasks and is also evaluated, creating generator-evaluatee overlap that the design mitigates but does not remove.
There is no human validation. Of 528 code-derivable tasks, 203 pass automated verification and 325 retain a warn verdict for minor imprecision.
The scope is Python-only and uses single-commit snapshots; documentation removal may also affect tests that inspect __doc__.
The primary set already shows ceiling effects, with three models above 87% in code-only.

Data Card

A concise account of what the dataset measures, how it is verified and scored, and whether it can be reused. Exact run parameters live in the technical record.

Public sources

Tasks & scale: 628 repository-level QA tasks
Coverage: Python · 10 public repositories
Verification: Gold answer + rubric
Leaderboard metric: Code-only mean · 0–100
Public licenses: Repository MIT · paper CC BY 4.0
Current version: Paper v1 · 2026-05-28

Construction & scoring

Four steps summarize how tasks are built, verified, and turned into leaderboard scores.

1
Answer first
A documentation chunk sets the topic; a tool-equipped agent explores real source and writes a code-evidenced gold answer before deriving the question and rubric.
2
Code-only audit
AST and tokenize remove docstrings, comments, and documentation files, followed by an audit of whether each gold answer remains code-recoverable.
3
Three conditions
Each task runs closed-book, code-only, and documented to separate memorization, code reading, and documentation use.
4
Three-axis judge
GPT-5.4 scores Accuracy, Completeness, and Specificity from 0-5; normalized per-task scores are averaged equally.

Back to the Lab All benchmarks

Code-QA-Bench

Leaderboard

Tasks & Verification

Code-derivable

Doc-dependent

Where does `ipartfrac` get called, and how does its data flow through `migcdex`, `igcdex`, and `gcdext`?

How does pytest handle finalizer failures during fixture teardown, and how are multiple exceptions from different teardown stages combined?

What is the role and architecture of `_normalize_path` in xarray's backend system, and where is it used across the codebase?

Why is `ogrinspect` split into a public function and a private `_ogrinspect` generator, and how does the management command exploit that design?

Analysis

Three experimental conditions

Three experimental conditions

Primary-set category scores

Data Card

Code-QA-Bench

Answer first

Code-only audit

Three conditions

Three-axis judge