Code-QA-Bench data and reproduction
Inspect real tasks, schema, and release versions. Download the data, run a model with the public evaluation framework, and generate your own results.
- Public tasks
- 628
- Task files
- 2
- Data format
- JSON
- Reproduction status
- Partially reproducible
Tasks and framework are public; paper run outputs and judge records are not.
Download data
Every file comes from the pinned repository commit. This page does not repackage or modify task content.
Complete repository archive at pinned commit d615139.
The 528-task primary leaderboard set.
The 100-task diagnostic set for documentation utility.
Task preview
These specimens come from the public task files and expose the input, answer scope, and verification rubric.
- Task ID
- code-derivable:sympy_gen_03
- Repository
- sympy @ 693a559a
- Key files
- sympy/functions/elementary/_trigonometric_special.py · trigonometric.py · sympy/core/intfunc.py
Where does `ipartfrac` get called, and how does its data flow through `migcdex`, `igcdex`, and `gcdext`?
- Identify that ipartfrac is called in cos._eval_rewrite_as_sqrt, not directly in the sin implementation.
- Explain that ipartfrac uses functools.reduce with a multiplication function to compute the denominator product.
- Explain that complement values are constructed with denom // x and passed to migcdex.
- Task ID
- code-derivable:pytest_gen_04
- Repository
- pytest @ 8ecf49ec
- Key files
- src/_pytest/fixtures.py · src/_pytest/runner.py · src/_pytest/skipping.py
How does pytest handle finalizer failures during fixture teardown, and how are multiple exceptions from different teardown stages combined?
- Explain that FixtureDef.finish() runs all finalizers even when some fail and collects exceptions.
- Identify that finalizers are popped in LIFO order.
- Identify that multiple failures are wrapped in a BaseExceptionGroup with the exception list reversed.
- Task ID
- code-derivable:xarray_gen_01
- Repository
- xarray @ 92601de1
- Key files
- xarray/backends/common.py · xarray/core/utils.py · xarray/backends/api.py
What is the role and architecture of `_normalize_path` in xarray's backend system, and where is it used across the codebase?
- Describe the three overload signatures and the implementation accepting os.PathLike | str | T.
- Describe the os.fspath and local-path expansion transformations.
- Explain that is_remote_uri detects remote URIs and leaves them unchanged.
- Task ID
- code-derivable:django_gen_06
- Repository
- django @ 856c9153
- Key files
- django/contrib/gis/utils/ogrinspect.py · management/commands/ogrinspect.py · gdal/geomtype.py
Why is `ogrinspect` split into a public function and a private `_ogrinspect` generator, and how does the management command exploit that design?
- Explain that _ogrinspect yields model-definition lines while ogrinspect joins them into a string.
- Explain why the management command calls _ogrinspect directly before appending the mapping dictionary.
- Identify get_func_args(_ogrinspect) as the mechanism used to filter accepted CLI options.
Dataset schema
The primary and diagnostic task sets share the same core JSON fields.
idstringRequiredUnique task identifier within its task set.
repostringRequiredPinned source repository used by the task.
questionstringRequiredRepository-level code question presented to the evaluated model.
categoryenumRequiredQuestion category: what, why, where, or how.
sub_typestringRequiredMore specific capability type such as architecture_exploration.
gold_answerstringRequiredReference answer produced by the tool-equipped generation workflow.
rubricstring[]RequiredAtomic criteria used by the judge to score a model answer.
key_filesstring[]RequiredRepository paths supporting the reference answer.
source_docstringRequiredOriginal documentation excerpt used to establish the task topic.
verification_verdictenumRequiredAutomated audit verdict: pass, warn, or fail.
verification_issuesstring[]RequiredIssues and evidence boundaries found by automated verification.
strip_verify_leakagenumberOptionalEstimated leakage remaining after documentation stripping.
Reproduce the evaluation
Commands come from the repository README. Reruns require model and judge API credentials, and results may vary with endpoint versions.
01 Install
Create an environment and install the repository in editable mode.
uv venv source .venv/bin/activate uv pip install -e ".[dev]"02 Prepare repositories
Fetch pinned repositories from repos.json and build stripped variants.
code-qa-bench setup-repos code-qa-bench strip-repos03 Run a model
Run the primary task set in the Code-only condition; credentials live in config.json.
code-qa-bench run --model my-model --condition stripped04 Judge and report
Score saved answers with a judge and produce an aggregate report.
code-qa-bench judge results/my-model.json --judge-model judge-model code-qa-bench report results/my-model_judged_by_judge-model.json
Evidence and versions
Separates inspectable data, paper-reported results, and run-level evidence that is not yet public.
The repository includes 528 code-derivable and 100 doc-dependent tasks plus pinned commits for ten upstream repositories.
Scores and 95% CIs come from arXiv:2605.29277v1; intervals use 10,000 bootstrap resamples with seed 42.
The paper reports aggregate tables, but the reviewed public commit does not include task-level model outputs or judge results, so results are not marked fully verified.
Version 1 reports neither human validation nor inter-judge agreement; both remain explicit limits on formal comparison.