Code-QA-Bench
一个全自动仓库级代码理解基准。它用闭卷、仅代码和含文档三种条件,将真正的代码阅读能力与文档记忆、预训练记忆分开测量。论文在 10 个固定版本的 Python 仓库上发布了 528 个 code-derivable 任务和 100 个 doc-dependent 任务。
- 发布状态
- 已发布
- 能力域
- 代码与仓库 Agent
- 任务总数
- 628
- 已评模型
- 4
- 主指标
- 归一化均分 / 100
Leaderboard
默认展示论文主指标:528 个 code-derivable 任务在 code-only 条件下的归一化均分。图表和表格中的数值均来自论文 v1。
Tasks & Verification
这一节只回答两件事:模型接到什么任务,以及输出如何被判定;上方榜单是这套 Task × Verify 机制汇总后的得分。
628 个已公开任务
Code-derivable
528 tasks (84%)
Doc-dependent
100 tasks (16%)
528 个 code-derivable 主任务与 100 个 doc-dependent 诊断任务,来自 10 个固定 commit 的 Python 仓库。
模型答案与 gold answer / rubric 对照,Accuracy、Completeness、Specificity 各 0–5 分。
主榜单统计 528 个 code-derivable 任务的 code-only 均分;当前 4 个模型,范围 77.2–89.2。
回看榜单Where does `ipartfrac` get called, and how does its data flow through `migcdex`, `igcdex`, and `gcdext`?
Where does `ipartfrac` get called, and how does its data flow through `migcdex`, `igcdex`, and `gcdext`?
- 1Identify that ipartfrac is called in cos._eval_rewrite_as_sqrt, not directly in the sin implementation.
- 2Explain that ipartfrac uses functools.reduce with a multiplication function to compute the denominator product.
- 3Explain that complement values are constructed with denom // x and passed to migcdex.
Judge 将模型答案与 gold answer / rubric 对照,按 Accuracy、Completeness、Specificity 各 0–5 分评分;归一化后计入上方榜单。
How does pytest handle finalizer failures during fixture teardown, and how are multiple exceptions from different teardown stages combined?
How does pytest handle finalizer failures during fixture teardown, and how are multiple exceptions from different teardown stages combined?
- 1Explain that FixtureDef.finish() runs all finalizers even when some fail and collects exceptions.
- 2Identify that finalizers are popped in LIFO order.
- 3Identify that multiple failures are wrapped in a BaseExceptionGroup with the exception list reversed.
Judge 将模型答案与 gold answer / rubric 对照,按 Accuracy、Completeness、Specificity 各 0–5 分评分;归一化后计入上方榜单。
What is the role and architecture of `_normalize_path` in xarray's backend system, and where is it used across the codebase?
What is the role and architecture of `_normalize_path` in xarray's backend system, and where is it used across the codebase?
- 1Describe the three overload signatures and the implementation accepting os.PathLike | str | T.
- 2Describe the os.fspath and local-path expansion transformations.
- 3Explain that is_remote_uri detects remote URIs and leaves them unchanged.
Judge 将模型答案与 gold answer / rubric 对照,按 Accuracy、Completeness、Specificity 各 0–5 分评分;归一化后计入上方榜单。
Why is `ogrinspect` split into a public function and a private `_ogrinspect` generator, and how does the management command exploit that design?
Why is `ogrinspect` split into a public function and a private `_ogrinspect` generator, and how does the management command exploit that design?
- 1Explain that _ogrinspect yields model-definition lines while ogrinspect joins them into a string.
- 2Explain why the management command calls _ogrinspect directly before appending the mapping dictionary.
- 3Identify get_func_args(_ogrinspect) as the mechanism used to filter accepted CLI options.
Judge 将模型答案与 gold answer / rubric 对照,按 Accuracy、Completeness、Specificity 各 0–5 分评分;归一化后计入上方榜单。
Analysis
用分数分布和证据覆盖解释当前结果能说明什么,同时明确它还不能支持哪些结论。
三种实验条件
Code-derivable · 528 tasks
三种实验条件
Doc-dependent · 100 tasks
主任务集的类别得分
Code-derivable · 528 tasks
Data Card
用最少的信息说明数据集测什么、如何验证、如何计分以及能否复用;精确运行参数保留在技术记录中。
Code-QA-Bench
面向真实 Python 仓库的代码理解数据集,通过三种上下文条件区分预训练记忆、代码阅读和文档利用。
- 任务与规模
- 628 个仓库级问答任务
- 覆盖范围
- Python · 10 个公开仓库
- Verify 机制
- Gold answer + rubric
- 榜单指标
- Code-only mean · 0–100
- 公开许可
- 仓库 MIT · 论文 CC BY 4.0
- 当前版本
- Paper v1 · 2026-05-28
528 个 code-derivable 主任务 + 100 个 doc-dependent 诊断任务
每个仓库固定到可复现的 commit 快照
GPT-5.4 按 Accuracy、Completeness、Specificity 三轴评分
主榜单统计 528 个 code-derivable 任务的归一化均分
任务文件随公开仓库发布,未单列额外许可
Repository snapshot d615139
四个步骤概括任务如何生成、验证并进入榜单。
- 1
Answer first
从文档片段确定主题,工具型 agent 先探索真实代码并生成带文件与控制流证据的 gold answer,再派生问题和 rubric。
- 2
Code-only audit
使用 AST 和 tokenize 移除 docstring、注释与文档文件,并对 gold answer 做 code-only 可恢复性审核。
- 3
Three conditions
同一任务分别在 closed-book、code-only 和 documented 条件下运行,以拆分记忆、代码阅读与文档利用。
- 4
Three-axis judge
GPT-5.4 按 Accuracy、Completeness、Specificity 各 0-5 分评分;三轴归一化后按任务简单平均。