ReviewBench：GitHub 的 219 个 PR 代码评审基准，以及它背后 96.6% 的一致率

Demo 集不是基准

语料构建是最值得先抄的部分。GitHub 先量出 pull request 在语言、仓库规模和改动形态上的真实分布，再按这个分布采样基准。语言和仓库规模直接对齐 GitHub 整体；PR 的规模则被刻意向「可评审的中间段与长尾」加权，这一步剔掉了大量单文件一行改动——那种 diff 上任何评审器的表现都是满分——把权重压到真正需要评审的多文件改动上。

结果是 219 个 PR、187 个公开开源仓库、19 种语言，完整数据集公开。这个规模小到可以反复跑，又大到单个幸运或倒霉的 PR 不足以左右分数。加权这个决定本身也是一次表态：当问题变成「评审质量」时，完全代表性的样本反而是错的目标。

# Grounded precision and recall from a findings file: the strict, apples-to-apples pair.
import json
from collections import Counter

def grounded(findings, gold):
    '''findings: list of {id, pr_id}; gold: set of (pr_id, gold_id) tuples'''
    matched = {(f["pr_id"], f.get("gold_id")) for f in findings if f.get("gold_id")}
    found = matched & gold
    precision = len(found) / max(1, len(findings))
    recall = len(found) / max(1, len(gold))
    f1 = 2 * precision * recall / max(1e-9, precision + recall)
    return {"matched": len(found), "findings": len(findings), "gold": len(gold),
            "precision": round(precision, 3), "recall": round(recall, 3), "f1": round(f1, 3)}

GOLD = {("pr_17", "g1"), ("pr_17", "g2"), ("pr_42", "g3"), ("pr_88", "g4")}
FINDINGS = [
    {"pr_id": "pr_17", "gold_id": "g1"},
    {"pr_id": "pr_17", "gold_id": "g2"},
    {"pr_id": "pr_42", "gold_id": "g3"},
    {"pr_id": "pr_42", "gold_id": None},   # a real issue nobody in the gold set found
    {"pr_id": "pr_88", "gold_id": None},   # noise
]
print(grounded(FINDINGS, GOLD))   # recall 0.75, and the unmatched two decide augmented metrics

四路真值，一套细则

没有任何单一来源能把 diff 里值得发现的问题全部找出来。GitHub 从真实的人工评审、从作者后续提交里推断出的问题、确定性分析工具，以及来自不同模型家族的多个前沿模型那里收集候选发现，再做语义去重——既避免多个来源互相印证把黄金集吹大，也避免黄金集继承任何单一来源的盲区。

下一步才是它值得信任的原因。一条发现的来源不决定它算不算数。只有当它「真实、相关、非琐碎」时才算真阳性，判断依据是一套明确的细则，而细则连同裁判用的 prompt 与模型配置一起公开。这比业界常见做法（人工留下的任何评论都当黄金）门槛高得多。

发布前，未参与构建数据集的资深工程师从零重新标注了每一条真值，真假阳性判断与基准的一致率为 96.6%。数据集、裁判与匹配器都带版本号，意味着半年后有人能把结果复现出来，而不是只剩一段传闻。

# The golden set is deduplicated semantically, because four producers often spot one bug.
# A cheap stand-in: cluster findings by (file, overlapping line range) then by similarity.
def dedupe(findings, similarity=0.85):
    clusters = []
    for f in sorted(findings, key=lambda x: (x["file"], x["start"])):
        for c in clusters:
            same_file = c[0]["file"] == f["file"]
            overlap = min(c[0]["end"], f["end"]) - max(c[0]["start"], f["start"]) >= 0
            if same_file and overlap:
                c.append(f)
                break
        else:
            clusters.append([f])
    merged = []
    for c in clusters:
        sources = sorted({x["source"] for x in c})
        merged.append({"file": c[0]["file"], "line": c[0]["start"],
                       "sources": sources, "votes": len(sources),
                       "text": max((x["text"] for x in c), key=len)})
    return merged

RAW = [
    {"file": "app/auth.py", "start": 88, "end": 92, "source": "human", "text": "Token not verified before use"},
    {"file": "app/auth.py", "start": 89, "end": 95, "source": "static", "text": "Unverified JWT payload"},
    {"file": "app/auth.py", "start": 90, "end": 91, "source": "llm",    "text": "You should verify the signature"},
    {"file": "app/db.py",   "start": 12, "end": 14, "source": "llm",    "text": "N+1 query in loop"},
]
for row in dedupe(RAW):
    print(row["votes"], row["file"], row["line"], row["text"])

六个指标，而不是两个

大多数基准只报固定黄金集上的 precision 与 recall。ReviewBench 报两个家族。Grounded 的 precision、recall、F1 只用已有的黄金标签：已知的问题你找到了多少，你的发现里有多大比例命中了已知问题。Augmented 的 precision、recall、F1 还会评估那些在黄金集里找不到对应的发现，由裁判独立判断它们是真问题还是噪声。

第二个家族的存在理由很简单：固定黄金集在系统变强的那一刻就不完整了。如果你的评审器发现了一个真人、工具和模型都没提到的真 bug，只用 grounded 指标会把它记成噪声。augmented 指标让它算数。代价是 augmented recall 的分母会随各 Agent 发现的东西膨胀，所以 GitHub 把 grounded recall 作为跨系统对比的头号指标，augmented 作为单系统诊断。这个取舍很谨慎，值得抄。

# Six metrics, and the beta that decides whether you rank by noise or by coverage.
def fbeta(precision, recall, beta=1.0):
    b2 = beta * beta
    return (1 + b2) * precision * recall / max(1e-9, b2 * precision + recall)

AGENTS = {
    "reviewer_a": {"precision": 0.71, "recall": 0.44},
    "reviewer_b": {"precision": 0.52, "recall": 0.63},
    "reviewer_c": {"precision": 0.83, "recall": 0.31},
}
for beta in (0.5, 1.0, 2.0):
    ranked = sorted(AGENTS.items(), key=lambda kv: fbeta(kv[1]["precision"], kv[1]["recall"], beta), reverse=True)
    print(f"beta={beta}:", [(name, round(fbeta(v['precision'], v['recall'], beta), 3)) for name, v in ranked])

# beta < 1 favours precision (fewer nits), beta > 1 favours recall (broader coverage).
# ReviewBench re-ranks the leaderboard when this preference changes, which is the point.

每条发现都带严重程度与类别，排行榜因此能按你在乎的维度重排。

严重程度与类别切片会改变排名

每条发现都被标注 Critical、Medium、Low，以及类别：正确性、安全、可靠性、可维护性、测试等等。结果可以按这些维度切片，而且当你在 F-beta 里改 beta 时，排行榜会重新排序。beta 越低越偏 precision，越高越偏 recall。

这很重要，因为不存在普适的最优评审器。被评论淹掉的团队要 precision；给受监管客户交付的团队，宁可噪声多一些也要把安全问题捞全。单一数字会把这件事藏起来，而一个拒绝给出单一数字的基准，本身就在告诉你问题的形状。

# Severity slices carry information that the aggregate hides.
SLICES = {
    "critical": {"findings": 12, "matched": 11},
    "medium":   {"findings": 48, "matched": 31},
    "low":      {"findings": 96, "matched": 27},
}
total_findings = sum(v["findings"] for v in SLICES.values())
total_matched = sum(v["matched"] for v in SLICES.values())

for name, v in SLICES.items():
    share_of_hits = v["matched"] / total_matched
    share_of_noise = (v["findings"] - v["matched"]) / (total_findings - total_matched)
    print(f"{name:9} matched {v['matched']:3}  hit-share {share_of_hits:5.1%}  noise-share {share_of_noise:5.1%}")

# A reviewer whose hit-share on critical issues sits near its noise-share is not
# prioritising anything -- it is just producing volume.

让这个基准可信的那次实验

基准靠预测来挣信任。GitHub 用 ReviewBench 评估 Copilot code review，并描述了一次 lite 档实验：把多个独立模型的运行结果合成一次评审。ReviewBench 预测 precision、recall 和评论量都会上升，同时单次评审成本下降。

线上 A/B 走了同一个方向：与 precision 对应的「被采纳率」涨 8.0%，recall 涨 13.6%，评论量涨 61%，单次评审成本降 8.0%，全部相对对照组。按严重程度看，离线的临界问题评论预测增长 227%，线上实测 262%。幅度没有被精确预测，也不需要——方向和大致量级足够支撑「这个改动值得上 A/B」的判断。

读任何厂商的基准声明时都记住这一点，包括这一条。离线分数只是筛出「哪些改动值得上生产验证」的过滤器，不是验证本身。

# Offline movement is only useful if it tracks online movement. Keep both columns.
import statistics

EXPERIMENTS = [
    # name, offline_predicted_delta, offline_actual, online_delta
    ("ensemble_review", {"precision": +0.05, "recall": +0.10, "volume": +0.60, "cost": -0.05},
                        {"precision": +0.06, "recall": +0.12, "volume": +0.55, "cost": -0.07},
                        {"addressed_rate": +0.08, "recall": +0.136, "volume": +0.61, "cost": -0.08}),
]

def sign_agreement(predicted, observed):
    '''Do offline and online at least point the same way?'''
    pairs = [("precision", "addressed_rate"), ("recall", "recall"), ("volume", "volume"), ("cost", "cost")]
    agree = sum(1 for off, on in pairs if (predicted[off] > 0) == (observed[on] > 0))
    return {"agree": agree, "of": len(pairs), "ratio": agree / len(pairs)}

for name, pred, actual, online in EXPERIMENTS:
    print(name, sign_agreement(pred, online))
    drift = {k: round(actual[k] - pred[k], 3) for k in pred}
    print("  offline error:", drift)

# Published Copilot result for reference: addressed rate +8.0%, recall +13.6%,
# comment volume +61%, cost per review -8.0%. Direction matched, which is the claim.

离线预测方向、线上确认幅度——这才是值得抄的部分。

把你的评审器跑一遍

流程是自助的。用 GitHub 登录，注册你的 Agent：提供容器镜像、配置和你自己的模型 key，然后在 25 个 PR 的测试集上带着逐 PR 明细反复调；准备好之后，跑完整的 219 个 PR、三轮，同一个裁判按同一标准给你打分，再上排行榜。

就算你永远不提交，这套结构也值得搬回内部做评估：公开一份细则；让没参与构建系统的人标注 dev 集；用这批人的判断校准你的裁判，并把一致率数字报出来；给数据集、裁判、匹配器打版本号；最后把离线变化和线上结果放进同一张表里，这样两者不一致的时候，你先知道，而不是客户先知道。