StructPR StructPR ← Back to home
Shield methodology

How Shield decides which PRs deserve your attention

The literature-grounded scorer behind StructPR Shield, with every signal cited and every threshold justified. We publish this because Shield's value depends on you trusting how the verdict is computed — and we'd rather show our work than ask you to take it on faith.

Last updated: 2026-05-13 · Verdict version: v1.1

Why we publish this

A wrong likely_slop verdict on a real contributor's first PR is a brand-burning event in open source. We will not ship a public PR-side surface — labels, auto-close, comment trailers — until the verdict agrees with real maintainer decisions over a measurable validation period.

That trust isn't earned by a better LLM. It's earned by showing exactly what we look at, why we weight it the way we do, and what we deliberately don't trust as a signal. This page is that.

The three verdict classes

Every PR scored by Shield lands in one of three buckets:

needs_human
A real contribution worth a maintainer's review time. Open the PR, review normally.
vouching_required
New contributor + large diff or critical-config touch + low context. Ask the author for a discussion before reviewing.
likely_slop
Strong AI-tells + low reputation + low effort. Consider closing — but verify on the dashboard first.

The verdict ships with a confidence score from 0–1 and a list of human-readable reasons. Nothing is hidden behind a black box. A maintainer can read the reasons and override based on context we don't have.

The vouching_required class exists specifically to avoid forcing the system into a slop / not-slop binary on edge cases — it routes the borderline PR to a discussion, not a delete button.

The four signal families

Shield combines four kinds of signal, each derived from data GitHub already exposes. No model trained on your code. No diff content stored.

1. Reputation

How established is the author in this repo's ecosystem?

  • Account age, with a 30-day "new account" flag.
  • Prior accepted PRs in this repo.
  • Follower count as a soft tiebreaker.
  • Best-effort: when the GitHub API is unavailable, Shield degrades to conservative defaults rather than guessing.

2. Effort

Did the author actually invest work that suggests intent?

  • has_plan? — does the PR body contain an explicit plan or steps section? The single strongest protective signal we use. Per the MSR 2026 Circuit Breaker paper, PRs with stated plans are far less likely to be abandoned mid-review. The regex achieves 91% precision in the paper's evaluation.
  • Tests changed.
  • Linked issue (fixes #N, closes #N, resolves #N).
  • Conventional commit title.
  • Body length.

3. Structural complexity

What's the size and shape of the change? Per the literature, this is the strongest single predictor of review effort — AUC 0.96 on 33,707 agent-authored PRs.

  • Additions, deletions, total changes.
  • File count, large-diff and extra-large-diff flags.
  • Critical-config touches: lockfiles, CI workflows, dependencies, Kubernetes / Terraform manifests, vendored code.
  • Language diversity across the diff.

4. Heuristics (AI-tells — recorded, not weighted)

  • Co-authored-by: claude / chatgpt / copilot / cursor / devin trailers.
  • Generic titles (fix, update, wip).
  • Description-vs-diff mismatch.
  • Common AI-assistant template phrases.

Why these signals don't push toward likely_slop on their own: Per arXiv:2411.04299 and arXiv:2409.01382, general AI-code detectors generalize poorly across LLMs (Claude vs GPT vs Gemini all leave different fingerprints) and degrade rapidly on edited code. They also unfairly penalize honest contributors who disclose their use of AI. So Shield records these signals for transparency but weights them at zero in the slop scorer — a 🤖 Generated with Claude Code trailer alone does not produce a slop verdict.

What the literature says

Shield is built on real research, not invented rules. The three papers that shape the design:

Dao Minh et al., Early-Stage Prediction of Review Effort in AI-Generated Pull Requests, MSR 2026 (arXiv:2601.00753)

Analyses 33,707 agent-authored PRs. LightGBM on 35 structural features predicts "high-cost" PRs with AUC 0.958. Top features: additions, body_length, total_changes, has_plan. CodeBERT / text-only baselines hit AUC 0.52 — semantic analysis loses to structural footprint by 0.4 AUC. Shield uses this paper's feature set, not invented heuristics.

Suh et al., An Empirical Study on Automatically Detecting AI-Generated Source Code (arXiv:2411.04299)

State-of-the-art GPTSniffer reaches F1 ~82, but cross-generator generalization is poor. General-purpose AI detectors perform near chance on code. Why Shield's primary signal is review-effort prediction, not AI detection.

Rahman et al., Automatic Detection of LLM-Generated Code: A Comparative Case Study (arXiv:2409.01382)

Across GPT-3.5, Claude, and GPT-OSS, Comment-to-Code Ratio is the sole universal discriminator. Modern models sit at AUC 0.68–0.80; surface stylometric tells brittle across generators. Why we don't bet Shield's accuracy on AI-detection.

Khatoonabadi et al., Predicting the First Response Latency of Maintainers and Contributors in Pull Requests (arXiv:2311.07786)

CatBoost on 21 features predicts maintainer response latency. Top features: contributor acceptance rate, historical responsiveness, average commit count. Why Shield includes prior_prs_in_repo as a top reputation signal.

What Shield deliberately doesn't do (v1)

The v1 release is read-only and private by design. We will not expand the public surface until we can show — with numbers — that we've earned the right to.

  • We don't post anything on your PRs.
  • We don't apply GitHub labels.
  • We don't auto-close anything.
  • We don't notify the contributor.
  • We don't email digests.

The verdict is visible only to repo admins on a private dashboard at /shield/:owner/:repo. A contributor whose PR is scored likely_slop sees nothing different — their notifications, comments, and CI experience are unchanged. We trade speed for trust.

The staged plan

Each step gates on the previous one earning trust through real numbers.

v1.1 Calibration tool · shipped

Private dashboard. Read-only verdicts. No PR-side surface. Maintainers spend four weeks privately checking Shield's verdicts against the calls they'd have made anyway. This is the honest framing — v1 is for trust-building, not labor-saving.

v1.5 Advisory PR-side surface · gated

Append a "Shield Verdict" advisory section to the public PR comment. Per-repo confidence threshold knob. Conservative language. Gate: AUC ≥0.80 against maintainer close/merge decisions, false-positive likely_slop rate ≤2%, over a ≥4-week labeled window.

v2 Actions · gated on v1.5 trust

Optional GitHub labels (shield:likely-slop, shield:vouching-required). Optional auto-close on high-confidence slop, per-repo opt-in, never default. Cross-PR similarity detection (the "same fix from 12 burner accounts" case). Weekly digest email.

Targets we measure against

The numbers we publish per design-partner repo:

Metric Target Source
AUC vs maintainer decisions ≥0.80 Circuit Breaker hits 0.83 repo-disjoint
Precision @ 20% review budget ≥0.50 Tighter budget than Circuit Breaker
False-positive likely_slop on accepted PRs ≤2% Brand-burning event threshold
has_plan? precision ≥85% Circuit Breaker reports 91%

What we store

Per-PR metadata only: title, author handle, file paths, additions/deletions, verdict, signal flags. We do not persist diff contents. We do not train any model on your repo's code. Verdicts live as JSONB on the existing pr_analyses table and never leave our infrastructure. Self-hosted deployment is available on the Pro tier.

Sources

Want Shield running on your project?

Four design-partner slots this quarter. Free for verified OSS projects. No contract, no credit card, no SLA promises. Spend four weeks privately checking our verdicts against your own decisions. Walk if it doesn't help.

Apply as a design partner ← Back to StructPR