Reviewing a pull request that is too big for your AI reviewer
The problem
AI coding agents make it cheap to open a pull request that touches a hundred files, and the automated reviewers many teams rely on stop short at that size. CodeRabbit's plans page lists a cap on how many files it reviews in one pass, 150 on the Free and Essentials plans and 300 on Team and above, counted after path filters. Augment's legacy GitHub review bot posts a comment on very large pull requests saying the change is abnormally large and would use a lot of tokens, and it only reviews after someone opts in with a comment. In both cases the biggest pull requests, the ones that most need a careful read, get the least automated help.
Step 1: sort the change by product area
Before reading any diff, sort the changed files by the part of the product they belong to rather than by folder. A hundred files usually turn out to be a few real changes, such as one feature, a migration and many tests, and knowing that up front tells you how many reviewers you need and who they are.
Step 2: read the risky areas first
Start with the files that can hurt production, such as schema migrations, authentication, billing and anything that touches shared configuration. Generated files, lockfiles, fixtures and snapshots can usually wait until the end, and often only need a quick look to confirm they match the source change.
Step 3: review tests next to the code they cover
Tests that sit in a separate folder are easy to skim past, so pair each test file with the source file it exercises. If an area changed and has no matching test changes, leave a comment asking about it.
Step 4: split the review by area
Asking the author to split a large pull request is still good advice, but once the agent has written the code that is not always practical. Splitting the review by area gets you most of the benefit, because each reviewer reads one part of the change and signs off on it.
Step 5: point the AI review at the risky areas
If your reviewer asks for an opt-in or stops at a file cap, run it on the areas you flagged in step 2 rather than the whole change, which keeps the cost down and puts its attention where a mistake would hurt most.
Try it on a public pull request
StructPR groups the files in a pull request by business area and suggests a review order. You can paste any public pull request link into the preview tool and see the grouping without installing anything.
Related
Sources
- CodeRabbit plans and pricing, files per review: https://docs.coderabbit.ai/management/plans
- Augment large pull request opt-in comment, example: https://github.com/sourcemeta/core/pull/3113
- Augment migration from the GitHub review bot to Cosmos: https://docs.augmentcode.com/codereview/migrate-to-cosmos