How it works
The review quality for a PR is evaluated by an LLM based on four criteria:Depth
Did the reviewer clearly go deep and understand the reviewed code at more than just a surface level? Are they giving suggestions on things like architecture or design, not just surface level code style? Are they exhibiting technical leadership?Quality of communication
Is the reviewer communicating well with the PR author? Are they being clear about what they’re suggesting, and why? Are they being helpful?Practicality
Is the reviewer being practical when suggesting changes? Are they allowing the PR to move forward if it’s an improvement, even if it’s not perfect; or are they being too much of a stickler for things that don’t matter as much?Impact
Did the feedback lead to meaningful code changes? Comparisons favor demonstrated improvements in correctness, security, performance, design, or maintainability. The value of a change matters more than its line count. Unrelated commits receive no credit, and missing or incomplete code evidence is treated as unknown rather than proof of no impact.How ratings are scored
Each review is rated by comparing it with other reviews, then scaled to a 0-100 score.- Approval-only reviews are excluded. Empty approvals and generic praise such as “LGTM” or “Looks good to me” receive no review-quality score, including no zero score. Approval followed by substantive feedback remains eligible. This exclusion applies only to review quality; review counts and cycles are unaffected. One short comment that catches a real issue can score well.
- Small PRs are excluded. Reviews on PRs with a code output under 1 don’t count toward the score, so a quick approval on a small change doesn’t lower it.
What it doesn’t do (yet)
The review quality metric does not actually review the entire PR. So it doesn’t catch errors of omission. For example, if the PR has a bug and the reviewer misses it, the review quality will not be affected. For a tool that can catch errors of omission like this automatically, we recommend Greptile.Self-reviews (where the PR author reviews their own PR) are excluded from this metric.