How it works
Each scored conversation is compared pairwise against others and rated on an Elo scale, then mapped to a 0—100 score. The LLM judge evaluates each conversation as input to a hypothetical capable agent — “would a reasonable, well-tooled agent be able to act effectively on these prompts?” — and scores five dimensions before issuing a verdict:Goal clarity
Did the engineer state what they wanted in a way the AI could act on? Are prompts specific, unambiguous, and grounded in concrete examples or error messages — as opposed to vague requests like “make it better”?Constraint articulation
Did the engineer specify format, scope, audience, or success criteria? “Add a unit test that covers the empty-array case” beats “add tests”.Context sufficiency
Did the engineer give the AI enough background to do good work? Did they share the relevant code, file paths, error messages, or constraints, instead of expecting the AI to guess?Decomposition
Were complex asks broken into sensible steps, or thrown at the AI as a single overloaded request?Adaptive follow-through
When the AI’s response wasn’t quite right, did the engineer iterate constructively — building on what worked, giving specific feedback, and steering the conversation — rather than restarting from scratch or fighting the AI?Skill invocations
Slash-command skill invocations (e.g./pr-merge-ready, /code-review) are recognized as deliberate, high-intent workflows. The judge does not penalize them for being terse — a skill invocation is treated as a complete unit of intent, even if the prompt body is just the command itself.
What it doesn’t do (yet)
The prompt quality metric evaluates the prompting itself, not whether the conversation reached a successful outcome. The judge sees how each conversation ended but treats that only as weak evidence, so a well-prompted conversation that ultimately didn’t solve the problem can still score highly, and vice versa.Only meaningful conversations are scored: each must contain at least two
prompts, be classified by the system, and have a terminal status (completed,
abandoned, or blocked). Ongoing conversations are excluded.
Why it matters
Effective prompting directly impacts:- AI output quality: Better prompts produce better code suggestions
- Development velocity: Clear communication reduces iteration time
- Tool ROI: Maximizes value from AI tool subscriptions
- Learning curve: Identifies engineers who may benefit from prompting training
Improving your score
- Provide context upfront: Include relevant code, files, and error messages in your first prompt
- Be specific: Replace “fix the bug” with “fix the null pointer exception on line 42 in auth.ts”
- State the constraints: Say what “done” looks like — file, format, scope, success criteria
- Decompose big asks: Break multi-step work into separate prompts rather than overloading one
- Iterate constructively: When the AI’s response isn’t perfect, explain what’s wrong and what you need instead
- Stay focused: Keep conversations on-topic and complete one task before moving to another
- Learn from high performers: Review conversations from engineers with high quality scores on your team