Grading AI coding agents with hidden tests
Demos are easy to make look good. We wanted a narrower answer: given a real bug report from a real project, does the agent fix it the way the maintainers did?
This page describes how we test Daniel, what we found, and the two changes to the engine that came out of it. The method is simple enough for anyone to rerun on their own bugs.
Method
- Pick recent, real bug fixes from widely used libraries.
- Fork each project and create a branch at the commit just before the maintainers’ fix.
- Write one issue-style prompt that describes the bug with no hints about the fix, and give the identical prompt to each agent.
- Grade each pull request by replacing the test files with the maintainers’ versions and running the project’s own test command.
The agents never see the grading tests. A pull request passes only if the code behaves the way the maintainers decided it should.
Results
Daniel solved all six. Devin also solved all six, in about the same total time (12m 12s against Daniel’s 12m 33s), at $4.91 against Daniel’s $1.45.
| Project and bug | Hidden tests | Pull request | Time | Cost |
|---|---|---|---|---|
| python-humanize intword() shows “1000.0 sextillion” instead of carrying to the next unit | Passed | PR #4 | 1m 53s | $0.25 |
| more-itertools bucket invents keys when you look up a missing one | Passed | PR #4 | 1m 46s | $0.15 |
| mime-types (Express) .mp4 resolves to the wrong type | Passed | PR #5 | 1m 06s | $0.16 |
| pretty-bytes rounding doesn’t carry across unit boundaries | Passed | PR #6 | 2m 06s | $0.33 |
| camelcase acronym plurals like IDs come out wrong | Passed | PR #4 | 3m 16s | $0.37 |
| path-to-regexp (Express) empty string accepted for a required parameter | Passed | PR #4 | 2m 26s | $0.19 |
Finding 1: test the last case, not just the common ones
The pretty-bytes bug requires every unit from B to YB to carry correctly when a value rounds up to 1000. Daniel’s first fix was checked at kB, MB and GB and passed, but failed at YB, where floating-point error is largest: it printed 0.999 YB instead of 1 YB.
Change: when a requirement says “every”, “all” or names a range, Daniel can’t finish until it has checked every case in that range, including the last one.
Finding 2: a helpful reviewer can cause regressions
With the first change in place, Daniel’s fix passed all 19 hidden tests. Then its own review step flagged a “problem” outside the bug report, and the follow-up commit broke a test that checks that exact behavior on purpose.
Change: the self-review may only require changes when the goal isn’t met, something that worked before is now broken, or the change is unsafe. Everything else goes in the pull-request notes, not the code.
What we’d tell anyone evaluating coding agents
- Use hidden tests. If the agent can see the grading tests, you are measuring how well it fits them.
- Test the extremes. The first and last case are where “works for the common case” stops.
- Grade the final commit. Review steps can make things worse.
- Watch scope. A reviewer that improves things nobody asked for is a regression generator.