Grading AI coding agents with hidden tests: Daniel vs Devin on 6 real bugs

AI coding agent demos are easy to make look good. I wanted to know something narrower: given a real bug report from a real project, does the agent fix it the way the maintainers did?

I build Daniel, an autonomous AI engineer that turns an issue into a tested pull request. So I ran it against Devin on the same bugs and let the projects' own tests decide.

The setup

I picked six recent bug fixes from widely used libraries:

For each one I:

1. Forked the project and made a bench branch at the commit just before the maintainers' fix.
2. Wrote one issue-style prompt describing the bug, with no hints about the fix.
3. Gave the identical prompt to both agents and told them not to read other branches or upstream history.
4. Graded each pull request by overwriting the test files with the maintainers' versions and running the project's own test command.

Neither agent ever saw those tests. A PR passes only if the code behaves the way the maintainers decided it should.

Results

Same score and about the same time; Daniel cost 70% less. Every pull request is public and linked from the full results page.

The interesting part is not the scoreboard. It's the two ways Daniel failed before it got there.

Failure 1: it tested every unit except the last one

The pretty-bytes bug says, in effect: any value that rounds up to 1000 of a unit must be shown as 1 of the next unit, for every unit from B up to YB.

Daniel wrote a reasonable fix and checked kB, MB, GB and a few more. It all passed. The maintainers' tests then checked every unit, and at the very top it failed:

- '0.999 YB'
+ '1 YB'

The cause was floating-point error, which grows at extreme magnitudes. After rounding pushed the value to the next unit, Daniel recomputed it from the raw input instead of converting the value it had already rounded. That is invisible at kB and wrong at YB.

The fix wasn't a patch for pretty-bytes. It was a rule in the engine: when a requirement is stated with "every", "all" or a range, the agent can't finish until it has run a check over every case in that range, including the very last one. Now the first time it tries to finish on such a task, it's sent back to prove the whole range.

Failure 2: the helpful reviewer

With that rule in place, Daniel's first commit passed all 19 hidden tests. Then its self-review step, which runs after the PR opens, flagged a "problem": the carry didn't happen when the caller passed fraction-digit options.

The issue never asked for that. The library deliberately doesn't carry in that case, and a hidden test checks exactly that. Daniel "fixed" it anyway and pushed a commit that broke the test.

So the second rule: the self-review may only demand changes for three reasons:

• the goal isn't met
• something that worked before is now broken
• it's unsafe

Anything else goes in the PR notes, not the code. The next run passed and stayed passing.

What I'd tell anyone evaluating coding agents

• Use hidden tests. If the agent can see the grading tests, you're measuring how well it fits them.
• Test the extremes. The first and last case are where "works for the common case" stops.
• Grade the final commit, not the first one. Review steps can make things worse.
• Watch scope. A reviewer that improves things nobody asked for is a regression generator.

Try it

Daniel runs in the cloud at cognitivedaniel.com, with a free trial and plans from $20/month. It also runs on your own machine:

npx github:2badagoldman/cognitive-daniel

I'd like to hear where it breaks. The method is simple enough to rerun on your own bugs: fork, branch before the fix, same prompt, grade with the maintainers' tests.