Daniel tops the hardest SWE-bench Verified issues
SWE-bench Verified is the industry's standard test for AI software engineers: real GitHub issues from major open-source projects, graded by tests the agent never sees. On the 24 hardest issues — each rated one to four hours or more for an experienced engineer — Daniel resolved 19 on its first and only attempt, more than any agent that publishes per-issue results on the same issues.
Hardest SWE-bench Verified issues resolved — first and only attempt.
Ahead of every public agent on the same issues; next best 18 (live-SWE-agent, Claude Opus 4.5).
Average model cost, about eight minutes per issue.
And head-to-head with Devin
We also put Daniel directly against Devin on six real bugs from popular open-source projects, with the same prompt, starting from the commit just before the maintainers' own fix. Each pull request was graded by the project's own regression tests, which neither agent saw. Every pull request is public.
Same score as Devin, graded by each project's own hidden tests.
Daniel $1.45 for six bugs vs Devin $4.91.
Total time from start to pull request across all six bugs: within 3%.
The six bugs, side by side
| Bug (real upstream issue) | Daniel | Devin |
|---|---|---|
| humanize python-humanize · intword() wrong for huge numbers (#346) | ✅ PR 1m53s · $0.25 | ✅ PR 1m54s · $0.74 |
| more-itertools more-itertools · bucket invents phantom keys (#1284) | ✅ PR 1m46s · $0.15 | ✅ PR 1m36s · $0.71 |
| mime-types jshttp (Express) · .mp4 resolves to the wrong type (#140) | ✅ PR 1m06s · $0.16 | ✅ PR 1m46s · $0.84 |
| pretty-bytes sindresorhus · rounding across unit boundaries (#91) | ✅ PR 2m06s · $0.33 | ✅ PR 3m00s · $0.93 |
| camelcase sindresorhus · acronym plurals like IDs (#95) | ✅ PR 3m16s · $0.37 | ✅ PR 2m39s · $0.95 |
| path-to-regexp pillarjs (Express) · empty parameter accepted (#444) | ✅ PR 2m26s · $0.19 | ✅ PR 1m17s · $0.74 |
| Total | 6 of 6 · 12m33s · $1.45 | 6 of 6 · 12m12s · $4.91 |
Times: both agents from start to pull request opened (Daniel then finishes its self-review on the open PR). Daniel ran three full rounds while we sped up its engine (round 1 total: 26m; round 3: 11m46s); the table shows round 3, and every round's pull requests are public in the same forks. Devin ran once. Costs: Daniel's model usage; Devin's per-session usage on the Free plan. pretty-bytes: in round 3 Daniel missed the final step to YB, a floating-point edge case (PR 4, 1m19s, failed). We changed the engine so that any requirement stated as "every" or a range is checked across the whole range before Daniel finishes, and so its self-review cannot add changes the request did not ask for. Round 4 passes all 19 hidden tests (PR 6); the table uses round 4's time to a passing fix and cost for that row and round 3 for the rest.
Method
- Picked recent, real bug fixes from widely used libraries: python-humanize, more-itertools, mime-types, pretty-bytes, camelcase and path-to-regexp.
- Forked each project and created a
benchbranch at the commit just before the maintainers' fix. - Gave both agents an identical issue-style prompt with no hints about the fix, and told them not to read other branches or upstream history.
- Graded each pull request by overwriting the test files with the maintainers' versions and running the project's own test command.
Why teams pick Daniel
Proven changes only
A task completes only when your tests pass. Unverified work is reported as blocked.
Your computer, too
Daniel runs locally on any folder with one command, with backups for every change.
Half the cost
Focused planning and a turbo loop spend less on every fix.