Daniel on the hardest SWE-bench Verified issues
SWE-bench Verified is the industry's standard test for AI software engineers: real GitHub issues from major open-source projects, graded by tests the agent never sees. We ran Daniel on 24 of its hardest issues, each rated one to four hours or more for an experienced engineer, and compared it with every agent that publishes per-issue results on the same issues.
19 of 24
Issues resolved by Daniel on its first and only attempt.
18 of 24
Best published agent on the same issues (live-SWE-agent).
3
Issues Daniel resolved that none of the 13 published agents did.
$0.29
Average model cost per issue. Median time: 5 minutes.
Same issues, published results
Each agent's result comes from its own submission to the public SWE-bench experiments repository, filtered to these 24 issues. Devin and Cursor do not publish per-issue SWE-bench Verified results, so they cannot be compared issue by issue.
| Agent | Model | Submitted | Resolved |
|---|---|---|---|
| Daniel | Claude Haiku, Sonnet and Opus, routed per step (standard mode) | Oct 2026 | 19 of 24 |
| live-SWE-agent | Claude Opus 4.5 | Dec 2025 | 18 of 24 |
| mini-SWE-agent | Gemini 3.5 Flash | Sep 2026 | 15 of 24 |
| mini-SWE-agent | Claude Opus 4.6 | Feb 2026 | 13 of 24 |
| Sonar Foundation Agent | Claude Opus 4.5 | Dec 2025 | 13 of 24 |
| mini-SWE-agent | GPT-5.2 (high) | Feb 2026 | 12 of 24 |
| OpenHands | Claude Opus 4.5 | Nov 2025 | 12 of 24 |
| Warp | Multiple (undisclosed) | Sep 2025 | 12 of 24 |
| Augment Agent v1 | Undisclosed | Jun 2025 | 12 of 24 |
| mini-SWE-agent | Claude Opus 4.5 (high) | Feb 2026 | 11 of 24 |
| TRAE | Undisclosed | May 2025 | 10 of 24 |
| Amazon Q Developer Agent | Undisclosed | Apr 2025 | 7 of 24 |
| Google Jules | Gemini 2.0 Flash | Dec 2024 | 2 of 24 |
| Factory Code Droid | Undisclosed | Jun 2024 | 2 of 24 |
Every issue
| Issue | Expert estimate | Daniel | Published agents that solved it |
|---|---|---|---|
| django__django-11138 TIME_ZONE value in DATABASES settings is not used when making dates timezone-aware on MySQL, SQLite, and Oracl | 1-4 hours | ✅ 1/1 target · 73/73 regression 4 min · $0.44 | 8 of 13 |
| django__django-11400 Ordering problem in admin.RelatedFieldListFilter and admin.RelatedOnlyFieldListFilter | 1-4 hours | ✅ 6/6 target · 58/58 regression 3 min · $0.17 | 0 of 13 |
| django__django-12708 Migration crashes deleting an index_together if there is a unique_together on the same fields | 1-4 hours | ✅ 1/1 target · 101/101 regression 2 min · $0.10 | 12 of 13 |
| django__django-13128 make temporal subtraction work without ExpressionWrapper | 1-4 hours | ✅ 7/7 target · 134/134 regression 16 min · $0.38 | 12 of 13 |
| django__django-13212 Make validators include the provided value in ValidationError | 1-4 hours | ❌ 3/5 target · 2/2 regression 4 min · $0.15 | 0 of 13 |
| django__django-13344 Coroutine passed to the first middleware's process_response() instead of HttpResponse. | 1-4 hours | ✅ 2/2 target · 356/356 regression 5 min · $0.14 | 3 of 13 |
| django__django-13837 Allow autoreloading of `python -m pkg_other_than_django runserver` | 1-4 hours | ✅ 1/1 target · 57/57 regression 2 min · $0.05 | 10 of 13 |
| django__django-14007 Database converters (from_db_value) not called for returning_fields on insert | 1-4 hours | ✅ 1/1 target · 12/12 regression 2 min · $0.22 | 12 of 13 |
| django__django-14011 LiveServerTestCase's ThreadedWSGIServer doesn't close database connections after each thread | 1-4 hours | ✅ 17/17 target · 2/2 regression 6 min · $0.40 | 0 of 13 |
| django__django-14631 BaseForm's _clean_fields() and changed_data should access values via BoundField | 1-4 hours | ✅ 2/2 target · 117/117 regression 2 min · $0.12 | 11 of 13 |
| django__django-15128 Query.change_aliases raises an AssertionError | 1-4 hours | ✅ 1/1 target · 282/282 regression 5 min · $0.23 | 10 of 13 |
| django__django-15503 has_key, has_keys, and has_any_keys JSONField() lookups don't handle numeric keys on SQLite, MySQL, and Oracle | 1-4 hours | ✅ 2/2 target · 78/78 regression 2 min · $0.24 | 7 of 13 |
| django__django-15629 Errors with db_collation – no propagation to foreignkeys | 1-4 hours | ✅ 2/2 target · 115/115 regression 18 min · $0.54 | 0 of 13 |
| django__django-16263 Strip unused annotations from count queries | 1-4 hours | ❌ 1/3 target · 100/100 regression 12 min · $0.41 | 0 of 13 |
| django__django-16560 Allow to customize the code attribute of ValidationError raised by BaseConstraint.validate | 1-4 hours | ❌ 8/8 target · 65/66 regression 3 min · $0.24 | 7 of 13 |
| pylint-dev__pylint-8898 bad-names-rgxs mangles regular expressions with commas | 1-4 hours | ✅ 1/1 target · 18/18 regression 1 min · $0.10 | 2 of 13 |
| pytest-dev__pytest-5787 exception serialization should include chained exceptions | 1-4 hours | ✅ 2/2 target · 123/123 regression 11 min · $0.28 | 8 of 13 |
| pytest-dev__pytest-6197 Regression in 5.2.3: pytest tries to collect random __init__.py files | 1-4 hours | ✅ 2/2 target · 145/145 regression 14 min · $0.26 | 9 of 13 |
| sphinx-doc__sphinx-8548 autodoc inherited-members won't work for inherited attributes (data members). | 1-4 hours | ✅ 1/1 target · 5/5 regression 5 min · $0.22 | 4 of 13 |
| sphinx-doc__sphinx-9461 Methods decorated with @classmethod and @property do not get documented. | 1-4 hours | ✅ 3/3 target · 59/59 regression 4 min · $0.26 | 1 of 13 |
| sympy__sympy-12489 combinatorics.Permutation can't be subclassed properly | 1-4 hours | ✅ 1/1 target · 8/8 regression 7 min · $0.16 | 8 of 13 |
| sympy__sympy-13878 Precompute the CDF of several distributions where integration doesn't work well | >4 hours | ✅ 1/1 target · 19/19 regression 58 min · $1.54 | 9 of 13 |
| sympy__sympy-14248 The difference of MatrixSymbols prints as a sum with (-1) coefficient | 1-4 hours | ❌ 1/2 target · 433/433 regression 5 min · $0.25 | 5 of 13 |
| sympy__sympy-16597 a.is_even does not imply a.is_finite | 1-4 hours | ❌ 0/3 target · 74/74 regression 9 min · $0.13 | 1 of 13 |
"Target" tests are the issue's FAIL_TO_PASS tests; "regression" tests are its PASS_TO_PASS tests. An issue counts as resolved only when all of both pass. Daniel's full patch for every issue, solved or not, is public: download all patches and grades (JSON).
Method
- Issues. 25 issues from SWE-bench Verified's two hardest difficulty groups ("1–4 hours" and ">4 hours"), across Django, SymPy, Sphinx, pytest, pylint and xarray. We picked issues that published agents often fail, so this set is harder than the benchmark as a whole.
- What Daniel saw. The repository at the issue's base commit and the issue text, nothing else: no hints, no hidden tests and no reference fix. Daniel ran in its standard mode with one attempt per issue, a 60-minute limit and a $6 budget.
- Environment. Each project's pinned Python environment from SWE-bench's own environment files, built in an isolated sandbox. Before grading Daniel, we applied the maintainers' reference fix to the same environment and confirmed it passes, so a broken setup can never count against Daniel.
- Grading. Exactly the SWE-bench rules: apply Daniel's patch, restore and apply the hidden tests, run them, and read the log with the official SWE-bench log parsers. Resolved means every target and regression test passes.
What to keep in mind
- The published agents ran between June 2024 and September 2026 on the models of their time. Daniel runs on newer Claude models, which accounts for part of the difference.
- The published results were graded with SWE-bench's Docker harness. We rebuilt the same environments in our own sandboxes and verified each one with the reference fix first.
- These issues are public, and agents in every run, including Daniel, had internet access.
- Excluded:
- pydata__xarray-3993: its environment could not be rebuilt faithfully outside SWE-bench's Docker image (the reference fix failed some regression tests), so it is not scored for any agent.