Quick Take
- Fact: On Mar 12, 2024, Cognition introduced Devin as “the first AI software engineer,” reporting that it resolved 13.86% of real GitHub issues on the SWE-bench test with no help.
- Why it matters: Devin set the template for the autonomous agent pitch that now runs through most AI coding products.
- Who cares: Engineering leaders deciding how much work to hand to agents, developers worried about their jobs, and investors in coding tools.
- Judgment: Did not hold up as an autonomous engineer. It found a large market as a supervised helper, which is a different claim.
This is part of Hype Over Time, where we go back to a famous AI promise and check it against what actually happened.
The claim
Cognition’s launch post framed Devin as a teammate that could plan and finish whole engineering jobs. The headline number was SWE-bench: Devin resolved 13.86% of issues end to end, against a previous best of 1.96%. Demos showed it learning new tools, fixing bugs in open source code and completing freelance jobs.
The fine print mattered. Devin was tested on a random 25% subset of the benchmark, while some earlier systems were given the exact files to edit. And 13.86% means it failed about 86% of the tasks.
What the evidence shows
The most detailed outside test came from three researchers at Answer.AI, who used Devin for a month and published the results on Jan 8, 2025. Of 20 tasks, Devin succeeded at 3, failed 14 and left 3 inconclusive. They found it hard to predict which tasks would work, saw it chase dead ends for long stretches instead of saying it was stuck, and found it weakest on existing codebases.
The business story went the other way. Cognition announced a funding round in May 2026, and TechCrunch reported it raised $1 billion at a $25 billion pre-money valuation, with $492 million in annualized revenue run rate. Those revenue figures are company reported.
The pitch also changed. Days after the round, CEO Scott Wu told TechCrunch that Devin is not meant to replace programmers. He described it as a helper for tedious maintenance work, and put it somewhere between a junior and a mid-level engineer depending on the task. Cognition says most code committed by its own engineers now comes from Devin, a figure that has not been independently checked.
What is still open
There is still no independent, repeatable test of how much of a real team’s work Devin finishes without a person stepping in. Revenue shows companies find it useful. It does not show what share of engineering it actually replaces.
Bandwagon Check
Verdict: Did not hold up. “The first AI software engineer” promised an autonomous colleague. Two years later, the company’s own CEO describes a supervised helper, and the only detailed outside test found a 15% success rate. The product may well be worth its price. The launch framing was not accurate.
Who cares and what to do
If you run an engineering team, pilot coding agents on well defined maintenance tasks, track how often a person has to step in, and compare that against the cost. If you are a developer, the near term change is reviewing more machine-written code, not being replaced by it.
Sources
- Cognition: Introducing Devin, the first AI software engineer (Mar 12, 2024)
- Answer.AI: Thoughts on a month with Devin (Jan 8, 2025)
- Cognition: More Devins in More Places (funding announcement)
- TechCrunch: AI coding startup Cognition raises $1B at $25B pre-money valuation
- TechCrunch: Cognition’s Scott Wu says AI coding agents shouldn’t replace humans
By Willa Ravenwire · The AI Bandwagon
