Hype Check

Devin Was Billed as the First AI Software Engineer. Two Years Later

By Willa Ravenwire · · 3 min

CHECK

Why it matters. Cognition launched Devin in 2024 as an autonomous engineer. An outside test found 3 wins in 20 tasks. The business boomed as a helper, not a…

Quick Take

  • Fact: On Mar 12, 2024, Cognition introduced Devin as “the first AI software engineer,” reporting that it resolved 13.86% of real GitHub issues on the SWE-bench test with no help.
  • Why it matters: Devin set the template for the autonomous agent pitch that now runs through most AI coding products.
  • Who cares: Engineering leaders deciding how much work to hand to agents, developers worried about their jobs, and investors in coding tools.
  • Judgment: Did not hold up as an autonomous engineer. It found a large market as a supervised helper, which is a different claim.

This is part of Hype Over Time, where we go back to a famous AI promise and check it against what actually happened.

The claim

Cognition’s launch post framed Devin as a teammate that could plan and finish whole engineering jobs. The headline number was SWE-bench: Devin resolved 13.86% of issues end to end, against a previous best of 1.96%. Demos showed it learning new tools, fixing bugs in open source code and completing freelance jobs.

The fine print mattered. Devin was tested on a random 25% subset of the benchmark, while some earlier systems were given the exact files to edit. And 13.86% means it failed about 86% of the tasks.

What the evidence shows

The most detailed outside test came from three researchers at Answer.AI, who used Devin for a month and published the results on Jan 8, 2025. Of 20 tasks, Devin succeeded at 3, failed 14 and left 3 inconclusive. They found it hard to predict which tasks would work, saw it chase dead ends for long stretches instead of saying it was stuck, and found it weakest on existing codebases.

The business story went the other way. Cognition announced a funding round in May 2026, and TechCrunch reported it raised $1 billion at a $25 billion pre-money valuation, with $492 million in annualized revenue run rate. Those revenue figures are company reported.

The pitch also changed. Days after the round, CEO Scott Wu told TechCrunch that Devin is not meant to replace programmers. He described it as a helper for tedious maintenance work, and put it somewhere between a junior and a mid-level engineer depending on the task. Cognition says most code committed by its own engineers now comes from Devin, a figure that has not been independently checked.

What is still open

There is still no independent, repeatable test of how much of a real team’s work Devin finishes without a person stepping in. Revenue shows companies find it useful. It does not show what share of engineering it actually replaces.

Bandwagon Check

Verdict: Did not hold up. “The first AI software engineer” promised an autonomous colleague. Two years later, the company’s own CEO describes a supervised helper, and the only detailed outside test found a 15% success rate. The product may well be worth its price. The launch framing was not accurate.

Who cares and what to do

If you run an engineering team, pilot coding agents on well defined maintenance tasks, track how often a person has to step in, and compare that against the cost. If you are a developer, the near term change is reviewing more machine-written code, not being replaced by it.

Share on X

Sources

  1. Cognition: Introducing Devin, the first AI software engineer (Mar 12, 2024)
  2. Answer.AI: Thoughts on a month with Devin (Jan 8, 2025)
  3. Cognition: More Devins in More Places (funding announcement)
  4. TechCrunch: AI coding startup Cognition raises $1B at $25B pre-money valuation
  5. TechCrunch: Cognition’s Scott Wu says AI coding agents shouldn’t replace humans

By Willa Ravenwire · The AI Bandwagon

Next from the wireGPT-4’s Top 10% Bar Exam Score, Checked Three Years Later

More in Hype Check

  1. Hype Check · 3 min

    GDPx: SemiAnalysis’s new expert-work benchmark tops out at 12%

    Opus 5.5 leads GDPx, but its 12% score needs context about task selection, safety filters and the benchmark's specialist partners.

    Nov 16, 2026

  2. Hype Check · 3 min

    Humane Bet on a Screenless Future. Its AI Pin Lasted 10 Months

    Humane raised over $230 million to replace screens with a wearable. The $699 AI Pin shipped in April 2024 and was switched off in February 2025.

    Nov 5, 2026

  3. Hype Check · 3 min

    GPT-4’s Top 10% Bar Exam Score, Checked Three Years Later

    OpenAI said GPT-4 scored around the top 10% on a simulated bar exam. A peer-reviewed recheck puts it near the 48th percentile among people who passed.

    Nov 5, 2026