Hype Check

GPT-4’s Top 10% Bar Exam Score, Checked Three Years Later

By Willa Ravenwire · · 3 min

CHECK

Why it matters. OpenAI said GPT-4 scored around the top 10% on a simulated bar exam. A peer-reviewed recheck puts it near the 48th percentile among people who passed.

Quick Take

  • Fact: When OpenAI launched GPT-4 on Mar 14, 2023, it said the model passed a simulated bar exam with a score around the top 10% of test takers.
  • Why it matters: The bar exam line became shorthand for “AI is as good as a lawyer,” and it is still repeated in pitches and policy debates.
  • Who cares: Lawyers and law firms buying AI tools, students, and anyone who reads benchmark percentiles in a launch post.
  • Judgment: Did not hold up. GPT-4 likely passed, but the top 10% figure rested on a weak comparison group.

This is part of Hype Over Time, where we go back to a famous AI promise and check it against what actually happened.

The claim

OpenAI’s GPT-4 announcement highlighted exams. On a simulated Uniform Bar Exam, it said GPT-4 scored around the top 10% of test takers, while the older GPT-3.5 scored around the bottom 10%. The technical report listed a score of 298 out of 400. The bar result drew on a study by Daniel Martin Katz and colleagues, titled GPT-4 Passes the Bar Exam.

What the evidence shows

In 2024, MIT researcher Eric Martínez published a peer-reviewed re-evaluation in the journal Artificial Intelligence and Law. His central finding: the 90th percentile estimate compared GPT-4 with people taking the February Illinois exam, a group weighted toward repeat takers who had already failed once and scored lower than the general pool.

Against other comparison groups, the paper’s abstract reports much lower percentiles:

  • Against a recent July exam: below the 69th percentile overall, and about the 48th on essays.
  • Against first-time takers: about the 62nd percentile overall, and about the 42nd on essays.
  • Against people who passed: about the 48th percentile overall, and about the 15th on essays.

Martínez also questioned how the essay sections were graded, since they were scored by comparison with sample answers rather than by trained bar graders. The original authors have defended their estimate as conservative.

One part of the claim holds. According to the exam body’s published minimums, passing scores range from 260 to 270, so a 298 would pass everywhere the Uniform Bar Exam is used. GPT-4 likely passed. It did not clearly beat nine out of 10 lawyers.

What is still open

Exact percentiles will stay fuzzy because the body that writes the exam does not publish full score distributions. Newer models have since been tested on legal tasks, but passing an exam and doing legal work are different tests. The more useful evidence now comes from audits of AI in real legal workflows, including how often tools invent case citations.

Bandwagon Check

Verdict: Did not hold up. “Passed the bar” was fair. “Top 10%” depended on who the model was compared against, and the comparison flattered it, especially on the written sections that look most like legal work. When a launch post gives a percentile, ask: a percentile of whom?

Who cares and what to do

If you are buying legal AI, ask vendors for results on your own document types and for error rates on citations, not exam scores. If you are a law student, the essay results are a reminder that clear legal writing is still where people beat the machine.

Share on X

Sources

  1. OpenAI: GPT-4 research announcement (Mar 14, 2023)
  2. OpenAI: GPT-4 Technical Report (arXiv)
  3. Katz, Bommarito, Gao, Arredondo: GPT-4 Passes the Bar Exam (SSRN)
  4. Martínez: Re-evaluating GPT-4’s bar exam performance (Artificial Intelligence and Law, 2024)
  5. MIT DSpace: Re-evaluating GPT-4’s bar exam performance (open access copy)
  6. NCBE: UBE minimum scores by jurisdiction

By Willa Ravenwire · The AI Bandwagon

Next from the wireHinton Said Stop Training Radiologists. Ten Years On, They’re Scarce

More in Hype Check

  1. Hype Check · 3 min

    GDPx: SemiAnalysis’s new expert-work benchmark tops out at 12%

    Opus 5.5 leads GDPx, but its 12% score needs context about task selection, safety filters and the benchmark's specialist partners.

    Nov 16, 2026

  2. Hype Check · 3 min

    Humane Bet on a Screenless Future. Its AI Pin Lasted 10 Months

    Humane raised over $230 million to replace screens with a wearable. The $699 AI Pin shipped in April 2024 and was switched off in February 2025.

    Nov 5, 2026

  3. Hype Check · 3 min

    Devin Was Billed as the First AI Software Engineer. Two Years Later

    Cognition launched Devin in 2024 as an autonomous engineer. An outside test found 3 wins in 20 tasks. The business boomed as a helper, not a…

    Nov 5, 2026