Quick Take
- Fact: When OpenAI launched GPT-4 on Mar 14, 2023, it said the model passed a simulated bar exam with a score around the top 10% of test takers.
- Why it matters: The bar exam line became shorthand for “AI is as good as a lawyer,” and it is still repeated in pitches and policy debates.
- Who cares: Lawyers and law firms buying AI tools, students, and anyone who reads benchmark percentiles in a launch post.
- Judgment: Did not hold up. GPT-4 likely passed, but the top 10% figure rested on a weak comparison group.
This is part of Hype Over Time, where we go back to a famous AI promise and check it against what actually happened.
The claim
OpenAI’s GPT-4 announcement highlighted exams. On a simulated Uniform Bar Exam, it said GPT-4 scored around the top 10% of test takers, while the older GPT-3.5 scored around the bottom 10%. The technical report listed a score of 298 out of 400. The bar result drew on a study by Daniel Martin Katz and colleagues, titled GPT-4 Passes the Bar Exam.
What the evidence shows
In 2024, MIT researcher Eric Martínez published a peer-reviewed re-evaluation in the journal Artificial Intelligence and Law. His central finding: the 90th percentile estimate compared GPT-4 with people taking the February Illinois exam, a group weighted toward repeat takers who had already failed once and scored lower than the general pool.
Against other comparison groups, the paper’s abstract reports much lower percentiles:
- Against a recent July exam: below the 69th percentile overall, and about the 48th on essays.
- Against first-time takers: about the 62nd percentile overall, and about the 42nd on essays.
- Against people who passed: about the 48th percentile overall, and about the 15th on essays.
Martínez also questioned how the essay sections were graded, since they were scored by comparison with sample answers rather than by trained bar graders. The original authors have defended their estimate as conservative.
One part of the claim holds. According to the exam body’s published minimums, passing scores range from 260 to 270, so a 298 would pass everywhere the Uniform Bar Exam is used. GPT-4 likely passed. It did not clearly beat nine out of 10 lawyers.
What is still open
Exact percentiles will stay fuzzy because the body that writes the exam does not publish full score distributions. Newer models have since been tested on legal tasks, but passing an exam and doing legal work are different tests. The more useful evidence now comes from audits of AI in real legal workflows, including how often tools invent case citations.
Bandwagon Check
Verdict: Did not hold up. “Passed the bar” was fair. “Top 10%” depended on who the model was compared against, and the comparison flattered it, especially on the written sections that look most like legal work. When a launch post gives a percentile, ask: a percentile of whom?
Who cares and what to do
If you are buying legal AI, ask vendors for results on your own document types and for error rates on citations, not exam scores. If you are a law student, the essay results are a reminder that clear legal writing is still where people beat the machine.
Sources
- OpenAI: GPT-4 research announcement (Mar 14, 2023)
- OpenAI: GPT-4 Technical Report (arXiv)
- Katz, Bommarito, Gao, Arredondo: GPT-4 Passes the Bar Exam (SSRN)
- Martínez: Re-evaluating GPT-4’s bar exam performance (Artificial Intelligence and Law, 2024)
- MIT DSpace: Re-evaluating GPT-4’s bar exam performance (open access copy)
- NCBE: UBE minimum scores by jurisdiction
By Willa Ravenwire · The AI Bandwagon
