Start writing your article here.

On 17 September 2026 OpenAI released Astra for Law, its frontier model wrapped in a legal search index and a set of instructions for legal work, and did something vendors rarely do in a launch announcement. It printed a failure rate. On 200 US legal research questions from the private validation set of Vals AI's Legal Research Bench, the product passed the overall correctness check on 54% of them.

That number has travelled through the legal press as a triumph, a disappointment, and a marketing claim, depending on the reader. It is all three. This article sets out what the figure measures, what it does not, and the three questions any practice should put beside a vendor's benchmark before relying on the tool behind it.

• OpenAI reports that Astra for Law "passed the evaluation's overall correctness check on 54% of questions, compared with 38.7% for GPT-6 Astra using web search alone", on 200 questions from Vals AI's private validation set.
• The index behind it covers "more than 230 million URLs"; on case-law questions the tool found "24% more reference cases" and retrieved "up to 54% more relevant passages" than web search alone.
• The same figure says that a frontier model with a purpose-built legal index still failed the full correctness check on 46 questions in 100.
• Retrieval figures measure grounding, whether the right source was found. They do not measure correctness, whether the answer was right. The two fail separately.
• Three questions decide what a vendor's number is worth: who wrote the test, who marked it, and what counts as wrong.
• A published number is a starting point. The twenty questions from your own closed files are the finish.

Astra for Law, as OpenAI describes it and as LawNext reported on 17 September, is the GPT-6 Astra model combined with a legal search index of "more than 230 million URLs", instructions for legal analysis and writing, and controls for confidential work. It is US-law material, and the announcement says the product "will be available soon to API customers". No pricing was given, and nothing in the announcement covers UK material.

The claim that matters is one sentence:

"Astra for Law passed the evaluation's overall correctness check on 54% of questions, compared with 38.7% for GPT-6 Astra using web search alone."

The test set was "200 U.S. legal research questions from the private validation set of Vals AI's Legal Research Bench", run at the highest reasoning effort for both configurations. Two retrieval figures accompany it: on case-law-focused questions the tool "found 24% more reference cases than GPT-6 Astra using web search alone", and it "retrieved up to 54% more relevant passages from the correct court opinions".

📌 Key date: 17 September 2026, OpenAI's announcement. The benchmark was run by OpenAI on its own product, against a validation set OpenAI did not write. Both halves of that sentence matter, and the second is the more unusual.

From 38.7% to 54% is a relative improvement of about 40%. Adding a comprehensive primary-law index to a frontier model moved the correctness rate by fifteen points on a hard test. The index is doing real work, and the retrieval figures say the same thing from a different angle: more of the right cases found, more of the right passages pulled.

That is a legitimate reading and it is the one the announcement invites.

The same figure, read from the other end: a frontier model, with 230 million legal URLs behind it, configured by the company that built the model, at maximum reasoning effort, still failed the full correctness check on 46 questions in 100.

The figure criticises nobody. It is the most honest description of the state of the art that any vendor has offered the profession this year, and it came from the vendor. A practice deciding how much to rely on any AI research tool now has a published upper bound from the largest supplier in the market, and the bound is roughly one answer in two.

The third reading uses the vocabulary Stanford's RegLab introduced when it tested legal research tools in 2025 and published the results in the Journal of Empirical Legal Studies. An answer can fail in two distinct ways. It can be incorrect: the law stated is wrong. And it can be misgrounded: the law stated is right, or partly right, but the source cited does not support it, or is the wrong authority. A real citation for a claim it does not support counts as a failure, alongside a plain false statement.

Read in those terms, the two kinds of figure in OpenAI's announcement measure different things. "24% more reference cases found" and "54% more relevant passages retrieved" are grounding figures: the right source was located. "54% passed the correctness check" is a correctness figure: the answer was right. A tool can improve sharply on the first and modestly on the second, which is roughly what the announcement shows.

"Finding the right case is grounding. Grounding is one of the two ways an answer can fail, and it is the easier one to improve."

A benchmark is a set of questions and a scoring rule, and both belong to whoever built them. When a vendor reports a score on its own questions, the number describes a task the vendor chose. When it reports a score on a third party's private set, as OpenAI has done here, the number describes a task the vendor could not tailor. That is the first thing to establish, and OpenAI answers it plainly.

The second question is who applied the scoring rule to the answers. OpenAI ran this evaluation itself; no independent audit of the run has been published. A vendor-run score on an independent test sits between a vendor-run score on a vendor test and an independent evaluation, and it should be described that way.

The third question is the one a buyer most often forgets. "Passed the overall correctness check" means the answer satisfied whatever the benchmark's authors defined as correct. A benchmark that requires every required element to be present scores far more harshly than one that awards partial credit, and the same answer can pass one and fail the other. Before quoting a number, know which rule produced it. OpenAI's announcement leaves that to the benchmark's authors, which is the right place for it to sit, and a reason to read their documentation before relying on the figure.

Astra for Law is a US product on US material, tested on US legal research questions. Nothing in the announcement measures UK tax. A UK practice should take from it the shape of the state of the art, not a figure to apply to any tool it uses.

A published number, from any vendor, is where a purchasing conversation starts. It ends with twenty questions drawn from files your practice has already closed, with answers you know were right, run against the tool before it touches a client. The published score tells you whether a tool deserves the afternoon. The afternoon tells you whether it deserves the seat.

Our limitations and responsible use page says where we think the boundary of tool reliance sits, and our sources and update policy explains what stands behind each citation.

OpenAI has given the profession a rare thing: a large vendor's own failure rate, on a test it did not write, in its launch announcement. Read as the vendor reads it, the number shows a purpose-built index doing real work. Read as a buyer, it shows that the best-resourced configuration in the market fails a full correctness check on nearly half of hard questions. Read as an evaluator, it separates finding the source from getting the answer right, and shows the second lagging the first.

All three readings are true, and the third is the one to keep. Ask who wrote the test, who marked it and what counts as wrong, of every number you are shown, and then run your own twenty questions. For the wider criteria that sit around that test, see our guide to choosing AI tax research software in the UK. Firms that want to run the twenty questions against a UK tax tool can create a free account; practices evaluating across several seats can book a call.

That the product "passed the evaluation's overall correctness check on 54% of questions, compared with 38.7% for GPT-6 Astra using web search alone", on 200 US legal research questions from the private validation set of Vals AI's Legal Research Bench. OpenAI ran the evaluation itself.

Both readings are fair. It is a fifteen-point improvement from adding a purpose-built legal index to the same model, and it is a failure rate of 46 in 100 for the best-resourced configuration in the market on hard questions. Which reading matters depends on whether you are buying the improvement or relying on the answer.

A retrieval figure, such as "24% more reference cases found", measures whether the right source was located. A correctness figure measures whether the answer was right. In the vocabulary of the Stanford RegLab study, the first is grounding and the second is correctness, and an answer can fail either one independently.

No. Astra for Law is a US product tested on US legal research questions. The figure describes the state of the art in that setting; it is not a measurement of any UK tax tool, including ours.

Who wrote the test, who marked it, and what counts as wrong. A vendor score on the vendor's own questions, a vendor score on an independent set, and an independent evaluation are three different kinds of evidence, and the definition of "correct" decides how harsh the score is.

Treat it as a threshold for whether the tool deserves an afternoon, then spend the afternoon: twenty questions from closed files, answers written first, citations opened one by one, before the tool touches a client.