Fabricated citations
A well-documented failure mode is the hallucination: an AI tool confidently produces case citations, quotations or holdings that look authentic but do not exist or are wrong. The fabricated material carries all the surface marks of real law, a plausible case name, a court, a year, a quotation in the right register, so nothing about it signals that it is invented.
They are convincing precisely because they look right. A reader scanning for the usual signs of authority finds them all present. The problem is not that the output looks suspicious and turns out to be false; it is that the output looks correct and is false anyway. That is what makes the error so easy to pass along.
This happens because of how the tools work. They generate text that fits the pattern of legal writing rather than retrieving and reporting verified records. When no real authority fits the pattern closely enough, the tool can produce one that fits the pattern but corresponds to nothing in the law reports.
Real consequences
This is not hypothetical. Courts have repeatedly sanctioned lawyers for filing briefs containing fake, AI-generated case citations. Judges who went to check the cited authorities found that they did not exist, and the lawyers who relied on them faced penalties and public embarrassment for putting fabricated law before the court.
The risk is even greater for self-represented people, who may have no way to spot the error. A lawyer at least knows how to verify a citation and is on notice that fabrications occur. A person handling their own matter may take a confident, well-formatted answer at face value and build their case on authority that was never real.
The harm is not limited to a single embarrassing moment. Arguments built on invented authority can collapse when tested, deadlines can pass while a person relies on the wrong law, and a court's trust in everything else the person submits can be shaken. The downstream cost of a single fabricated citation can be far larger than it first appears.
Safeguards
The defence is verification: check every authority against a primary source, keep a human in the loop, and be transparent about AI use. Before any citation is relied on or filed, it should be traced back to the actual decision or statute in an official or trusted source, not merely accepted because the tool produced it.
Keeping a human in the loop means that the person, not the software, remains responsible for what goes forward. The tool can draft and suggest, but a person checks, judges and decides. Transparency about AI use lets others, including a court, understand how the material was produced and apply appropriate scrutiny.
No output should be relied on without checking, a principle courts and bar bodies now emphasize. The discipline is simple to state and easy to skip under time pressure, which is exactly why it has to be treated as a fixed rule rather than an optional step.
What this is and is not
This is research and educational material on AI errors and hallucinations in legal work. It is general legal information, not legal advice, and it does not describe any individual's matter.