In June 2023, a lawyer in New York stood before a federal judge and cited six court decisions said to support his client’s case. There was just one difficulty: not one of them was real. He had asked ChatGPT to dig up precedent, and the model handed him six citations that were confident, properly formatted, and entirely made up, down to the invented quotes and docket numbers. The judge fined him $5,000. The story circled the globe as a cautionary tale.
Three years on, it has stopped being a cautionary tale. It has become a pattern. Damien Charlotin, a researcher at HEC Paris, keeps a public database of court cases in which someone filed AI-hallucinated material and was caught doing it. The list now runs to more than 1,700 decisions spread across dozens of countries, and it adds several more each week. The penalties have risen from that opening $5,000 fine into six figures, and a number of lawyers have been suspended as a result.
Here is the part that ought to trouble anyone in compliance. Lawyers get caught because a fake citation can be checked. Open the reporter, search the docket, and the fabrication comes apart within minutes. A compliance workflow seldom has that safety net wired in. When an AI tool declares a counterparty clean, or delivers a founder a risk score of 22 out of 100, hardly anyone pulls on the thread. The answer is simply filed and the team moves along. If it happened to be wrong, you usually learn as much later, once a regulator, a journalist, or a loss has found it on your behalf.
So the question sitting underneath “can I trust AI in compliance?” is not whether the model is clever. The models of today are extremely clever. The question is whether you are able to check its work, because in a regulated function an answer you cannot verify is not an asset. It is exposure with good grammar.
The problem isn’t that AI is wrong. It’s that it’s confident.
Large language models never actually look anything up. They predict text. You give the model a prompt, and it generates whatever continuation is statistically most likely, a result that comes out right most of the time and, once in a while, comes out wrong in the most fluent way imaginable. The industry has settled on the word hallucinations for these misfires, yet the label understates how dangerous they are. A hallucination arrives with no announcement. There is no warning attached to it and no reduced confidence score to tip you off. It looks exactly like a correct answer and speaks in the same calm, authoritative register as everything else the model produces.
In most contexts that is a mild inconvenience. In a regulated one it is a liability, and the clearest way to see why is to consider the two ways a screening tool can go wrong. A tool that conjures up a sanctions match will cost you an afternoon: an analyst chases the apparition, establishes that it is nothing, and closes it out. Frustrating, but plainly visible and self-correcting. A tool that conjures up a clean result behaves in the opposite way. It fails quietly. Nobody ever investigates a green light. The miss stays lodged in your file, looking for all the world like diligence, until the one day it finally does not.
Those failure modes map cleanly onto the work of compliance. Think of an adverse-media check that hangs a fraud case on the wrong person, someone who simply shares a name. Or a beneficial-ownership summary that sets out a structure with utter confidence and gets it wrong in some small but decisive way. Or a risk rationale that reads wonderfully and refers to nothing you can genuinely find. All of it looks like signal. None of it actually is.
And the trouble does not stop at consumer chatbots, whatever comfort vendors may try to offer on that score. A widely cited Stanford study tested purpose-built legal research tools, the ones sold specifically to professionals on the strength of a reliability promise, and still uncovered incorrect or misleading answers on a meaningful fraction of the queries. Pointing a model at a serious industry does nothing to switch hallucination off. All it really does is raise the cost of pretending the problem has been put to rest.
Two kinds of AI, and only one you can defend
The AI now entering compliance work generally comes in two varieties, and telling them apart matters more than any accuracy statistic on a glossy deck.
There is the black box. Hand it an entity and it produces a ruling, typically a clean number. Risk: 82. That number is generated by scoring logic sealed inside a model you cannot open, drawn from data hidden from you, and weighted according to factors the vendor itself often struggles to describe. While the score is accurate, it looks like something close to sorcery. Once it is off, you are left with nothing, because you cannot recognize the mistake and cannot dispute it, having no means to reconstruct how the tool arrived where it did. A conclusion you cannot reconstruct is a conclusion you cannot stand behind, either in front of your own committee or in front of anyone else.
Then there is the source-linked variety. In place of a ruling, it delivers a finding joined to the evidence that produced it. This person is listed on this sanctions register. This firm is cited in this filing. This claim originates in this news article, complete with a date and a link to the source itself. The AI continues to carry the load. It reads across hundreds of thousands of documents, strips out the clutter, works out which entity corresponds to which, and distills what is worth knowing. The crucial part is that every signal it brings forward points to material a human being can pull up and study directly.
The distinction isn’t cosmetic, and it shows up at exactly the moment that counts: when the AI is wrong, because both kinds will be wrong sometimes. With a black box, the error is invisible, folded into a number that looks as trustworthy as any other. With source-linked output, the error is just a link that doesn’t say what the tool claimed it said, and a reviewer catches it in under a minute. One system fails silently. The other fails safely. In a function whose entire job is catching what others miss, that gap isn’t a detail. It’s the whole design.
“Show your work” is becoming the law, not a nicety
For a long time, opacity was just part of the deal. You ran a model, you leaned on the output, and few people bothered to ask how the conclusion had been reached. Those days are ending fast.
The EU AI Act is a good place to see it. It reached general application in August 2026, and it starts from the premise that transparency is non-negotiable, then asks for more the higher the stakes climb. Its high-risk category pulls in uses such as credit scoring, and for those it requires a system transparent enough that the people relying on it can read its output correctly and use it as they should. Fall short and you are looking at fines in the tens of millions of euros. You can argue about when each specific provision bites, but you cannot really argue about the trajectory. Regulators have decided they want the reasoning.
And Europe is only one example of something that predates the current wave of AI anyway. The GDPR already lets individuals say no to decisions that are made purely by machine when the consequences for them are serious, and it lets them ask for meaningful information about the logic involved. Over in the United States, fair-lending rules have forced lenders to give a specific reason for every rejection going back decades, and “the algorithm decided” has never been on the list of acceptable ones. The same logic runs through anti-money laundering: the FATF standards behind AML regimes almost everywhere are really about keeping records and being able to back up your decisions, and a regulator such as FinCEN in the US expects firms to identify a risk, act on it, and keep documentation that holds together under scrutiny.
So you have a patchwork of regulators who came at this from different places, and they have all landed on the same expectation. If AI helped you decide, be ready to explain the decision and show the evidence for it. A black box cannot clear that bar. A source-linked system clears it without anyone having to do anything extra, since the evidence was there from the beginning.
The audit is where the black box dies
Set the theory aside and picture an ordinary Tuesday. A regulator, an internal auditor, or a nervous LP asks a simple question: why did you clear this particular company fourteen months ago? Since then it has surfaced in an unflattering investigation, and someone wants to know what you knew, and when you knew it.
If your tool was a black box, you have a number from over a year ago and very little to put around it. You can’t show the sources it weighed, because it never showed them to you. You can’t demonstrate that you did reasonable diligence, because the reasoning was never yours to see in the first place. You’re left trying to explain a decision you cannot reconstruct. That’s close to the worst position to occupy in an examination, and it’s a position you chose the day you bought the tool, whether you realised it or not.
If your tool was source-linked, you simply pull the record back up. Here were the signals as they stood at the time. Here are the sources underpinning each of them. This much was public then, and this other part only emerged later and could not reasonably have been on your radar. The decision holds up as defensible, because you are able to lay out the evidence it was built on. That is what separates a control you can genuinely stand behind from a guess you happened to trust one Tuesday over a year ago.
Auditability isn’t a feature you appreciate on a normal day. It sits in the background, quietly, adding nothing to the demo. It’s what you’ll wish you had on the one bad day, and by then it’s far too late to bolt on.
What trustworthy compliance AI actually looks like
You don’t have to take any vendor’s word for this, ours included. There’s a short, practical test you can run against any AI tool that touches a regulated decision, before you sign anything.
- Every finding links to a primary source you can open and read yourself. No link, no trust. This is the one that matters most, and the rest follow from it.
- The AI summarises and classifies, but it never states a fact it can’t point to. If it can’t cite it, it shouldn’t claim it.
- Entity matching is inspectable. You can see why the tool decided this “John Smith” is that one, instead of being asked to take the match on faith.
- Every decision leaves a timestamped record you can pull up months later, without a data-recovery project.
- A human stays in the loop. The tool’s job is to surface and evidence risk, not to pass final judgment on its own.
Notice that none of this is about making the AI more powerful. It’s about making it accountable. The industry spent a few years chasing models that sound more and more sure of themselves. What a compliance team actually needs is the opposite: an AI modest enough to show you where it got everything, so you can check it and own the decision yourself.
The honest part
It would be convenient to wrap up by declaring that source-linked AI never errs. But of course it does. A source can grow outdated. A registry can contain a mistaken entry. A reviewer can misjudge the very evidence placed before them. Joining every signal to its source does not amount to a perfect system, and any vendor holding out the promise of perfection is offering you exactly the overconfidence that put those lawyers in trouble at the outset.
The actual claim is a smaller one, and it stands up when it is tested. Verifiable AI fails in a way you are able to catch. Black-box AI fails in a way you are not. When a silent error can cost you a regulator, a headline, or a loss running into seven figures, being “catchable” is no longer a nice feature to have. It is the entire reason for doing any of this.
Frequently Asked Questions
So, can you trust AI in compliance?
Trust, in the end, is simply the wrong lens for this. There is no need to trust a tool that lays its work out in front of you, just as there is no need to trust a calculator whose answer you can work out by hand. You check it, it takes seconds, and you move on. That is exactly the possibility source-linking opens up, and exactly the one that black-box scoring takes away without a word.
Beady is built on that principle. It screens entities against more than 100,000 public sources, strips out roughly 95% of the noise, and links every signal back to its original source, so each sanctions and adverse-media finding is something your team, and a regulator, can verify independently. It summarises and classifies. It doesn’t invent.
If you’d like to see what verifiable risk intelligence looks like against your own book, book a demo and we’ll show you, with the sources attached.
