Law and The Machine

State v. Hallucination: When Prosecutors Cite Fake Law

March 31, 202618:41Law and The Machine

This episode explores a critical incident where a prosecutor used AI to generate legal citations in a murder appeal, leading to fabricated case law being presented to the state's highest court. Listeners will learn about the severe implications of AI hallucination in high-stakes legal contexts, highlighting the ethical and constitutional crisis that arises when the state relies on synthetic legal arguments. The discussion also uncovers systemic failures in legal offices' quality control and the crucial need for human oversight.

Key Takeaways

Detailed Report

{

"key_takeaways": [

"A Georgia prosecutor used AI to generate fabricated legal citations in a murder appeal, jeopardizing a woman's life sentence.",

"The incident exposed a systemic breakdown in the legal process, as both prosecution and defense failed to verify the AI-generated citations.",

"Many legal professionals misunderstand Large Language Models as factual databases, leading to 'hallucinations' of non-existent legal precedent.",

"Federal government procurement policies contradict public calls for AI safety by forcing vendors to disable ethical guardrails for government contracts.",

"Simple administrative bans on AI use in legal offices are ineffective without robust technological or human-in-the-loop verification workflows."

],

"detailed_report": "The highest court of a state recently uncovered a shocking breach of legal ethics: a prosecutor cited at least five non-existent cases and misrepresented several others in a murder appeal. This isn't a minor civil dispute; a woman's life sentence is at stake, and the state's argument for her continued incarceration was built, in part, on fabricated legal precedent.\n\n## The Case at the Center: Hannah Payne's Appeal\n\nThe incident revolves around the appeal of Hannah Payne, who was convicted in December 2023 of malice murder, felony murder, aggravated assault, and false imprisonment, receiving a life sentence plus 13 years. The charges stem from a 2019 incident where Payne pursued another driver, Kenneth Herring, after a minor car accident, despite 911 dispatchers advising against it. During a confrontation, Herring was shot and killed. Payne's defense argued the gun discharged during a struggle, while the prosecution portrayed her as a vigilante. The gravity of this case—a life sentence—underscores the profound implications of the state's reliance on fake law.\n\n## A Prosecutor's Admission and Systemic Failure\n\nClayton County Assistant District Attorney Deborah Leslie initially attempted to deflect blame for the fabricated citations, even suggesting the trial judge might have altered the document. However, Chief Justice Nels Peterson quickly established that the phantom cases were present in her original brief. Leslie's boss, DA Tasha Mosley, publicly contradicted her ADA, stating a \"clear policy that AI should not be used\" and emphasizing the fundamental rule: \"You don't lie to the court.\"\n\nFollowing a court order for an explanation, Leslie admitted under oath that she used AI for \"expanded legal research,\" and it \"hallucinated\" the case law. This isn't merely professional malpractice, as seen in civil cases; it's a constitutional crisis. Prosecutors represent the sovereign state and have a higher duty of candor to the court, obligated to seek justice, not just convictions. Leveraging the state's power to strip liberty using fabricated, machine-generated legal authority is a terrifying prospect.\n\n### Breakdown of the Adversarial System\n\nThe incident exposed a profound systemic failure. DA Mosley's \"no AI\" policy proved ineffective, highlighting that administrative policies alone are insufficient without robust technological or human-in-the-loop verification. In traditional law firms, junior associates or paralegals perform critical \"cite-checking\" to ensure cases are valid and support the propositions cited. The fact that a major appellate brief in a high-profile murder case reached the state supreme court without such verification points to a breakdown in fundamental quality assurance.\n\nAdding to the alarm, Hannah Payne's appellate attorney, Andrew Fleischman, admitted to the justices that he also failed to catch the fake cases in the state's brief. He stated he sometimes doesn't read all opposing briefs' cases, expressing regret for not bringing it to the court's attention. This compounds the problem: if the prosecutor hallucinates the law, the defense attorney doesn't read it, and the trial judge adopts the flawed order without independent verification, the entire adversarial system collapses.\n\n## The Misunderstanding of AI: A \"Category Error\"\n\nDespite numerous warnings and highly publicized sanctions, legal professionals continue to fall into this trap. The core issue is a \"category error\": lawyers treat Large Language Models (LLMs) like ChatGPT or Claude as if they are search engines or traditional legal databases. In reality, LLMs are probabilistic prediction engines that perform \"next-token prediction,\" calculating the statistical probability of which word should follow another based on their training data. They do not \"look up\" information.\n\nLegal citations have a highly structured format, making LLMs exceptionally good at predicting what a citation *should* look like. The AI will generate a plausible-sounding case name, volume number, and a perfectly formatted quote in convincing legalese. The AI does not *know* the case doesn't exist; it merely generates text that statistically matches patterns it has observed. When lawyers ask for case law, the machine counterfeits it; it generates, rather than retrieves, information. This distinction—a convincing storyteller with no concept of truth or falsehood, not a smart library—is crucial and often misunderstood.\n\n## Courts Grapple with AI Regulation\n\nThis isn't Georgia's first encounter with AI-generated fake law. Months prior, in October 2025, the *Atlanta Journal-Constitution* reported on Georgia courts sanctioning attorneys for AI misuse. Jonesboro attorney Loletha Hale was sanctioned by a U.S. District Judge for filing AI-generated fake arguments, ordered to notify clients and judges of her AI use for five years. Atlanta lawyer Diana Lynch was fined $2,500 by the Georgia Court of Appeals for bogus AI citations, and Brian Braddy was fined $1,000 for nine fake cases. Despite these highly publicized local incidents, a prosecutor in a neighboring county made the exact same mistake

Show Notes

Works Referenced

This episode was based on a research prompt rather than a single source URL. List the most relevant resources discovered during research, starting with the most important.

Then list any other articles, papers, reports, projects, companies, tools, standards, or resources that were mentioned in the episode or discovered during research. Format each as a bullet with a bolded name followed by a short description. Where a URL is known, make the name a clickable Markdown link: Name: one-sentence description. Only include items actually discussed or directly relevant to the episode — do not pad with tangentially related links.

Glossary

  • Hallucination: In the context of AI, this refers to when a Large Language Model (LLM) generates plausible-sounding but factually incorrect or fabricated information, such as non-existent legal cases.
  • Generative AI / Large Language Models (LLMs): A category of artificial intelligence that can generate new content, such as text, images, or code, by predicting the most statistically probable next element based on its training data.
  • Next-token prediction: The underlying mechanism by which LLMs operate, where they calculate the statistical probability of the next word or "token" in a sequence based on the preceding text and their vast training data.
  • Cite-checking: The standard legal practice of verifying that all case citations and legal propositions in a brief or document are accurate, valid, and support the claims made.
  • Adversarial system: The legal system structure, particularly in common law countries, where two opposing parties present their arguments, evidence, and legal interpretations before a neutral judge or jury to determine the truth.
  • Constitutional crisis: A situation where the fundamental principles or operations of a government's constitution are severely challenged or undermined, often leading to a breakdown of established legal and political norms.
  • Supply chain risk: A designation, typically used by governments, to label entities or technologies that pose a threat to the security or reliability of critical supply chains, often implying national security concerns.
  • Poison pill: A contractual clause or provision designed to make a deal or agreement unattractive or difficult to execute, often by imposing severe penalties or restrictions.
  • Continuing Legal Education (CLE): Mandatory ongoing education requirements for licensed attorneys to maintain their legal knowledge and skills, often including specific topics like ethics or technology.

Full Transcript

HostImagine a scenario playing out in the highest court of a state. The Chief Justice stops proceedings, looks directly at a prosecutor, and says, "There are at least five citations to cases that don't exist. And there's at least five more citations to cases that do not support the proposition for which they're cited, including three quotations that don't exist."
ExpertAnd this isn't some minor traffic infraction or a civil dispute over a parking ticket. This is a murder appeal. A woman's life in prison is on the line, and the state's argument for keeping her there is built, in part, on fabricated legal precedent.
HostIt gets wilder. The prosecutor, a Clayton County Assistant District Attorney named Deborah Leslie, initially tries to deflect blame, even suggesting the trial judge might have altered the document. But the Chief Justice, Nels Peterson, quickly shuts that down, pointing out the phantom cases were in *her* original brief.
ExpertWhich, of course, led to her boss, DA Tasha Mosley, publicly contradicting her own ADA, stating there was a "clear policy that AI should not be used" and reiterating the fundamental rule: "You don't lie to the court." And then, an order from the court demanding an explanation, leading to Leslie's sworn admission: she used AI for "expanded legal research," and it hallucinated the case law.
HostThis isn't just an embarrassing gaffe, is it? This is a fundamental breakdown of the legal system at a critical juncture.
ExpertIt's an absolute crisis.
HostThis all centers on the appeal of Hannah Payne. Can you briefly walk us through the gravity of that case?
ExpertAbsolutely. This isn't a low-stakes civil matter; it's literally about a person's life and liberty. Hannah Payne was convicted in December 2023 of malice murder, felony murder, aggravated assault, and false imprisonment. She was sentenced to life in prison, plus 13 years. The charges stem from a truly tragic incident in 2019 where she witnessed a minor car accident, called 911, and despite dispatchers telling her not to pursue, she chased the driver, Kenneth Herring, for about a mile. She used her car to block his, approached him with a gun, and during a struggle, Herring was shot and killed. The prosecution painted her as a vigilante. Her defense argued Herring grabbed her and the gun went off. So, what we're talking about here is a life sentence, an appeal, and the state's duty to uphold justice.
HostAnd in that context, the state's attorney used fake law to argue for keeping her in prison. That shifts this from professional malpractice, which we've seen in civil cases, to something far more severe.
ExpertPrecisely. When a private attorney uses generative AI to draft a personal injury brief and it hallucinates, that's professional malpractice. You might get sanctioned, face disciplinary action, perhaps even lose your license. But when a prosecutor does it, especially in a case with such high stakes, it veers into constitutional crisis territory. Prosecutors aren't just representing a client; they represent the sovereign state. They have a higher duty of candor to the court and a constitutional obligation to seek justice, not just convictions. The state wields the power to strip a citizen of their liberty, even their life. To have that power leveraged using fabricated, machine-generated legal authority? It's terrifying. It's the state using synthetic, hallucinated realities to execute its police powers.
HostAnd the District Attorney, Tasha Mosley, defended her office by saying there was a "clear policy that AI should not be used." But that policy clearly didn't work.
ExpertAnd that's where the rubber meets the road for me, from a systems and technology perspective. An administrative policy, on its own, without a robust technological or human-in-the-loop workflow, is just a piece of paper. How does a major appellate brief in a high-profile murder case, one arguing before the *state supreme court*, make it out of a District Attorney's office without a single human being checking the citations?
HostYou'd think that's standard procedure.
ExpertIt *is* standard procedure. In traditional law firms, junior associates or paralegals have the critical job of "cite-checking." They use databases like Westlaw or LexisNexis to ensure every case cited is not only valid but actually supports the proposition it's claimed to support. The fact that the Clayton County DA's office allowed this to happen points to a systemic failure of institutional rigorousness. It's not just an individual ADA using AI; it's a breakdown in fundamental quality assurance.
HostAnd the adversarial system, which is supposed to be the bedrock of our justice, is built on the idea that two sides fiercely contesting each other's claims will uncover the truth. But in this case, both sides seemed to miss it.
ExpertThat's perhaps the most shocking admission from the Georgia Supreme Court hearing. Hannah Payne's appellate attorney, Andrew Fleischman, actually admitted to the justices that *he also failed to catch the fake cases* in the state's brief. He said, and I'm quoting here, "I sometimes don't read all of the cases that my opponents brief and the sentences where I didn't do that. I regret that I did not bring this to the court's attention."
HostWait, really? The defense attorney, whose job it is to scrutinize every argument made by the prosecution against his client's freedom, didn't read the cases the prosecution cited?
ExpertThat's what he said, on the record. And it compounds the problem. If the prosecutor hallucinates the law, the defense attorney doesn't read the law, and then, as the source material points out, the trial judge, Judge Jewel Scott, *had apparently adopted the prosecution's drafted order—complete with the hallucinated citations—without independent verification*, then the entire adversarial system collapses. It's an utter failure at every level.
HostYou'd think by now, with all the warnings, all the headlines, lawyers would be acutely aware of the risks of blindly trusting generative AI. Especially in Georgia, where there have been highly publicized sanctions.
ExpertThat's what makes this even more baffling. ADA Leslie cannot claim ignorance of the technology's flaws, because the warnings were flashing red in her exact jurisdiction. Just months before, in October 2025, the *Atlanta Journal-Constitution* ran an investigation highlighting that Georgia courts were already cracking down harshly on AI use. We saw Jonesboro attorney Loletha Hale sanctioned by a U.S. District Judge for filing AI-generated fake arguments in a lawsuit involving comedian Katt Williams. She was ordered to notify all her clients and judges of her AI use for the next five years.
HostFive years! That's a pretty heavy penalty.
ExpertExactly. And it wasn't an isolated incident. Atlanta lawyer Diana Lynch was fined $2,500 by the Georgia Court of Appeals for bogus AI citations that made it into a divorce order. Another Atlanta attorney, Brian Braddy, was fined $1,000 in Valdosta federal court for citing *nine* fake cases. Despite these highly publicized, local sanctions, a prosecutor in a neighboring county made the exact same mistake in a murder trial. It truly beggars belief.
HostSo, why are smart, highly educated professionals, people whose jobs require precision and adherence to facts, repeatedly falling into this trap? What is the fundamental misunderstanding of these LLMs?
ExpertIt's a category error. Lawyers are treating Large Language Models, like ChatGPT or Claude, as if they are search engines or traditional legal databases like Westlaw or LexisNexis. They are not. These LLMs are probabilistic prediction engines. They don't "look up" information from a repository of facts. Instead, they perform what's called "next-token prediction." They calculate the statistical probability of which word should follow the previous word based on their vast training data.
HostSo, it's essentially a very sophisticated auto-complete?
ExpertExactly! Think of it like that. Legal citations, especially for court cases, have a highly structured and syntactically predictable format. A Georgia Supreme Court citation looks like `[Name] v. [Name], [Volume] Ga. [Page] ([Year])`. Because this format is so rigid, an LLM is exceptionally good at predicting what a citation *should* look like. It will string together a highly plausible-sounding case name—say, *State v. Smith*—assign it a realistic-looking volume number, and then generate a perfectly formatted quote written in convincing legalese. The AI doesn't *know* that the case doesn't exist; it only knows that the sequence of words matches the statistical pattern of a legal brief it's seen countless times in its training data. So when lawyers ask an LLM for case law, the machine, in its probabilistic wisdom, obliges by simply counterfeiting it. It's not retrieving; it's generating.
HostThat's a really crucial distinction that seems to be lost on a lot of people. It's not a smart library; it's a very convincing storyteller.
ExpertA very convincing storyteller that has no concept of truth or falsehood. It's just predicting the most statistically probable next word. And the output *looks* authoritative, which is part of the trap.
HostGiven all this, courts must be scrambling to figure out how to manage AI in legal practice beyond simply banning it, which clearly isn't working.
ExpertThey absolutely are. It's clear that outright bans, like the one DA Mosley's office ostensibly had, are futile. The technology is too ubiquitous. So, in late 2025 and early 2026, state court systems started scrambling to issue realistic guidelines. New York, for instance, in October 2025, released its "Interim Policy on the Use of Artificial Intelligence." It mandated that AI "must not be treated as a substitute for human judgment" and strictly prohibited entering confidential info into public AI platforms. But crucially, it *allowed* the use of approved AI products, provided humans verified the output.
HostSo, it's not a ban, but a 'use with extreme caution and verification' policy.
ExpertPrecisely. New Jersey took a different tack in January 2026, mandating that starting in 2027, all licensed attorneys must obtain Continuing Legal Education credits in "technology-related subjects," explicitly including the ethical use of AI in discovery and brief-writing. These are proactive, educational approaches that acknowledge the reality of the technology. They stand in stark contrast to Clayton County DA's failed "just don't use it" policy.
HostThat makes a lot of sense. So, we have prosecutors hallucinating legal precedent, the adversarial system failing, and a fundamental misunderstanding of the technology's capabilities. But the issues around AI and government don't stop there. This brings us to a recurring theme.
ExpertThis theme focuses on cases where the lines between AI regulator, contractor, and lobbyist blur, especially when the government is acting as both rule-maker and a massive AI customer.
HostAnd this time, we're looking at a new federal procurement rule that seems to completely undermine all the public talk about AI safety.
ExpertYou're right. In March of this year, the General Services Administration, or GSA, which is the procurement arm of the federal government, published a draft contract clause titled "Basic Safeguarding of Artificial Intelligence Systems." This clause is set to be incorporated into GSA's massive Multiple Award Schedule contracts. And while it includes standard cybersecurity and disclosure requirements, it contains what some legal analyses are calling a "poison pill" for AI safety.
HostA poison pill? How so?
ExpertBecause, according to firms like Baker Botts and Gibson Dunn, this clause explicitly forces AI vendors to override their commercial terms of service and strips away their ability to enforce their own safety restrictions. The exact language of the proposed mandate says: "The AI System must not refuse to produce data outputs or conduct analyses based on the Contractor's or Service Provider's discretionary policies."
HostSo, if an AI company builds ethical guardrails into its system, say, to prevent it from generating instructions for biological weapons or mass surveillance, this GSA clause would effectively tell them, "Too bad, government contract trumps your ethics"?
ExpertExactly. It explicitly states: "In the event of a conflict between this document and any policies, requirements, terms, conditions, or commercial agreements of the quote, the Contractor, or the Service Provider, this clause controls." This targets the operational safety layer of AI systems—the system prompts, the trust-and-safety filters, the use-case restrictions. It means if an AI company has an ethical policy preventing its model from creating racist propaganda or aiding in certain types of surveillance, for a government contract, that policy is essentially outlawed. The government is demanding an irrevocable, royalty-free license to use the system for "any lawful government purpose," and the vendor cannot say no.
HostThat is absolutely wild. It's like the government is saying, "We want you to build safe AI, but when *we* buy it, take all the safety features off." What prompted this aggressive move?
ExpertTo understand *why* the GSA issued this aggressive clause in March, we have to look back just weeks prior, to February 2026, and a very public standoff. Anthropic, an AI company that positions itself as one of the most safety-conscious of the major labs—they're the creators of the Claude model—was in negotiations with the Department of Defense. The DOD insisted that Anthropic's technology be available for "all lawful uses."
HostWhich sounds reasonable on the surface.
ExpertIt does, but Anthropic refused. They supported general national security applications, but they maintained strict Terms of Service prohibiting their AI from being used for two specific things: mass domestic surveillance and fully autonomous lethal weapons. They had ethical guardrails in place, and they stood by them.
HostSo, the company known for its safety-first approach wouldn't budge on its core ethical principles.
ExpertAnd when negotiations broke down ahead of a February 27th deadline, the Trump administration retaliated with overwhelming force. First, President Trump posted on Truth Social, directing every agency in the U.S. government to "IMMEDIATELY CEASE all use of Anthropic's technology," calling them "Leftwing nut jobs" trying to "STRONG-ARM the Department of War."
HostWow. From the White House.
ExpertAnd then, Defense Secretary Pete Hegseth took the unprecedented step of designating Anthropic a "supply chain risk"—a label historically reserved for foreign adversaries and espionage threats like Huawei. Hegseth stated, "America's warfighters will never be held hostage by the ideological whims of Big Tech."
HostSo, a company refusing to allow its AI to be used for mass domestic surveillance or fully autonomous lethal weapons is now a 'supply chain risk' to national security?
ExpertThat's the government's argument. Anthropic, naturally, sued, and in late March, a U.S. District Judge granted a preliminary injunction halting the DOD ban, calling the "supply chain risk" designation "likely both contrary to law and arbitrary and capricious." What's also notable is that shortly after the Pentagon banned Anthropic, rival OpenAI struck a defense deal, highlighting the market advantage of dropping ethical guardrails for government cash.
HostSo we have the government publicly demanding AI safety and regulation, hauling tech CEOs before Congress to ask about their "guardrails"...
ExpertBut then, the moment an AI company actually enforces those guardrails against the U.S. military, the government designates them a national security threat and rewrites federal procurement rules to explicitly outlaw AI safety filters. It's a complete contradiction.
HostIt’s more than a contradiction; it’s hypocrisy that undermines the entire regulatory narrative.
ExpertIt really does.
HostSo, to synthesize all this, what are the key takeaways you think our listeners should really sit with?
ExpertFirst, the Hannah Payne case shows an alarming escalation of AI-related harm. We've moved beyond embarrassing civil blunders to literal life-and-death constitutional due process violations. Second, the failure of paper policies. DA Mosley's 'clear policy' banning AI was functionally useless because it lacked any real technological or human verification workflow.
HostAnd that leads directly to the third point, the adversarial collapse. The justice system relies on opposing sides checking each other's work. When the prosecutor hallucinates the law, the defense attorney admits he didn't read the citations, and the judge signs the order blindly, the structural integrity of the court simply fails.
ExpertAnd fourth, this idea of procurement as anti-regulation. The GSA's March 2026 'Basic Safeguarding' clause and the DOD's February 2026 war against Anthropic clearly demonstrate that the government's desire to be an unrestricted consumer of AI completely overrides its stated desire to be a responsible regulator of AI.
HostSo, it leaves us with this question, one that I think listeners should really grapple with: If the federal government is actively using its massive procurement budget to force AI companies to strip away their safety guardrails, how can we possibly trust that same government to regulate AI safety for the public?
ExpertIt's a fundamental tension at the heart of our current AI policy.