Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Mountain-climbing advice

    (, )

    The US Forest Service reports that “a party of three young, novice climbers” climbed Mt. Shasta in late August. They got lost on the way down, and had to be rescued.

    So why am I posting about that here? Well:

    Some key factors in this incident: The group reported using an AI platform, YouTube videos and AllTrails to plan their climb. Specifically, they relied on AI to determine what types of food to bring, and how much food and water they would need for the trip.

    Never trust what generative AI tells you. I’ve been saying that for a while, but I feel like it’s especially important in a context where your survival depends on having accurate information.

  • Made-up witnesses

    ()

    Yet another lawyer relied on GPT, resulting in yet another false court filing that no human checked.

    But in this case, we have a video of a hearing, at which five New Mexico Supreme Court justices ask the lawyer what happened and why, and what he thinks his punishment should be.

    The video is titled “Show Cause Hearing: State v. Sandoval.” The proceedings start around 25:00 in that video and go to around 50:00.

    Some notes about the hearing:

    According to an article about the issue, the lawyer, Stephen Aarons, filed “a legal brief that referenced fictional witnesses and testimony apparently fabricated by the artificial intelligence he used to help him write it.”

    In the hearing, Aarons says that he had previously used GPT and hadn’t had any problems, so he assumed that it couldn’t cause any problems.

    Justice Shannon Bacon, disbelievingly, says: “Do you read anything about what’s going on in the world?” And: “My thirteen-year-old nephew knows about [generative-AI] hallucinations. My 75-year-old stepmother knows about hallucinations.”

    Aarons says that his situation is different from other lawyers-using-generative-AI situations, because in those other situations, the problem (he says) has been made-up citations, and that wasn’t what happened here.

    He then suggests that the solution to this problem is for judges to tell lawyers that AI can hallucinate. (The false filing happened a year ago, and he somehow believes that at that point nobody knew about the problem.) He says that if only the court had told him to be careful about this, the whole problem could’ve been avoided. He later suggests that if only lawyers were shown more horror stories about AI, they would of course be more careful. And if only the court had granted his opponent’s motion to strike his filing, he would have fixed things. And that he hopes the mistake he made will prevent others from making the same mistake in the future. And so on and so on.

    One of the justices points out that the main problem here is that Aarons didn’t check everything before he put his name on something: “You have presented briefing to the court that we cannot rely on, and your client is the one suffering from this.”

    Aarons eventually reveals that he hasn’t told his client yet what he did. (He told the client’s relatives that there was a problem, but not about his use of GPT, and it’s unclear whether the relatives told the client.)

    At around 50:00, the justices leave the room to discuss what happens next, and the video keeps running for another 20 minutes but then ends without anyone returning to the room.

    The conclusion of the hearing is in a separate brief video, starting at 05:30 and running for two minutes. The justices tell Aarons that they’re referring him to the disciplinary board and sanctioning him $5,000, payable to the client protection fund.

    Every time I see another case like this, I wonder how long lawyers are going to continue saying that they didn’t know that generative AI could make stuff up.

  • Unauthorized biographies

    (, )

    In the New York Times, journalist Kashmir Hill writes that she recently discovered that there was an AI-generated biography of her available for sale.

    A biography of Kashmir Hill — title: “The Biography of Kashmir Hill” — [was released] in August 2025. My life story had a mottled brown cover and a publisher I’d never heard of before.

    […]

    My biography is 90 pages long and should be shorter. It combines facts about me that are widely available on the internet, such as where I grew up, with generic insights that could be true of anyone, like a horoscope spread over dozens of pages. “You cannot understand Kashmir Hill without understanding her contradictions,” my biographer wrote, along with an excruciatingly long description of my elaborate coffee-making ritual. (Fact check: My husband does it.)

    […]

    I clicked on the author, one John Crane Miller. His bio page described him as a “seasoned biographer and cultural analyst,” and his portrait was a widely used stock photo of a white man in a suit speaking at a conference.

    […]

    “The Biography of Kashmir Hill[…]” was one of 10 biographies that Mr. Miller had published in a single week

    […]

    [Miller] claimed to have read my childhood diary and spoken with people who know me well “and those who wish they didn’t.”

    Hill goes on to interview a different “author” of AI-generated books, Bill Johns, who has published 445 books on Amazon (he sells “a few hundred books per month, each earning him roughly $7”), and to discuss some other general topics around AI-generated books. But the above excerpts are the most relevant bits for my purposes.

  • Google AI lies about Scalzi

    (, , )

    John Scalzi writes:

    Every assertion in this “AI Overview” of the question “What coffee does John Scalzi drink” is wrong. I don’t regularly drink coffee (and never black) I’ve never had black sesame jasmine cream tea, and I don’t hang in coffee shops. Don’t trust “AI” ever!

    The post was accompanied by a screensnap from Google’s AI Overview, answering the question “What coffee does John Scalzi drink”. As usual, the AI Overview provided links that supposedly linked to sources for the information that it was giving, but the linked-to pages don’t say what Google says they say.

    After Scalzi’s posted the above to Bluesky, Google’s answer changed to link to his post. But just because AI Overview might eventually be updated to give correct information, that doesn’t mean we should trust it.

  • AI models answer questions about medical images they haven’t seen

    (, )

     “AI models happily came up with answers to questions about a supposedly accompanying image — even if the researchers never even showed it an image.”

    The article quotes the researchers as talking about this in terms of there being “information […] hidden in a sentence or a question” that allows the AIs to correctly answer the questions even without access to the images that the questions are about, but that seems weirdly implausible to me. Later in the article, there are implications of what seems to me far more likely: that the AI was trained on these questions and answers, and thus can provide the answers without having to have access to the images that accompany the text.

    More from the researchers:

    “Another implication is that, now that we know an AI can say ‘I see evidence of malignant melanoma on your skin’ without even having access to any images, how much can we trust it when it says the same while actually seeing the image?” Asadi posited. “We definitely need more effort being put in safety and alignment of such models, and might need to think twice before deploying them in user/patient-facing systems.”

    […] “The number one [takeaway] would be that just because the AI is saying, very convincingly, that it is seeing something, it doesn’t mean that it is actually seeing that.”

  • Canadian Immigration Department uses lying generative AI

    ()

    “Canada rejected her permanent residence application. Her job duties were made up — by Immigration’s AI reviewer”

    Postdoc Kémy Adé applied for permanent residence in Canada, but was rejected because a generative-AI tool hallucinated a set of job duties that she didn’t have, and the Immigration Department therefore ruled that the work she had done (that the AI made up) didn’t match the work she claimed to have done (that she really did).

  • Police officer turns into frog

    ()

    You know how organizations and doctors and therapists and lots of other people want to use generative-AI tools to write up summaries of meetings?

    Now there are a couple of tools that police can use to create reports summarizing body-camera audio and/or video.

    And some police love it:

    “Most of our officers are kind of awestruck with (Draft One) because it’s such a new, innovative thing for us,” Weishar said. “It’s like that brand new car that’s got all the features to it. For us, it’s crazy that you can just press a button and it’ll tell you everything about the case that you were on and give you a pretty decent police report to edit.”

    But (gasp! shock!) it has certain pitfalls:

    “I read the report, and I’m like, ‘Man, this really looks like an officer wrote it,’” Sever recalled. “But when it got to one part, it said, ‘And then the officer turned into a frog, and a magic book appeared and began granting wishes.’ … It was because they had, like, ‘Harry Potter’ on in the background. So it picked up the noise from the TV and added it to the report.”

    The second half of the article has some reasonably good discussion of some of the reasons police shouldn’t be using these tools. But it won’t surprise me if some police departments start doing it anyway. This kind of software is a great time-saver, as long as you don’t mind when the resulting reports describe officers turning into frogs.

  • Law firm dissolved

    ()

    In Mississippi, a lawyer included AI-generated fake citations and was told not to do that, but kept doing it.

    The latest judge to receive her fake citations was not amused, and has issued sanctions against her and the two partners in the firm that she worked for.

    If I’m understanding right, the partners have now dissolved the firm.

    (It looks like Ms. Watson, the lawyer who used AI-generated fake citations in ten different cases, may be the daughter of one of the two partners.)

    The judge reacted strongly to Ms. Watson’s behavior:

    In light of repeated warnings from federal courts about the risk of hallucinated cases, as well as CLE trainings she attended, direct notice and knowledge of the same prior mistakes, her violation of the Firm’s AI policy, and the sheer number of filings, Ms. Watson’s misconduct is particularly egregious and prolific.

    The partners are also being sanctioned for failing to notice the problems. For example:

    a large portion of Billups’ argument relies on a case styled Jackson v. Gautreaux, 3 F. 4th 182, 190 (5th Cir. 2021). […] In fact, this case is cited eight times, even arguing that a jury should be instructed under its holding. […] In reality, Jackson is an excessive force and failure to train case and is wholly irrelevant to the case at bar. A seasoned attorney examining the brief should have read a case so heavily relied upon. Had he done so, he would have easily discovered the problems.

    The judge noted that the usual penalty for this sort of thing has been fines, but quoted another case about why fines are insufficient:

    “If fines and public embarrassment were effective deterrents, there would not be so many [AI misuse] cases to cite.”

    (Given that there are so many such cases, I probably won’t post about all the ones I hear about, but this one did seem especially egregious.)

  • More hallucitations

    ()

    “AI Is Inventing Academic Papers That Don’t Exist — and They’re Being Cited in Real Journals”

    Rolling Stone says:

    [Academic] articles which include references to nonexistent research material […] are themselves being cited in other papers, which effectively launders their erroneous citations. This leads to students and academics (and any large language models they may ask for help) identifying those “sources” as reliable without ever confirming their veracity. The more these false citations are unquestioningly repeated from one article to the next, the more the illusion of their authenticity is reinforced.

  • Fake blobfish

    ()

    “Deep Sea Social Media is Flooded by AI Slop”

    I focus more on text than on images on this site, but when AI-generated images inaccurately portray what real-world creatures look like, I figure that more or less fits my theme.