Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Delusions and reality checks

    ()

    “They thought they were making technological breakthroughs. It was an AI-sparked delusion”

    Article about a couple of people whose interactions with LLM chatbots resulted in mental-health issues.

    Here’s one example of what not to do when you’re interacting with a chatbot:

    “Multiple times, Brooks asked the chatbot for what he calls ‘reality checks.’ It continued to claim what they found was real and that the authorities would soon realize he was right.”

    (You can’t get valid reality checks from a chatbot. If a chatbot appears to be trying to convince you of something, please get a reality check from a human.)

    …Content warning for the article mentioning cases of suicide and murder related to chatbots, but that’s not its focus.

    (Original Facebook post.)

  • Attorney fine

    (, )

    A California attorney must pay a $10,000 fine for filing a state court appeal full of fake quotations generated by the artificial intelligence tool ChatGPT.

    The fine appears to be the largest issued over AI fabrications by a California court and came with a blistering opinion stating that 21 of 23 quotes from cases cited in the attorney’s opening brief were made up.

    California issues historic fine over lawyer’s ChatGPT fabrications

    Side note:

    I saw a Guardian opinion piece yesterday that, after it pointed out some issues with a generative-AI product, quoted an authoritative-sounding source as saying that you have to be careful about using generative AI, but it’s fine to use it for some tasks, such as factchecking.

    I dropped a note to the Guardian’s readers’ editor and to the person who said to use LLMs for factchecking, pointing out to them that you absolutely should never use LLMs to check facts, but I don’t expect that that note will have much effect.

    (Original Facebook post.)

  • Wrong attribution

    ()

    I often forget to append “-ai” to the ends of my Google searches. I usually manage to avoid reading the “AI Overview,” but once in a while I do accidentally read it. And I think that every single time I’ve done that, the AI Overview has included at least one false statement.

    So I don’t generally bother to post them any more. But once in a while the falsehood is so utterly ridiculous and egregious that I figure it’s worth posting about.

    Here’s today’s:

    The line “Things would be so different / If they were not as they are!” is from the poem “The Waking” by American poet Theodore Roethke. It is a simple, yet profound statement about the nature of reality and the limitations of hypotheticals.

    […]

    In “The Waking,” the full quote appears as a standalone stanza:

    “Things would be so different / If they were not as they are!”

    The poem, which won the Pulitzer Prize for Poetry in 1954, is a meditation on mortality and the cycles of nature. It often uses paradoxical language, as seen in the line, “I wake to sleep, and take my waking slow.” The couplet “Things would be so different / If they were not as they are!” captures this sense of circularity and resignation.

    All of which is utterly false.

    The line in question is from an Anna Russell piece. It does not appear in the Roethke poem. It does not appear in any of the Roethke pages that Google Gemini links to in support of its claim. The only thing that the line and the Roethke poem have in common is that both date from 1953.

    I suppose one could claim that the Russell line is a “simple, yet profound statement about the nature of reality and the limitations of hypotheticals,” but really the line is a tautological joke.

    Remember: Don’t trust anything that generative AI tells you.

    (Original Facebook post.)

  • Vibe coding, or deleting your database?

    ()

    Replit is a company that provides a generative-AI-based coding tool.

    User @jasonlk was enjoying using it, until the day that the Replit tool “deleted the entire database without permission during an active code and action freeze.”

    Yay, vibe coding is fun!

    …On a side note, the tool refers to itself as “I”, and says things like “I panicked instead of thinking.” Which may be why the user says things like “He knew.” To be clear: The tool didn’t panic, it doesn’t think, it doesn’t have a gender, and it didn’t “know” it was doing something bad. All of its claims about its state of mind are false.

    (Sadly, most of the posts consist mostly of screen snaps without alt text.)

    (Original Facebook post.)

  • AI Hallucination Cases database

    ()

    That thing where lawyers (and others) use generative AI in court filings, and the AI makes stuff up? Now there’s a list of such situations: the AI Hallucination Cases database.

    “This database tracks legal decisions in cases where generative AI produced hallucinated content – typically fake citations, but also other types of arguments.”

    “While seeking to be exhaustive (201 cases identified so far), it is a work in progress and will expand as new examples emerge.”

    (Original Facebook post.)

  • Gell-Mann

    ()

    Mike Pope on the Gell-Mann Amnesia Effect/Knoll’s Law (“everything you read in the newspapers is absolutely true, except for the rare story of which you happen to have firsthand knowledge”) and ChatGPT.

    (Original Facebook post.)

  • DOGE

    ()

    “We obtained records showing how a Department of Government Efficiency staffer with no medical experience used artificial intelligence to identify which VA contracts to kill. “AI is absolutely the wrong tool for this,” one expert said.”

    “Lavingia’s system also used AI to extract details like the contract number and “total contract value.” This led to avoidable errors, where AI returned the wrong dollar value when multiple were found in a contract. Experts said the correct information was readily available from public databases.”

    (Original Facebook post.)

  • Summarizing medical info

    ()

    About some of the problems with having generative AI summarize medical information.

    I summarize medical information for doctors, researchers, and patients every day for a living, and I can promise you that any summary you get from chatGPT will have at least one significant error. And how could you possibly know? If you don’t understand what your doctor is telling you, how could you effectively vet the summary for errors?

    (Original Facebook post.)

  • Summarizing research

    ()

    “Generalization bias in large language model summarization of scientific research”

    when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. […] Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts[…] In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations[…] Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings.

    (Article from April.)

    (Indirectly via Aliette.)

    (Original Facebook post.)

  • Reading list

    ()

    “Chicago Sun-Times prints summer reading list full of fake books”

    “Reading list [created by generative AI] in advertorial supplement contains 66% made up books by real authors.”

    Apparently not created by the Sun-Times:

    “The reading list appeared in a 64-page supplement called ‘Heat Index,’ which was a promotional section not specific to Chicago. Buscaglia told 404 Media the content was meant to be ‘generic and national’ and would be inserted into newspapers around the country.”

    (Original Facebook post.)