Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Code review

    ()

    Discussion on GitHub from last week about a pull request* consisting of 13,000 lines of code written and reviewed by generative AI.

    * (A pull request is a request sent to the maintainers of a software project, showing them some code and asking them to “pull” the code into their project.)

    The first part of the page is the person who’s submitting the code (user joelreymont) explaining what it does, and then there’s a list of 40 “commits” (changes) that they want to make. You can skip all of that; the interesting-to-me part is the discussion after that list of commits.

    Among other things, user gasche and a couple of other people point out that the code as provided includes copyright headers attributing the code to someone named Mark Shinwell. In response, joelreymont asks a generative AI tool to do a copyright analysis, comparing the generated code to Shinwell’s real code in another project. The AI concludes that no code was copied from Shinwell.

    joelreymont later claims that “AI has a very deep understanding of how this code works.” (And even later, explains that what they mean by that is that if you ask the AI questions about the code, it can provide answers.)

    My favorite part is the exchange late in the thread where user yallop writes:

    “Here’s my question: why did the files that you submitted name Mark Shinwell as the author?”

    And joelreymont replies:

    “Beats me. AI decided to do so and I didn’t question it.”

    (Original Facebook post.)

  • Factor Fexcectorn

    ()

    An article in Scientific Reports (one of many journals published by Nature Portfolio), published a week ago, includes yet another laughably bad AI-generated graphic.

    Among other things, the graphic includes text like:

    MISSING
    VALUE
    &runctitional
    features

    and:

    Historical
    Medical frymmblal
    & Environental features

    and:

    To/
    Line
    storee

    and:

    Factor Fexcectorn

    and:

    RELU
    DROP-OUT
    Totalbottl,
    REMECH N

    …To view the graphic in context, scroll down about a third of the way through the article, or search for the caption “Overall working of the framework presented as an infographic.”

    According to Wikipedia, “Scientific Reports is a peer-reviewed open-access scientific mega journal published by Nature Portfolio, covering all areas of the natural sciences. The journal was established in 2011. The journal states that their aim is to assess solely the scientific validity of a submitted paper”

    (Three years ago, someone who said they were a member of the editorial board described the peer review process for the journal as “pretty standard.”)

    (Original Facebook post.)

  • Freelance articles

    ()

    “A suspicious pitch [for an article] from a freelancer led editor Nicholas Hune-Brown to dig into their past work. By the end, four publications, including The Guardian and Dwell, had removed articles from their sites.”

    Hune-Brown writes:

    “I was embarrassed. I had been naively operating with a pre-ChatGPT mindset, still assuming a pitch’s ideas and prose were actually connected to the person who sent it.”

    “this generation’s internet scammers are […] taking advantage of an ecosystem uniquely susceptible to fraud—where publications with prestigious names publish rickety journalism under their brands, where fact-checkers have been axed and editors are overworked, where technology has made falsifying pitches and entire articles trivially easy[…]”

    Investigating a Possible Scammer in Journalism’s AI Era

    (Original Facebook post.)

  • Delusions and reality checks

    ()

    “They thought they were making technological breakthroughs. It was an AI-sparked delusion”

    Article about a couple of people whose interactions with LLM chatbots resulted in mental-health issues.

    Here’s one example of what not to do when you’re interacting with a chatbot:

    “Multiple times, Brooks asked the chatbot for what he calls ‘reality checks.’ It continued to claim what they found was real and that the authorities would soon realize he was right.”

    (You can’t get valid reality checks from a chatbot. If a chatbot appears to be trying to convince you of something, please get a reality check from a human.)

    …Content warning for the article mentioning cases of suicide and murder related to chatbots, but that’s not its focus.

    (Original Facebook post.)

  • Attorney fine

    (, )

    A California attorney must pay a $10,000 fine for filing a state court appeal full of fake quotations generated by the artificial intelligence tool ChatGPT.

    The fine appears to be the largest issued over AI fabrications by a California court and came with a blistering opinion stating that 21 of 23 quotes from cases cited in the attorney’s opening brief were made up.

    California issues historic fine over lawyer’s ChatGPT fabrications

    Side note:

    I saw a Guardian opinion piece yesterday that, after it pointed out some issues with a generative-AI product, quoted an authoritative-sounding source as saying that you have to be careful about using generative AI, but it’s fine to use it for some tasks, such as factchecking.

    I dropped a note to the Guardian’s readers’ editor and to the person who said to use LLMs for factchecking, pointing out to them that you absolutely should never use LLMs to check facts, but I don’t expect that that note will have much effect.

    (Original Facebook post.)

  • Wrong attribution

    ()

    I often forget to append “-ai” to the ends of my Google searches. I usually manage to avoid reading the “AI Overview,” but once in a while I do accidentally read it. And I think that every single time I’ve done that, the AI Overview has included at least one false statement.

    So I don’t generally bother to post them any more. But once in a while the falsehood is so utterly ridiculous and egregious that I figure it’s worth posting about.

    Here’s today’s:

    The line “Things would be so different / If they were not as they are!” is from the poem “The Waking” by American poet Theodore Roethke. It is a simple, yet profound statement about the nature of reality and the limitations of hypotheticals.

    […]

    In “The Waking,” the full quote appears as a standalone stanza:

    “Things would be so different / If they were not as they are!”

    The poem, which won the Pulitzer Prize for Poetry in 1954, is a meditation on mortality and the cycles of nature. It often uses paradoxical language, as seen in the line, “I wake to sleep, and take my waking slow.” The couplet “Things would be so different / If they were not as they are!” captures this sense of circularity and resignation.

    All of which is utterly false.

    The line in question is from an Anna Russell piece. It does not appear in the Roethke poem. It does not appear in any of the Roethke pages that Google Gemini links to in support of its claim. The only thing that the line and the Roethke poem have in common is that both date from 1953.

    I suppose one could claim that the Russell line is a “simple, yet profound statement about the nature of reality and the limitations of hypotheticals,” but really the line is a tautological joke.

    Remember: Don’t trust anything that generative AI tells you.

    (Original Facebook post.)

  • Vibe coding, or deleting your database?

    ()

    Replit is a company that provides a generative-AI-based coding tool.

    User @jasonlk was enjoying using it, until the day that the Replit tool “deleted the entire database without permission during an active code and action freeze.”

    Yay, vibe coding is fun!

    …On a side note, the tool refers to itself as “I”, and says things like “I panicked instead of thinking.” Which may be why the user says things like “He knew.” To be clear: The tool didn’t panic, it doesn’t think, it doesn’t have a gender, and it didn’t “know” it was doing something bad. All of its claims about its state of mind are false.

    (Sadly, most of the posts consist mostly of screen snaps without alt text.)

    (Original Facebook post.)

  • AI Hallucination Cases database

    ()

    That thing where lawyers (and others) use generative AI in court filings, and the AI makes stuff up? Now there’s a list of such situations: the AI Hallucination Cases database.

    “This database tracks legal decisions in cases where generative AI produced hallucinated content – typically fake citations, but also other types of arguments.”

    “While seeking to be exhaustive (201 cases identified so far), it is a work in progress and will expand as new examples emerge.”

    (Original Facebook post.)

  • Gell-Mann

    ()

    Mike Pope on the Gell-Mann Amnesia Effect/Knoll’s Law (“everything you read in the newspapers is absolutely true, except for the rare story of which you happen to have firsthand knowledge”) and ChatGPT.

    (Original Facebook post.)

  • DOGE

    ()

    “We obtained records showing how a Department of Government Efficiency staffer with no medical experience used artificial intelligence to identify which VA contracts to kill. “AI is absolutely the wrong tool for this,” one expert said.”

    “Lavingia’s system also used AI to extract details like the contract number and “total contract value.” This led to avoidable errors, where AI returned the wrong dollar value when multiple were found in a contract. Experts said the correct information was readily available from public databases.”

    (Original Facebook post.)