Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Is 2026 next year?

    ()

    Hacker News discussion of various generative-AI systems’ responses to the query [Is 2026 next year?]

    For example, here are the first couple of responses shown on that page:

    Google search AI response: “No, 2026 is not next year; it will be the year after next. Since the current year is 2025, “next year” would be 2026, which is only a little over a year away. “

    ChatGPT does something similar; response:

    “No — if today is 2025, then 2026 is the year after next. Next year would be 2026 only if we were currently in 2025. Right now (based on the system date) it is December 2025, so: Next year = 2026 The year after next = 2027 So in this moment: yes, 2026 is next year.”

    Also:

    Claude [version] Haiku 4.5

    “No, 2026 is not next year. Since it’s currently December 2025, next year would be 2026. Wait—let me correct that: 2026 is next year from the perspective of 2025. So yes, 2026 is next year.”

    And:

    Grok [version] 4 fast: “No, 2026 is not next year.

    Today is December 2, 2025, so:

    – “Next year” means 2026 (starting January 1, 2026). – But since we’re already in the last month of 2025, 2026 is indeed the very next calendar year.

    So yes — 2026 is next year. “

    …Some people (probably humans) commented on that page that this issue might be related to when the models’ training data was last updated. In keeping with that theory, here’s another AI’s answer:

    LLama [version] 4 Scout: “To determine if 2026 is next year, we need to consider the current year. As of my knowledge cutoff, the current year is 2024.

    If we assume the current year is 2024, then:

        2025 would be next year.
    2026 would be the year after next.

    However, if the current year is 2025, then:

        2026 would indeed be next year.

    Could you please clarify or provide the current year for a more accurate response?”

    But that’s presumably not the entire issue, because the other systems quoted above did explicitly say that the current year is 2025, while still responding as if the current year were 2024.

    …I should mention that several of the models quoted on that Hacker News page did correctly answer the question. (And Google’s answer seems to have improved over time.) But the fact that several of them failed in similar ways does make it sound to me like there may be something more specific going on than just the usual LLMs-make-stuff-up issue.

  • Code review

    ()

    Discussion on GitHub from last week about a pull request* consisting of 13,000 lines of code written and reviewed by generative AI.

    * (A pull request is a request sent to the maintainers of a software project, showing them some code and asking them to “pull” the code into their project.)

    The first part of the page is the person who’s submitting the code (user joelreymont) explaining what it does, and then there’s a list of 40 “commits” (changes) that they want to make. You can skip all of that; the interesting-to-me part is the discussion after that list of commits.

    Among other things, user gasche and a couple of other people point out that the code as provided includes copyright headers attributing the code to someone named Mark Shinwell. In response, joelreymont asks a generative AI tool to do a copyright analysis, comparing the generated code to Shinwell’s real code in another project. The AI concludes that no code was copied from Shinwell.

    joelreymont later claims that “AI has a very deep understanding of how this code works.” (And even later, explains that what they mean by that is that if you ask the AI questions about the code, it can provide answers.)

    My favorite part is the exchange late in the thread where user yallop writes:

    “Here’s my question: why did the files that you submitted name Mark Shinwell as the author?”

    And joelreymont replies:

    “Beats me. AI decided to do so and I didn’t question it.”

    (Original Facebook post.)

  • Factor Fexcectorn

    ()

    An article in Scientific Reports (one of many journals published by Nature Portfolio), published a week ago, includes yet another laughably bad AI-generated graphic.

    Among other things, the graphic includes text like:

    MISSING
    VALUE
    &runctitional
    features

    and:

    Historical
    Medical frymmblal
    & Environental features

    and:

    To/
    Line
    storee

    and:

    Factor Fexcectorn

    and:

    RELU
    DROP-OUT
    Totalbottl,
    REMECH N

    …To view the graphic in context, scroll down about a third of the way through the article, or search for the caption “Overall working of the framework presented as an infographic.”

    According to Wikipedia, “Scientific Reports is a peer-reviewed open-access scientific mega journal published by Nature Portfolio, covering all areas of the natural sciences. The journal was established in 2011. The journal states that their aim is to assess solely the scientific validity of a submitted paper”

    (Three years ago, someone who said they were a member of the editorial board described the peer review process for the journal as “pretty standard.”)

    (Original Facebook post.)

  • Freelance articles

    ()

    “A suspicious pitch [for an article] from a freelancer led editor Nicholas Hune-Brown to dig into their past work. By the end, four publications, including The Guardian and Dwell, had removed articles from their sites.”

    Hune-Brown writes:

    “I was embarrassed. I had been naively operating with a pre-ChatGPT mindset, still assuming a pitch’s ideas and prose were actually connected to the person who sent it.”

    “this generation’s internet scammers are […] taking advantage of an ecosystem uniquely susceptible to fraud—where publications with prestigious names publish rickety journalism under their brands, where fact-checkers have been axed and editors are overworked, where technology has made falsifying pitches and entire articles trivially easy[…]”

    Investigating a Possible Scammer in Journalism’s AI Era

    (Original Facebook post.)

  • Delusions and reality checks

    ()

    “They thought they were making technological breakthroughs. It was an AI-sparked delusion”

    Article about a couple of people whose interactions with LLM chatbots resulted in mental-health issues.

    Here’s one example of what not to do when you’re interacting with a chatbot:

    “Multiple times, Brooks asked the chatbot for what he calls ‘reality checks.’ It continued to claim what they found was real and that the authorities would soon realize he was right.”

    (You can’t get valid reality checks from a chatbot. If a chatbot appears to be trying to convince you of something, please get a reality check from a human.)

    …Content warning for the article mentioning cases of suicide and murder related to chatbots, but that’s not its focus.

    (Original Facebook post.)

  • Attorney fine

    (, )

    A California attorney must pay a $10,000 fine for filing a state court appeal full of fake quotations generated by the artificial intelligence tool ChatGPT.

    The fine appears to be the largest issued over AI fabrications by a California court and came with a blistering opinion stating that 21 of 23 quotes from cases cited in the attorney’s opening brief were made up.

    California issues historic fine over lawyer’s ChatGPT fabrications

    Side note:

    I saw a Guardian opinion piece yesterday that, after it pointed out some issues with a generative-AI product, quoted an authoritative-sounding source as saying that you have to be careful about using generative AI, but it’s fine to use it for some tasks, such as factchecking.

    I dropped a note to the Guardian’s readers’ editor and to the person who said to use LLMs for factchecking, pointing out to them that you absolutely should never use LLMs to check facts, but I don’t expect that that note will have much effect.

    (Original Facebook post.)

  • Wrong attribution

    ()

    I often forget to append “-ai” to the ends of my Google searches. I usually manage to avoid reading the “AI Overview,” but once in a while I do accidentally read it. And I think that every single time I’ve done that, the AI Overview has included at least one false statement.

    So I don’t generally bother to post them any more. But once in a while the falsehood is so utterly ridiculous and egregious that I figure it’s worth posting about.

    Here’s today’s:

    The line “Things would be so different / If they were not as they are!” is from the poem “The Waking” by American poet Theodore Roethke. It is a simple, yet profound statement about the nature of reality and the limitations of hypotheticals.

    […]

    In “The Waking,” the full quote appears as a standalone stanza:

    “Things would be so different / If they were not as they are!”

    The poem, which won the Pulitzer Prize for Poetry in 1954, is a meditation on mortality and the cycles of nature. It often uses paradoxical language, as seen in the line, “I wake to sleep, and take my waking slow.” The couplet “Things would be so different / If they were not as they are!” captures this sense of circularity and resignation.

    All of which is utterly false.

    The line in question is from an Anna Russell piece. It does not appear in the Roethke poem. It does not appear in any of the Roethke pages that Google Gemini links to in support of its claim. The only thing that the line and the Roethke poem have in common is that both date from 1953.

    I suppose one could claim that the Russell line is a “simple, yet profound statement about the nature of reality and the limitations of hypotheticals,” but really the line is a tautological joke.

    Remember: Don’t trust anything that generative AI tells you.

    (Original Facebook post.)

  • Vibe coding, or deleting your database?

    ()

    Replit is a company that provides a generative-AI-based coding tool.

    User @jasonlk was enjoying using it, until the day that the Replit tool “deleted the entire database without permission during an active code and action freeze.”

    Yay, vibe coding is fun!

    …On a side note, the tool refers to itself as “I”, and says things like “I panicked instead of thinking.” Which may be why the user says things like “He knew.” To be clear: The tool didn’t panic, it doesn’t think, it doesn’t have a gender, and it didn’t “know” it was doing something bad. All of its claims about its state of mind are false.

    (Sadly, most of the posts consist mostly of screen snaps without alt text.)

    (Original Facebook post.)

  • AI Hallucination Cases database

    ()

    That thing where lawyers (and others) use generative AI in court filings, and the AI makes stuff up? Now there’s a list of such situations: the AI Hallucination Cases database.

    “This database tracks legal decisions in cases where generative AI produced hallucinated content – typically fake citations, but also other types of arguments.”

    “While seeking to be exhaustive (201 cases identified so far), it is a work in progress and will expand as new examples emerge.”

    (Original Facebook post.)

  • Gell-Mann

    ()

    Mike Pope on the Gell-Mann Amnesia Effect/Knoll’s Law (“everything you read in the newspapers is absolutely true, except for the rare story of which you happen to have firsthand knowledge”) and ChatGPT.

    (Original Facebook post.)