Generative AI Lies

Examples of generative AI making stuff up

Posts

  • Scaling and reliability

    ()

    “A common assumption is that scaling up [LLMs] will improve their reliability—for instance, by increasing the amount of data they are trained on, or the number of parameters they use to process information. However, more recent and larger versions of these language models have actually become more unreliable, not less, according to a new study.”

    “This decrease in reliability is partly due to changes that made more recent models significantly less likely to say that they don’t know an answer, or to give a reply that doesn’t answer the question. Instead, later models are more likely to confidently generate an incorrect answer.”

    (Article from Oct. 3.)

    (Original Facebook post.)

  • Apple Intelligence

    ()

    In Apple’s launch event for the iPhone 16 yesterday, I was not thrilled with the amount of emphasis they put on the new “Apple Intelligence” features. But I did think that if those features work well, some of them could be pretty useful.

    Unfortunately, this review makes me even more dubious.

    “In the preview I’m using, Apple Intelligence does an uncomfortable amount of making things up.”

    “like the time it alerted me that Donald Trump had endorsed Tim Walz for president. (Ha.) And the time it made up the idea that I’m teaching at UC Berkeley. (No.) And the time it elevated an obvious Social Security scam to my ‘priority’ inbox. (Yikes). And the time it edited a selfie to make me bald. (Double yikes.)”

    “it feels weird […] to see fabrications and misinterpretations of your life appear on your lock screen, inbox and other core parts of your iPhone.”

    “I told Apple about the many times I saw Apple Intelligence get facts wrong (I’ve had at least five to 10 laugh-out-loud moments per day). It says it is working to improve accuracy. But so is every other AI company — and that has proved to be a giant challenge.”

    (Original Facebook post.)

  • Fact-checking

    ()

    Email from a political organization:

    “As we navigate new waters and integrate cutting-edge AI technology into our fact-checking processes, we understand that occasional hurdles are inevitable.”

    …I sure hope that they’re not talking about using generative AI as a fact-checker. I’ve now sent them an email to point out what a terrible idea that would be, just in case.

    (Original Facebook post.)

  • More financial advice

    ()

    Yet another article that talks about using ChatGPT for your finances. As usual for these articles, it both explicitly says that ChatGPT gets things ruinously wrong, and frames things as if using ChatGPT for your finances is a perfectly reasonable thing to do.

    Make sure to triple check all the numbers and suggestions it gives you, because some of the suggestions we got made no sense and were totally incorrect — if I had followed what ChatGPT suggested, it could have seriously wrecked my credit score and got me in even more debt.

    Here’s what I did and how you can do the same

    I continue to be baffled by this kind of take. If you went to a human financial adviser and they gave you advice that would wreck your credit score, you wouldn’t say “So when you go to this adviser, be sure to check their numbers!”; you would say “Don’t go to this adviser.”

    (Original Facebook post.)

  • Medical-journal image

    ()

    Obviously AI-generated image from an article in the journal Medicine.

    The article has now been retracted “after concerns were raised over the integrity of the data and an inaccurate figure,” but you can still view the whole article or just the image.

    The AI-generated image is figure 2 in the article, labeled “Mechanism diagram of alkaline water treatment for chronic gouty arthritis.” It seems clear that nobody reviewed that image in any way.

    Medicine’s website says: “The Medicine® review process emphasizes the scientific, technical and ethical validity of submissions.”

    (via Mary Anne)

    (Original Facebook post.)

  • Human nonsense generator

    ()

    “Listen, you don’t need a large language model like chatGPT. You can ask me questions and I’ll generate confident sounding nonsense at a fraction of the resource consumption. Half the time, I don’t even remember to drink water!”

    —existennialmemes

    https://www.tumblr.com/existennialmemes/751305003016339456/listen-you-dont-need-a-large-language-model-like

    (Original Facebook post.)

  • Retrieval Augmented Generation

    ()

    Article (from May 4) about feeding relevant information to a generative AI to reduce the likelihood of it making stuff up, an approach known as “RAG,” or “Retrieval Augmented Generation.”

    “RAG can help reduce a model’s hallucinations — but it’s not the answer to all of AI’s hallucinatory problems. Beware of any vendor that tries to claim otherwise.”

    (Original Facebook post.)

  • Khan Academy

    (, )

    Khan Academy pilot/beta program: using GPT to teach kids.

    “Simple concepts such as multiplication and division of integers and decimals repeatedly offer incorrect answers, and will even sometimes treat students’ wrong inputs as the correct solutions.”

    (Article from August, 2023.)

    (Original Facebook post.)

  • Eclipse

    (, )

    “Elon Musk’s Grok [AI] Creates Bizarre Fake News About the Solar Eclipse Thanks to Jokes on X”

    Grok’s headline: “Sun’s Odd Behavior: Experts Baffled.”

    Gizmodo article explains: “The tweets that apparently prompted this bizarre proclamation from Grok included joke after joke of people wondering where the sun had gone. It appears that so many people making the same joke confused Grok, which tried to turn the jokes into something we should all be worried about.”

    (Original Facebook post.)

  • Who are we talking to?

    ()

    Long but excellent article about generative AI (LLM) chatbots: “Who are we talking to when we talk to these bots?”, by Colin Fraser.

    (Article from early 2023; I’m sure some details of LLM behavior have changed since then, but I think the core ideas here are still valid and relevant.)

    Fraser’s central points are that LLMs are designed to create sequences of words, and that a chatbot persona is a fictional character overlaid (by means of training) on that word-sequence-creation software. And that our experience of having a conversation with an LLM chatbot is partly shaped by the user interface and by our expectations about how conversations go.

    A few quotes:

    “At least half of the reason that interacting with the bot feels like a conversation to the user is that the user actively participates as though it is one.”

    “the first-person text generated by the computer is complete fiction. It’s the voice of a fictional ChatGPT character, one of the two protagonists in the fictional conversation that you are co-authoring with the LLM.”

    “A strong demonstration that the LLM and the ChatGPT character are distinct is that it’s quite easy to switch roles with the LLM. All the LLM wants to do is produce a conversation transcript between a User character and the ChatGPT character, it will produce whatever text it needs to to get there.”

    “my interlocutor is not really the ChatGPT character, but rather, an unfeeling robot hell-bent on generating dialogues.”

    “The real story isn’t that the chat bot is bad at summarizing blog posts—it’s that the chat bot is completely lying to you”

    “[AI companies] actually have very little control what text the LLM can and can’t generate—including text that describes its own capabilities or limitations.”

    “The LLM [is never truthfully refusing to answer a question]. It’s always just trying to generate text similar to the text in its training data”

    (Original Facebook post.)