“We obtained records showing how a Department of Government Efficiency staffer with no medical experience used artificial intelligence to identify which VA contracts to kill. “AI is absolutely the wrong tool for this,” one expert said.”
“Lavingia’s system also used AI to extract details like the contract number and “total contract value.” This led to avoidable errors, where AI returned the wrong dollar value when multiple were found in a contract. Experts said the correct information was readily available from public databases.”
I summarize medical information for doctors, researchers, and patients every day for a living, and I can promise you that any summary you get from chatGPT will have at least one significant error. And how could you possibly know? If you don’t understand what your doctor is telling you, how could you effectively vet the summary for errors?
when summarizing scientific texts, LLMs may omit details that limit the scope of research conclusions, leading to generalizations of results broader than warranted by the original study. […] Even when explicitly prompted for accuracy, most LLMs produced broader generalizations of scientific results than those in the original texts[…] In a direct comparison of LLM-generated and human-authored science summaries, LLM summaries were nearly five times more likely to contain broad generalizations[…] Notably, newer models tended to perform worse in generalization accuracy than earlier ones. Our results indicate a strong bias in many widely used LLMs towards overgeneralizing scientific conclusions, posing a significant risk of large-scale misinterpretations of research findings.
“Reading list [created by generative AI] in advertorial supplement contains 66% made up books by real authors.”
Apparently not created by the Sun-Times:
“The reading list appeared in a 64-page supplement called ‘Heat Index,’ which was a promotional section not specific to Chicago. Buscaglia told 404 Media the content was meant to be ‘generic and national’ and would be inserted into newspapers around the country.”
Generative AI company Anthropic tests its “Chain-of-Thought” “reasoning models” to see whether they’re “faithful”—that is, to see whether the models accurately report the steps that they’re following. Turns out that they don’t.
“Reasoning models are more capable than previous models. But our research shows that we can’t always rely on what they tell us about their reasoning. If we want to be able to use their Chains-of-Thought to monitor their behaviors and make sure they’re aligned with our intentions, we’ll need to work out ways to increase faithfulness.”
“A new wave of ‘reasoning’ systems from companies like OpenAI is producing incorrect information more often. Even the companies don’t know why.”
“[OpenAI] found that o3 — its most powerful system — hallucinated 33 percent of the time when running its PersonQA benchmark test, which involves answering questions about public figures. That is more than twice the hallucination rate of OpenAI’s previous reasoning system, called o1. The new o4-mini hallucinated at an even higher rate: 48 percent.”
The little kid shouted, “The emperor has no clothes!”
The other citizens all glared at the kid. “In the future, the emperor’s clothes will be awesome!” they said. “So awesome that they will solve all of our problems, including problems that have nothing to do with clothes. Besides, it’s really our fault—if we just looked at the emperor from the right angle, he would surely have clothes, so we need to get better at guessing how to look at him.”
This is a post about generative AI.
(This post was brought to you by the people who I’ve heard say things like “The answers I get are usually wrong, but that’s just because I haven’t learned how to write a good prompt yet.”)
I try to avoid reading the AI Overviews when I do Google searches that show them. But this time I happened to glance at the AI Overview. And it was blatantly false.
I searched for [sum of three “consecutive cubes”]. Here’s what the AI Overview told me:
The sum of three consecutive cubes is always equal to the square of the sum of those three numbers; in other words, if you take any three consecutive integers, add them together, then square that sum, you will get the same result as adding the cubes of those three numbers.
Example:
• Consider the numbers 1, 2, and 3. • Adding them gives: 1 + 2 + 3 = 6 • Squaring the sum gives: 6^2 = 36 • Adding their cubes gives: 1^3 + 2^3 + 3^3 = 1 + 8 + 27 = 36
Key points:
• This property holds true for any set of three consecutive integers. • You can use this fact to quickly calculate the sum of three consecutive cubes without having to individually cube each number.
It’s true that this pattern holds true if you happen to pick the numbers 1, 2, and 3. Super cool!
So let’s look at the very next set of three consecutive integers: 2, 3, and 4.
What’s wrong is that the AI Overview’s claim is false for all cases except for (1,2,3), (0,1,2), and (-1,0,1). There’s one other case where the sum of the cubes of three consecutive integers is a perfect square (of a different number), but the only cases where it’s specifically the square of the sum of those three consecutive integers are (1,2,3), (0,1,2), and (-1,0,1).
(I’m reminded of a joke proof that all odd numbers are prime: 3 is prime; 5 is prime; 7 is prime; therefore, by induction, all odd numbers are prime.)
Google’s AI Overview links to sources to support its statements. The sources that it linked to in this case are the following. Not one of them makes the claim that the AI Overview is making.
https://iitutor.com/proving-sum-of-consecutive-cubes-formula/ (“The sum of [the first] n consecutive cube numbers [starting with 1] is equal to the square of the [first] n numbers [also starting with 1].” That’s nifty, but it’s a very different claim than Google’s.)
As usual, the moral of this story is: Don’t believe anything that generative AI tells you.
I often remember to add ” -ai” to the ends of searches (to tell Google not to give an AI Overview), but I often don’t.
But I’ve been hearing reports that that isn’t working any more for some people. And, indeed, when I re-run this search with -ai, sometimes it gives me an AI Overview and sometimes it doesn’t.
Interestingly, the specific contents of the AI Overview vary—if I do the search without -ai, I get the one that I posted about, but if I do the search with -ai, I get a different claim (that doesn’t mention squares) that I haven’t checked yet.
Adding swear words does still seem to work to remove the AI Overview. It also provides search results that use those swear words, which may or may not improve your search results, depending on what kinds of search results you want.
The reason I was doing this search was that an interesting fact about a particular number was mentioned in a recent TV show. I had played around with the relevant numbers a bit, and had reached the point where I was curious about what work had been done on this topic.
If I hadn’t tried out the sums of cubes of three consecutive numbers on my own just prior, I might have been tempted to believe what the AI Overview said. But because I had just calculated several such answers myself, I knew immediately that the AI Overview was wrong.
