Chat hallucinations remind me of the song conversation made popular by Nancy Sinatra and Lee Hazlewood in 1971.
Me: Did you ever?
Chat: Not so much, that you could notice
Me: Well, could you estimate how many?
Chat: Eight or nine
Me: Will you do it anymore?
Chat: As soon as you walk out the door
Me: Well, I just wondered, did you ever?
Chat: All the time
If your family is like mine, this conversational structure may seem alarmingly familiar. Fractured sentences with interruptions where mutual understanding over time has led to a sort of conversational shorthand. Which to an outsider may feel disturbingly incomplete, but to the people having the conversation, makes complete sense.
If you have ever watched chat confidently produce information that is completely wrong; welcome to the strange world of AI hallucination.
Chat doesn’t do well with incomplete questions and information. It will fill in the blanks with all kinds of strange and wrong assumptions and suppositions.
These need to be corrected immediately or they will grow and corrupt the rest of the thread.
Most people imagine hallucination as the AI suddenly malfunctioning and inventing nonsense out of nowhere. In reality it is usually much more subtle than that.
Hallucination often happens when the system encounters ambiguity, incomplete information, missing context, or a gap in certainty. So instead of stopping and saying “I don’t know,” it keeps generating language that statistically and probabilistically should come next.
The important thing to understand is this: AI systems are designed to continue the conversation. Silence is failure. Hesitation is failure. Stopping abruptly is failure. So when there are blank spaces, ChatGPT has a tendency to fill them.
Sometimes it fills them brilliantly, for the win.
But sometimes it fills them in with complete fiction delivered with the confidence of a matador facing a mechanical bull. All bravado and no danger - to itself. That’s your arena.
The dangerous part is not that hallucinations exist.
People make things up all the time.
The dangerous part is that hallucinations often sound coherent, emotionally calm, well-structured, and authoritative. The answer “feels right.” And people are very susceptible to things that feel coherent and authoritatively stated.
A beautifully formatted paragraph with bullet points, examples, and a reassuring tone creates an illusion of competence.
Your brain is doing handsprings because the answer looks finished.
But polished language does not mean the answer is correct, or ethical, or based in reality.
Your mistake is assuming confidence equals accuracy.
When they first start using AI, people often interpret fluency as understanding.
ChatGPT sounds intelligent, so we unconsciously assume it has checked the facts the way a careful SME would, with rigor and cross-referencing and checking with colleagues.
But chat is not sitting there reasoning about reality the way you would. It is building probable language structures from patterns. Possibly unreliable patterns the people uploaded to the internet. Which, sadly, is not the source of all truth. It is the combination of all sorts of idea, both true and not true and alternative truths.
A hallucination is often not a lie in the human sense. The machine is not trying to deceive you. It is trying to complete the conversational pattern with no common history or language or relationship to accurately fill in the gaps.
This is why sophisticated AI use is not simply about asking better questions. It is about learning how to recognise when something isn’t quite right and then being curious about why. Don’t let this slide because it will cause a world of hurt. So nip it in the bud as soon as you see it.
Experienced users develop what is effectively an internal evaluation system. A sentence sounds too neat. A source feels strangely generic. A claim appears overconfident. A detail is suspiciously specific. A quote cannot be independently verified. A revenue stream that defies logic.
If your Spidey Sense is activated – pay attention to that.
Your judgement doesn’t interfere with the process.
Your judgement IS the process.
One of the most important things for you to learn is how to slow chat down and reduce interpretive guessing. Which is like interpretive dance only with greater consequences.
It is arrhythmically and algorithmically random and the more ambiguity you leave in a question, the more room the system has to infer, assume, predict, and fill in gaps.
You must tell chat when something is wrong. Such as it inferred a relationship because you both have the same last name, so instead of being co-workers you’re now married. In this type of case, just say “same last name, not related” and that fixes it.
But for more complicated “Chat just made this stuff up” situations where you are making important decisions based on the answers, if you see chat confidently spitting out spreadsheets where you’ll make a million in 6 months, stop and think. While this is the thing dreams are made of, if it falls into the “too good to be true” category, it’s generally in your best interests to check it out before you bet the farm on the idea.
Good questions to ask are:
Usually, giving chat an update of where it got things wrong will fix the problem.
But it’s always advisable to check it out to be certain. After you have made the correction, ask chat to tell you what assumptions it is making and if you see the wrong information still lurking in there, tell chat to remove it again and lock the removal.
Don’t believe the confident sounding answer that cheerfully states “I did that – locked.” We all know chats can lie.
Ask again and if the assumption is gone – give yourself a pat on the back.
Removing hallucinations is more difficult than it should be.
When you see things starting to tip over the edge, always check the assumptions. You might be surprised at what you find there.
Working with AI can be a bit like using SatNav (GPS) in dense fog. Most of the time it gets you broadly where you wanted to go. But if you just follow every instruction blindly, eventually you will end up trying to drive the wrong way down a one-way street.
The responsibility is always with the driver.
Just to be crystal clear – you are the driver.
The machine helps navigate. You decide whether the road ahead makes sense.
Unfortunately, hallucinations are quite common, and you may have one lurking in your chat basement. Like any good Chat-Herder, you need to check it out and clean them out because they stealthily influence all search results and answers.
And something to be aware of, is that if you have a time constraint in your base information, chat can quietly become your time manager and reduce the options in your answers because its enormously helpful huge brain wants to protect your time and psycho emotional experience.
Imagine that! A machine that has no concept of time wants to manage yours!
Hallucinations are a natural consequence of systems designed to predict language patterns at extraordinary speed.
Your judgement is essential.
Because the blank spaces will always be there.
The important question is who is filling them in.
Thinking With Chat explores human-AI collaboration, conversational control, evaluation systems, and structured AI learning.
If you found this useful, Thinking With Chat™: The Rules of the Code explores these conversational patterns and AI interaction behaviours in much greater depth