It is predicting, not looking up

Unless it is visibly searching the web, an AI tool does not go and find the answer when you ask it a question. It produces the most likely next words, given everything it has read and everything you have said. For most questions, the most likely words are also the true ones, because the truth is what it has seen most. For some questions, the most likely words are a plausible-sounding thing that has never been true, and it produces those with exactly the same fluency.

This is why the errors are so hard to spot. A human who is unsure sounds unsure. A model that is unsure sounds the same as one that is certain, because it is not experiencing certainty at all. It is completing a pattern.

The practical consequence: fluency is not evidence. A well-written answer is a well-written answer. Whether it is true is a separate question, and you have to ask it separately.

In practice. A policy officer asked an AI tool for the date a particular regulation came into force. It gave a date, a section number and a short quotation from the explanatory notes. All three were wrong. The regulation existed; the details were assembled from what regulations of that kind usually look like.

She caught it because she happened to have the document open. She now says she assumes every specific date, number and quotation is invented until she has seen it in the source.

At your desk. Ask your AI tool three specific factual questions from your own field where you already know the answer: a date, a figure, a name. Note how confident each answer sounds, and how many are correct. The tone will not vary. The accuracy might.

Write it down.

• My three test questions:
• How many were right:
• Did the wrong one sound any different: