A whetstone, not an oracle

There is an old question, can a machine think?, that has lately stopped being the preserve of philosophers alone and now concerns anyone who opens a text box in the morning and asks a language model something. And the honest answer to that question is more uncomfortable than either camp wants to sell you. The prophets say: it thinks, it will soon be superhuman, join in or lose out. The deniers say: it's a glorified parrot, a trick, nothing to see here. Both sell certainty, and certainty simply sells better than doubt. Neither sells you the truth.

The truth lies in the uncomfortable middle, and it is more useful than either extreme. These systems are genuinely astonishing and genuinely useful; they are also genuinely limited and make mistakes with a straight face. In one sense they understand a great deal; in another sense they lack the foundation, the direct, bodily bond with the world that humans have and they do not. They have, to put it vividly, the shadow of human knowledge, not the sun itself. They know how people write about the taste of salt; tasted it they never have.

The wrong question and the right one

The question "does it really think?" turns out to be the wrong question, not because it is uninteresting, but because it is undecidable and gets you no further in practice. The more fruitful question is: how do you collaborate well with it? What belongs to the machine, and what to you? To the machine: the fast processing of enormous quantities of text, the thinking along, the drawing of connections, the contradicting, the first draft, the tenth variant, the patient explanation at two in the morning. To you: the judgement, the responsibility, the direct bond with people and the world, the moral weighing, and the last word.

A language model is an excellent thinking partner and a poor boss; a brilliant apprentice and a dangerous autopilot. It is at its best not when you believe it, but when you use it to think more sharply and then check its work afterwards. A contradictor that never tires, a mirror that talks back. Not an oracle. A whetstone.

Why that distinction is everything

The danger lies not in what the machine can do, but in what we do with it when we forget where its limit lies. Whoever blindly adopts the outcome replaces their own thinking with that of a system that can produce nonsense with conviction. Whoever uses the outcome as raw material, as a starting point to build on, to push back against, to sharpen against, multiplies their own capacity without handing it over. The difference between those two attitudes is the difference between a tool and a crutch.

A thought to carry with you: the next time a system gives you an answer, don't ask "is this correct?" but "how do I check this?". The first makes you dependent. The second makes you sharper. The machine is the whetstone; the knife, and the hand that holds it, remains you.