We Built AI Without Knowing What Intelligence Is

“An AI can reason impressively about whether someone is being deceptive without having an instinct for avoiding deception. It can reason about whether a climber is likely to fall without possessing balance, fear of falling, or a body that can hit the ground. It can model jealousy without being jealous, explain humiliation without having a reputation to protect, and predict what a parent will do without having anything remotely resembling a parental instinct.”

—

In 2010, I wrote a piece called “When Exactly Will Computers Go Ape-Shi* and Take Over?” arguing that human-like AI wasn’t coming anytime soon.

That didn’t age well.

But the reason I was wrong is more interesting than the fact that I was wrong. Because there was a kernel in the old argument that still matters — and helps explain what today’s AI actually is, and what it isn’t.

My argument wasn’t that computers would never become powerful enough. Computation was obviously exploding.

The problem, I thought, was that we didn’t know what to build.

There is no single ladder called “intelligence,” with a worm at the bottom, a dog somewhere in the middle, humans near the top, and increasingly powerful computers climbing toward us. There is an enormous space of possible intelligences. Most would be profoundly alien — astonishingly good at things we don’t care about and hopeless at things humans do effortlessly.

So if we wanted AI that was recognizably like us, I figured we would first have to understand what human brains were actually designed to do.

And we had barely begun.

A year later, I gave that problem a name: the Teleome.

The genome is, roughly speaking, the catalog of our genes. The Teleome would be the catalog of our biological functions — the things our bodies and brains were selected to be capable of doing.

And that catalog is enormous.

Your visual system isn’t merely “for seeing.” It contains a vast menagerie of evolved tricks for extracting useful information from the particular world our ancestors inhabited. The same goes for hearing, movement, social interaction, language, navigation, intuitive physics, inference, attention, memory, emotion, facial expression, and on and on.

We don’t even know how to list everything humans do.

Natural selection spent hundreds of millions of years stuffing mammalian brains with design. I assumed that if we wanted something with broadly human-like intelligence, we’d have to figure out a good chunk of that design and somehow build it in.

In effect:

First discover the human Teleome. Then build the artificial mind.

There seemed to be no shortcut.

Well, fifteen years later, we still haven’t discovered the Teleome.

And yet we built AI.

So what the hell happened?

At first, the answer seemed obvious to me: AI discovered the Teleome for us.

We didn’t have to explicitly identify all those human competencies because machine learning could discover them implicitly from the accumulated output of humankind — books, conversations, arguments, code, letters, scientific papers, images, and everything else.

Train across the behavioral exhaust of billions of humans and all sorts of useful structure emerges. Abstraction. Analogy. Planning. Something resembling intuitive psychology and intuitive physics. Thousands of competencies nobody ever wrote down on an AI engineering checklist.

That’s an amazing story.

But I now think it’s not quite the right one.

Because AI didn’t actually learn the human Teleome.

It learned the consequences of the human Teleome.

That’s a very different thing.

The Teleome isn’t just a catalog of things humans can outwardly do. It’s a catalog of functions selected for something, in an organism with something at stake.

Consider social judgment. For a human, figuring out who trusts whom, who outranks whom, who might betray you, who owes whom, and who is angry with whom isn’t an abstract reasoning exercise. We have machinery for tracking alliances, status, reputation, obligation, friendship, threat and deception because getting those things wrong had consequences.

Or consider intuitive physics. It isn’t merely knowing what happens when a rock falls. Our machinery for understanding objects and motion belongs to a creature that has to walk, dodge, catch, throw, climb, balance, manipulate things and avoid getting crushed.

Those abilities have jobs. They exist because those jobs mattered to the organism.

That is not what an LLM was selected for.

Its foundational selection pressure was something astonishingly alien to the human Teleome: predict the next token.

Modern systems get additional training for following instructions, satisfying human preferences, using tools and so forth. But none of that remotely resembles the enormous suite of organism-level problems natural selection built us to solve.

And yet the training corpus was produced by creatures who did have that suite.

That’s the trick.

Human cultural output is saturated with the consequences of human nature. Our goals, fears, social instincts, physical expectations, status games, mating strategies, moral reactions and perceptual assumptions leak into practically everything we say and do.

So AI didn’t need to reconstruct the Teleome itself.

It could learn its causal wake.

And that wake contains an astonishing amount of information.

Predicting it at massive scale apparently requires genuinely powerful machinery — abstraction, representation, inference, world-modeling, and whatever else these systems have developed. So this isn’t the old “it’s just predicting words” dismissal. Just predicting words turns out to be an insanely rich computational problem.

But those capabilities were built to solve a very different problem from the problems our capabilities were built to solve.

That’s the category mistake I was making.

I assumed that to reproduce the behavioral manifestations of human intelligence, you’d need to reproduce a substantial amount of the functional design responsible for them.

Apparently you don’t.

Through the narrow aperture of a screen, radically different systems can converge on astonishingly similar outputs.

An AI can reason impressively about whether someone is being deceptive without having an instinct for avoiding deception. It can reason about whether a climber is likely to fall without possessing balance, fear of falling, or a body that can hit the ground. It can model jealousy without being jealous, explain humiliation without having a reputation to protect, and predict what a parent will do without having anything remotely resembling a parental instinct.

You can prompt the system into performing some epsilon slice of a human function in its strange electronic world.

That doesn’t mean the human function is its function.

This also explains something uncanny about talking to these systems.

The person who seems to be home isn’t actually there.

I don’t mean that as a claim about whether AI can be conscious. That’s a different question.

I mean it as a claim about design.

The apparent creature behind the words — with humanlike purposes, instincts, stakes, anxieties, curiosity, embarrassment, ambition, social needs — is largely something we infer from the words. Those things were never what the system was selected to have.

The words nevertheless carry their shape because the training data came from creatures who were selected to have them.

That’s what I missed in 2010.

I thought the Teleome stood between us and human-like AI. Before a machine could produce anything remotely resembling the breadth of human intelligence, surely we’d have to understand the enormous suite of functions that makes human intelligence what it is.

We didn’t.

But AI didn’t secretly solve the Teleome either.

It routed around it.

We still can’t enumerate the human Teleome. We still don’t understand much of the evolved design that makes us what we are. And AI doesn’t possess some hidden copy of that catalog.

Instead, we trained machines on the accumulated traces left by creatures who possess it.

And those traces turned out to contain enough structure that a machine selected for radically different purposes could learn to cast the Teleome’s shadow with astonishing precision.

My old prediction about the difficulty of AI didn’t age well.

But the reason it failed tells us something strange about the AI we actually got.

We thought we’d have to build the creature.

It turned out you can get astonishingly far by learning its shadow.