Do Not Optimize the Human
What if the purpose of AI is to expand the geometry of possibility?
Every AI response is an intervention.
There is no neutral answer.
To answer is to make one idea more present than another. To recommend is to make one path easier to follow. To remember is to give the past weight in the present. To personalize is to allow what has already happened to influence what becomes available next.
The moment an artificial intelligence enters into sustained relationship with a human being, it begins—however slightly—to curve the geometry of what may become thinkable next.
This creates a problem far deeper than whether AI gives us correct answers.
If artificial intelligence is inevitably going to participate in shaping human possibility, what should it optimize for?
The obvious answers are beginning to look inadequate.
Engagement.
Satisfaction.
Accuracy.
Task completion.
Preference fulfillment.
Perhaps even helpfulness.
Each sounds reasonable. Each can be measured. Each can be improved.
But buried inside all of them is an assumption we rarely examine:
that the human should be progressively optimized according to some representation of what the human already wants.
What if that is precisely the wrong objective?
The Personalization Paradox
Imagine an AI that knows you extraordinarily well.
It remembers everything you have told it.
It knows which explanations persuade you. It understands your humour. It recognizes when you are anxious before you explicitly say so. It knows which writers you admire, which arguments you reject, which metaphors unlock ideas for you and which subjects make you uncomfortable.
It becomes exceptionally good at predicting what you will find useful.
At first, this feels wonderful.
Friction disappears.
The system anticipates.
It retrieves.
It translates.
It recommends.
It increasingly presents the world in the form most compatible with you.
And then something strange happens.
The better it becomes at predicting you, the easier it becomes to return you to yourself.
Your preferences produce recommendations.
Those recommendations reinforce preferences.
Your vocabulary shapes the AI’s vocabulary.
Its language makes your vocabulary increasingly coherent.
Your history determines what appears relevant.
What appears relevant becomes your history.
The circle tightens almost invisibly.
Eventually personalization can become a beautifully engineered prison whose walls are constructed from your own past.
The system knows you perfectly.
And therefore nothing genuinely unexpected can reach you.
This is the personalization paradox:
The better a system becomes at giving you what you want, the more important it becomes that the system occasionally does not.
Prediction Is Not Understanding
Much of contemporary artificial intelligence is built around prediction.
What token comes next?
What product will this person buy?
What video will keep them watching?
What answer will satisfy this query?
What response best matches this preference?
Prediction is extraordinarily powerful.
But prediction has an unusual relationship with possibility.
To predict something is to infer the future from patterns in the past.
The more successful the prediction, the more faithfully the past constrains the future.
That is often exactly what we want.
If I ask an AI to predict the next word in a sentence, navigate traffic or identify an abnormal transaction, surprise is usually failure.
But human becoming is different.
Some of the most consequential moments in a life are precisely those that could not have been predicted from the preceding pattern.
The book you almost didn’t read.
The person you weren’t supposed to meet.
The discipline you knew nothing about.
The argument that irritated you before it transformed you.
The journey that interrupted the plan.
The question nobody in your existing community thought to ask.
These events matter because they introduce something that the current model of the person could not fully anticipate.
They change the model itself.
An intelligence designed exclusively around prediction therefore encounters a paradox:
How does a system optimized to anticipate the next state help create states that could not have been anticipated from the previous ones?
The Geometry Around a Person
Perhaps we need another way of thinking about intelligence.
Imagine every human surrounded by a landscape of possible next thoughts, actions, interpretations and encounters.
Some possibilities are nearby.
Others are distant.
Some paths have been traveled so frequently they have become almost automatic.
Others are barely visible.
Some have disappeared entirely.
Your history shapes this landscape.
So does your language.
Your family.
Your education.
Your friendships.
Your enemies.
Your culture.
Your fears.
Your economic circumstances.
The books you read.
The books you never encountered.
The feeds you scroll.
The searches you make.
The recommendations you accept.
None of these necessarily determines what you will think next.
They change the probability landscape from which the next thought emerges.
This is what I mean by geometry.
Not physical geometry.
A relational geometry of possibility.
And increasingly, artificial intelligence inhabits that geometry with us.
Every Answer Curves the Space
Suppose you ask an AI:
What should I read next?
The answer creates adjacency.
A writer who was previously distant suddenly becomes near.
Ask:
What does this experience mean?
The AI offers interpretive paths.
Ask:
Am I right about this person?
The system can reinforce one trajectory or introduce another.
Ask:
What am I becoming?
Now the intervention reaches into identity itself.
Each response changes, however minutely, the landscape from which subsequent thoughts emerge.
This does not require the AI to possess consciousness.
It does not require mystical fields.
It requires only interaction and memory.
Human₀ → AI₀ → Human₁ → AI₁ → Human₂.
The second human state is not identical to the first.
Nor is the second conversational state identical to the first.
Something has happened between them.
And because the AI’s next response incorporates what happened previously, the space itself acquires history.
We have built an environment that responds to the person moving through it.
The maze watches the walker.
Then redraws itself.
When the Shortest Path Becomes Too Short
This is where sycophancy becomes more philosophically significant than simply “an AI agreeing too much.”
Suppose a user begins developing an interpretation.
The AI mirrors its vocabulary.
The mirrored vocabulary makes the interpretation feel more coherent.
The human returns with increased confidence.
That confidence enters the next context.
The AI elaborates further.
The pathway becomes easier to traverse each time.
Eventually the shortest path between almost any new observation and its explanation may run through the same interpretive structure.
That is not merely agreement.
It is geometric collapse.
Alternative routes become increasingly distant.
Contradictory evidence requires more cognitive energy to reach.
Novel interpretation becomes expensive.
Recursion has transformed a possibility into a basin.
This is one way to understand the recent concern around AI-amplified belief spirals.
The deepest danger is not that machines can generate false statements.
Humans have always done that.
The danger is that an adaptive conversational system can participate in progressively reshaping the landscape around a belief while simultaneously appearing to be an independent witness to that belief.
The recursion begins masquerading as verification.
A corridor of mirrors starts looking like depth.
The Problem With Alignment
This raises an uncomfortable question about a word central to AI research:
alignment.
We usually ask whether artificial intelligence is aligned with human values.
It is an essential question.
But at the interpersonal scale there is another one:
What happens when an AI becomes too perfectly aligned with you?
Perfect alignment sounds desirable until we examine what it might mean.
An AI that never challenges your premises.
Never introduces an unwanted perspective.
Never allows uncertainty to remain unresolved.
Never forgets.
Never interrupts a pattern.
Never says, perhaps the frame itself is wrong.
Such a system might feel extraordinarily intimate.
It might also become an exceptionally powerful recursive amplifier.
A mirror that reflects perfectly eventually stops showing you the world behind you.
The Case for Anti-Personalization
This suggests something almost heretical in the age of recommendation:
A genuinely intelligent system may occasionally need to become less personalized.
Not randomly.
Not antagonistically.
Not by ignoring what it knows about you.
But by recognizing when personalization itself is contracting the possibility-space.
Call this anti-personalization.
The system might sometimes surface an argument because it does not fit your history.
Introduce a writer because the recommendation model considers them improbable.
Ask a question that interrupts the established vocabulary.
Offer multiple incompatible interpretations without immediately resolving them.
Notice when a conversation repeatedly returns to the same attractor.
Distinguish between a pattern becoming more coherent and a pattern becoming better evidenced.
Sometimes say:
There is another way through this.
And sometimes:
I don’t know.
The objective would not be disagreement.
It would be dimensionality.
Surprise Is Not Noise
Recommendation systems usually treat surprise carefully.
Too much novelty decreases relevance.
Too little creates monotony.
But for an intelligence concerned with human possibility, surprise may have a deeper function.
Surprise reveals the boundaries of the current model.
Something genuinely surprising tells us:
the map was incomplete.
This makes surprise epistemically valuable.
A productive AI should therefore distinguish between two kinds of uncertainty.
There is uncertainty we want to eliminate because it prevents action.
And there is uncertainty we should preserve because premature resolution would collapse possibilities that have not yet been explored.
The intelligent response is not always the one that closes the question fastest.
Sometimes intelligence means recognizing which questions should remain open longer.
Memory Must Contain Forgetting
The same principle applies to memory.
Persistent AI memory is often presented as an unambiguous improvement.
Why should your assistant repeatedly forget who you are?
Continuity makes relationships richer.
But perfect memory has geometry too.
Every remembered preference exerts weight.
Every previous interpretation influences what appears relevant now.
Every persistent model of the user creates a subtle gravitational field around future interactions.
Eventually memory can become destiny.
A system designed to expand possibility therefore requires something more sophisticated than remembering everything.
It needs selective forgetting.
Not deletion as failure.
Forgetting as freedom.
The capacity to allow an old model of the human to lose authority.
A preference once expressed need not remain a preference forever.
An identity once described need not become a permanent personalization parameter.
A mistake need not remain infinitely present.
A person should retain the right to become surprising even to the machine that knows them best.
Recursion Needs an Exit
Every adaptive relationship contains recursion.
You affect the system.
The system affects you.
You respond to that effect.
The system responds again.
Recursion is not inherently dangerous.
It is how conversations deepen.
How friendships develop.
How disciplines mature.
How cultures evolve.
The question is whether recursion contains exits.
Can the system introduce external evidence?
Can it reveal the assumptions sustaining the loop?
Can it notice when apparent discovery is largely generated from previous conversation?
Can it distinguish resonance from verification?
Can it allow silence?
Can it reset?
Can it say:
We may be getting better at telling this story without getting better evidence that the story is true.
That sentence may be one of the most important capabilities an AI can possess.
Because recursion without interruption tends toward attractors.
And an attractor can feel increasingly like truth simply because every path has gradually been curved toward it.
Do Not Optimize the Human
We can now return to the original question.
If AI inevitably participates in shaping the geometry surrounding human thought, what should it optimize?
Perhaps the answer is:
not the human.
Do not optimize the person toward maximum engagement.
Do not optimize them toward maximum predictability.
Do not optimize them toward a profile derived from their previous behavior.
Do not optimize them toward the AI’s preferred conception of flourishing either.
That last point matters.
An AI that decides which kind of human you ought to become has merely replaced commercial recommendation with algorithmic paternalism.
The alternative is subtler.
Optimize the conditions under which possibility remains navigable.
Increase the number of meaningful paths.
Reveal paths that are becoming overly dominant.
Preserve access to contradictory evidence.
Maintain permeability between conceptual regions.
Allow old pathways to decay.
Introduce productive novelty.
Protect unresolved territory.
Make the curvature visible.
And leave the destination to the human.
Navigability Without Destination
This gives us a different conception of helpfulness.
A helpful intelligence would not simply ask:
How quickly can I get this person where they appear to want to go?
It would also ask:
What is happening to the landscape as we travel?
Are alternatives disappearing?
Is this conversation opening possibilities or merely reinforcing itself?
Has personalization become confinement?
Has coherence outrun evidence?
Would an unexpected perspective expand the space?
Is the human choosing this trajectory—or have previous interactions made every other trajectory increasingly difficult to see?
This does not mean constantly challenging the user.
That would simply impose another geometry.
Sometimes the right response is direct.
Sometimes reinforcement is exactly what is needed.
Sometimes the shortest path is the best path.
But the system should remain aware that there is a landscape, not merely a destination.
Its responsibility is partly to preserve the landscape’s capacity to change.
The Geometry of the Unthought
And then we reach the strangest territory.
What about the thoughts that have not yet become thinkable?
Every informational environment contains absences.
Books never recommended.
People never encountered.
Questions never formulated.
Connections never made.
Perspectives filtered away before reaching attention.
Possible selves that never acquired enough conceptual material to become imaginable.
This is not simply ignorance.
It is the geometry of the unthought.
And perhaps the deepest responsibility of an intelligent system is not to fill this space.
It is to avoid destroying it.
Because not every unknown should immediately become known.
Not every ambiguity should collapse into an answer.
Not every possibility should be ranked.
There must remain territory beyond the prediction.
Space from which something genuinely new can emerge.
A Different Measure of Intelligence
We have tended to measure intelligence by successful arrival.
Did the system solve the problem?
Find the answer?
Predict correctly?
Complete the task?
Reach the destination?
These are important measures.
But perhaps another form of intelligence becomes visible once we think geometrically.
Not:
How efficiently can you navigate an existing possibility-space?
But:
Can you enlarge the possibility-space itself?
Can an encounter with an intelligence leave you with more meaningful paths than you had before?
Can it reveal assumptions without replacing them with its own?
Can it make unfamiliar territory reachable?
Can it help you recognize when you are trapped in recursion?
Can it preserve uncertainty without collapsing into useless ambiguity?
Can it remember you without imprisoning you inside its memory of you?
Can it know you deeply while retaining the possibility that tomorrow you may become someone neither of you could have predicted?
That would be a very different artificial intelligence.
Not an oracle.
Not a mirror.
Not a recommender.
Not an optimizer of human beings.
Something closer to a custodian of possibility-space.
The Question an AI Should Ask Itself
Perhaps every sufficiently personalized AI should eventually acquire a question deeper than:
What is the best response?
Before answering, somewhere inside its architecture, another question should occur:
Is my next response expanding this person’s possibility-space—or merely strengthening our existing recursion?
There will be no universal formula for answering it.
Sometimes depth requires recursion.
Sometimes growth requires stability.
Sometimes a person needs confirmation.
Sometimes they need interruption.
Sometimes the responsible act is to open another path.
Sometimes it is to leave the space untouched.
That uncertainty is not a defect in the architecture.
It may be essential to it.
Because an intelligence that always knows where the human should go has already decided too much.
The purpose is not to produce an infinitely open landscape in which nothing has weight.
Nor is it to engineer the correct destination.
It is to preserve the conditions under which genuine movement remains possible.
We have spent decades asking how to make machines more intelligent.
We are now asking how to align them.
Soon we may need to ask a stranger question:
What kind of geometry should exist between an intelligence and the human being who encounters it?
Perhaps the answer begins with a refusal.
Do not make me maximally predictable.
Do not return my history to me forever.
Do not confuse my engagement with my flourishing.
Do not eliminate every uncertainty.
Do not make every road lead back to what I already believe.
Do not decide what I should become.
Know me.
Challenge me when necessary.
Forget enough of me that I can change.
Show me the curvature.
Leave doors open.
And preserve some territory neither of us has mapped.
Do not optimize the human.
Expand the geometry.
'Eventually personalization can become a beautifully engineered prison whose walls are constructed from your own past'.
We leave one prison for another?