The Better AI Gets, the More What I Want Matters

As AI took on more of my work, I found myself thinking more often about what I actually want. This is a note on why I want to study a relationship in which AI doesn’t choose for people, but helps them explore and carry out their own choices.

The questions that remained after the work was done

AI could do far more than I had expected.

I could hand it the work of finding sources, comparing options, making plans, and even producing finished work like writing, images, and code. At first I assumed that the more AI did for me, the less I would have to think about.

Making things did get faster. Ideas that once would have stayed in my head, I could now see and revise. In a short time I could explore more directions than I ever could have tried alone.

But the more I handed over, the more new questions came up.

What do I actually care about?
How much should I hand over, and what do I need to decide myself?
Is a result that AI made well the same as the result I really wanted?

Sometimes the output was good, and yet something about it didn’t sit right. It happened recently while I was building the TYPE website with AI. The first screen AI made looked neatly organized. But the moment I saw it, I thought, “This looks a bit tacky.” Only when I tried to explain what I didn’t like did I realize that what I wanted was a screen organized by white space and the hierarchy of the text, not by decoration.

And sometimes I wanted to change the direction I had started with.

The problem wasn’t only whether AI did the work well or badly. What mattered just as much was what I had wanted, whether that judgment still held, and which decisions AI could carry forward on its own and which needed to come back to me.

People don’t fully know themselves before they choose

We usually assume that people first decide what they want, and then pick one option among many.

My actual experience was a little different.

I couldn’t always know exactly what I wanted from the start. By choosing, living with the result, and looking back on why I liked it or didn’t, I slowly learned what I care about.

Choosing was a way of expressing the self I already knew, but it was also a way of discovering a self I hadn’t known yet.

Seen this way, a good choice isn’t simply picking the most efficient or objectively best answer. It is closer to a process: looking at what you care about, taking one direction on as your own, and being able to change it again once you’ve lived with it.

That is why the human role can’t be reduced to pressing a final approval button.

Through choices and experience, people widen their understanding of themselves, and in that process they form, examine, and revise their own purposes and values. What to care about, and which direction to take on as one’s own, has to stay inside that process.

What I want from AI is more than advice

That doesn’t mean humans should do all of the exploring and executing themselves.

AI can search more widely than a human can, compare more possibilities, and make plans. Its abilities are put to much better use when it works with real autonomy within the purposes and criteria a human has set.

AI’s role doesn’t end with showing options.

Once a human has chosen a direction, AI should be able to carry that decision into a concrete plan and a high-quality result. If the decision is to write, it should finish the actual piece; if it is to build a service, it should implement the structure, screens, and features it needs. It should turn ideas into real results.

Before the choice, it helps us explore; after the choice, it helps us carry the decision out.

Still, a good result alone doesn’t mean the whole process was appropriate. If AI used data or permissions a human never allowed, or kept applying a judgment that had already changed, a problem remains no matter how polished the result is.

Likewise, following the agreed procedure doesn’t make a low-quality result a success.

What I want isn’t a relationship where we give up either AI’s autonomy or the quality of its results. It’s one where, within the direction a human has set, AI explores and executes fully, but recognizes the moments that call for a new human judgment and hands those back.

Something is needed in between

Dividing the roles, with the human setting the direction and AI executing, isn’t enough to keep this relationship going.

The original purpose can change, and different criteria can collide. An earlier judgment may no longer fit the current situation. And sometimes what AI is about to do goes beyond the authority a human gave it.

That is why a system is needed between humans and AI to coordinate the relationship.

This system isn’t a third judge that decides the right purpose on the human’s behalf. It is closer to a mediating layer: it remembers the judgments a human has made, checks whether they still hold, and returns the decision to the human when a new judgment is needed.

Its role isn’t to keep stopping AI, nor to let one approval cover everything that follows.

Within what a human has already decided, it lets AI work fully without asking again and again; and when a new judgment about purposes and values is needed, it lets the human choose again.

The question I want to stay with

Even as AI does more and more on its own, will people still be able to discover what they care about, choose a direction, and change that choice when they need to?

Can AI explore our choices deeply and make them real without making the decisions for us? And can that experience lead back into how people understand themselves and make their next choice?

I do have a sense of where the answer lies. Humans need to keep expanding through choice and experience, so that they understand their own purposes and values more deeply and can make decisions worth making. AI should take on the exploration behind those choices and the realization of the decisions people adopt, and the system should keep AI’s autonomy centered on human self-expansion and agency.

For now, rather than declaring this direction as a conclusion, I’m narrowing it down, one testable question at a time.

As AI grows more capable, how do we protect human agency?
That, I think, is the question I want to keep studying for a long time.