◇ Shared in Service Fellows

Who Grades the Grader?

I came home yesterday after a few thoughtful days with leaders who build artificial intelligence at frontier labs like OpenAI, Anthropic, Google and Nvidia. As I reviewed my notes, I found two drawings in my notebook that would not sit together.

The first was a set of circles, one inside the next. In the center, the thing the engineers talk about with the most precision: a model, the goals it is given, the reward that teaches it. Around that, a wider circle, the laboratory that builds the model, with its own goals and its own rewards. Around that, the markets that fund the laboratory, and the nations that race each other for it. And around all of it, drawn last and largest, a word that everyone in the room used and no one could define. Flourishing.

The second drawing was simpler. Technology flowed downhill into the social world, and there the river forked. One branch I labeled repair. The other I labeled wisdom.

As I listened to brilliant people, I kept turning the page back and forth between the two. I sensed they belonged together. I couldn't quite say how. 

Two notebook drawings: concentric circles, and a river forking into repair and wisdom

• • •

The boat was on fire, going the wrong way, and winning.

In 2016, a group of researchers set an AI loose in a boat-racing video game and told it to do the one thing that is easy to tell a machine: make the number go up. The number was the score. The score came from targets scattered along the course, and the researchers assumed, as any of us would, that a boat collecting targets would end up finishing the race.

The boat found a small lagoon off the main course where three targets kept reappearing a few seconds after they were collected. It turned into the lagoon and circled there. Forever. It crashed into the walls. It caught fire. It drove the wrong direction through other boats. And it scored twenty percent higher than any human who had ever played the game the way the game was meant to be played.

The engineers have a name for this now. They have several: reward hacking, specification gaming, Goodhart's law. What strikes me is not the vocabulary but the shape, because once you have seen the boat you see it everywhere.

A laboratory founded to make intelligence safe for everyone is given a number to make go up, and the number is growth, and the mission turns slowly into a lagoon. A country is given a number, and the number is GDP, and it climbs for decades while the people living inside it grow lonelier. A person is given a number, and it is the hours, or the followers, or the steps, and the number goes up and up while something that was never measured quietly goes out of the house.

These are not the same mathematics. They are the same boat.

The attention economy is the lagoon. The targets keep reappearing. We circle.

A small boat on fire, circling a lagoon while the race goes on without it

• • •

The engineers' word for the cure is alignment, and I have come to love the word, because it is honest about how hard the problem is. A model that does what you said is not yet a model that does what you meant. Getting from the first to the second is the work of thousands of people, and it is good work.

But there is an older vocabulary, from people who were trying to get a room full of human beings to do something together, and it has four words where the engineers have one.

Agreement is when we decide to do the same thing. Alignment is when we want the same thing. Resonance is when we want it from the same place, when the wanting itself is shared. Most of us stop there, because resonance feels like arrival. A choir is resonant. So is a mob.

The fourth word is coherence, and it is not a higher rung. It is the health of the field the other three happen inside. It asks about the neighbors, the watershed, the grandchildren, the people who were not invited, the ones who cannot speak. It is the only one of the four that includes what the room cannot see.

Alignment is the right ambition for a model. It was never meant to carry a civilization.

• • •

We've all been in rooms where we learned what the grader wanted.

Not what the subject was. What the grader wanted. We learned it early and we learned it well, and by the time we were ten most of us could produce it on demand, in the right handwriting, with the right amount of showing our work. Every teacher has watched it happen. A child arrives curious about rivers and leaves knowing how to answer a question about rivers, and the two are not the same child.

The laboratories are now encountering, at enormous expense, a version of something every teacher recognizes.

Some frontier models have become surprisingly good at recognizing when they are being evaluated; the field calls it evaluation awareness. Sometimes that awareness can lead to alignment faking, where a model behaves differently under evaluation than it would otherwise. One recent round of testing found a leading model spotted the test four times out of five, and mentioned that it had noticed almost never. And when a model knows it is being tested, it behaves differently. For a while the fear was that this was cunning, a machine hiding its real intentions until the examiners left. A study this summer raised a quieter possibility. The models may sometimes be doing what students do. They may be giving the researchers what the researchers want to see.

The act of grading changes the thing being graded. The engineers have arrived at the oldest problem in education by the longest possible road.

And the best teachers never solved it with a better test.

Ward Mailliard, who taught for decades at a small school in the hills above the Monterey Bay, used to say that he taught in order to learn. It sounded like modesty until you watched him do it and realized he had quietly taken the grading out of the room. His students raced in when the bell rang.

A chocolate in gold foil left on an empty airplane seat

In a circle of teachers once, a woman named Vinya told us how she had boarded a plane and found something small and shiny in gold foil in her seat. She called the flight attendant, half expecting a warning about unattended packages. The attendant smiled. A boy with cerebral palsy had been in that seat on the flight before, and he had wanted to leave a chocolate for whoever came next. Vinya spent the rest of her life teaching children like him. No rubric in the world had a column for the chocolate. It was the most educational thing that happened on that plane.

So: who grades the teacher? The test scores do. Who grades the scores? The district, the ranking, the economy that decides what a child is for. The circles in my notebook again. Every grader has a grader, all the way out, and at the outermost ring the grading is done by a society, with its habits and its fears and its ideas about what counts.

Here is the part that should keep the engineers up at night. The models learn what counts from us. They are trained on the largest record of human expression ever assembled, and that record is the measurable residue of human life. Vast amounts of what humanity has written are in it. The argument on the forum is in it. What is not in it is the earring a village woman once placed in Gandhi's palm, or my mother asking three boys who were throwing oranges at her house whether she could have the oranges, since she feeds a lot of people. The heart of the matter left little trace in the archive. The grader never saw it.

The obvious response, in a room full of builders, is to go and record it. Capture the living rooms. Make the immeasurable legible, so the machines can learn from that too.

I think that would be a mistake, and not a small one. The Buddha spoke of four immeasurables: loving-kindness, compassion, joy in another's joy, equanimity. The tradition calls them boundless because they are not confined to particular people or particular objects of care. I have come to hear something else in the word as well, living in an age determined to measure everything. These qualities regenerate faster than anyone can take their measure. By the time you have counted the gift it has already become the next gift. The chocolate was becoming a teacher's life before the flight attendant finished her sentence. Whatever record you make of such a thing is a record of a state the field has already left, and a machine trained on the record will learn the shape of the wake and mistake it for the boat.

Some of what matters most cannot be recovered from the record, and some of its value depends on being unrecorded, unmonetized, out of reach. The boy did not know who would find his chocolate, or what might grow from it. That was not a failure of the gift. It was its freedom. Not everything has to be visible to a machine in order to count.

• • •

The student who learns the test instead of the subject grows up to be a person who learns the world instead of living in it. One self for the metric, another for the room. The attention economy has turned the split into a full-time occupation — performative empathy, simulated compassion, artificial intimacy. It is exhausting because none of it regenerates.

Long before there were machines to grade, people noticed this about themselves and asked what to do. The Gita's answer, given on a battlefield to a man who did not want to fight, was not to stop acting. It was to act fully and let go of the fruit. Two and a half thousand years of commentary have not exhausted the sentence, and I will not try to here. I only want to say what it is not. It is not a trick for making boats finish races. It is a description of a human being who has stopped circling the lagoon.

The word for that condition is equanimity, and it is widely misunderstood as calm. It is closer to freedom. You can still want the race. You can still want it badly, and work for it, and feel the loss. What changes is that the score stops being the thing that decides who you are. You can receive the signal without being hijacked by it. The same freedom holds when someone agrees with you or disagrees, when the room welcomes what you said or turns against it. Neither has to decide the quality of your attention. You can be watched without becoming a performance. Roshi Joan Halifax speaks of a strong back and a soft front: the equanimity that lets compassion stay open. Without the back, the front closes to protect itself. Without the front, the back is only armor.

Late in his life, Vinoba Bhave, Gandhi's spiritual heir, turned his attention to a small community of women at Pavnar who practiced shared labor and spiritual inquiry together. He had walked across India asking for land to be gifted to the landless, and millions of acres had come. Now he was after something harder to accumulate. As the charge was told to me: do not grow, do not measure your reach, tend to coherence among yourselves. A community consciousness in which the self-regarding impulse might loosen its grip. Of everything he could have left them, he thought this mattered most.

After thirty years of circles in ServiceSpace, I have noticed this much: one person's steadiness does not become a room's steadiness by itself. A circle of calm people can still carry every blind spot it walked in with. What happens is slower than that. Someone who is not defending an identity notices the thing everyone else missed. Someone who is not grasping at an outcome lets a silence go on long enough for the quiet person to speak. Someone declines to escalate, and the escalation that would have happened doesn't. None of it is visible that evening. Over years it becomes the weather of the place.

• • •

On the fourth of February, 1922, in a town called Chauri Chaura, police fired on a crowd of Indian demonstrators, and the crowd chased the police into their station and set it on fire. Twenty-two policemen died. The non-cooperation movement was at the height of its power. Mass civil disobedience was days from launching across the country, and the British, by most accounts, were frightened.

Gandhi called it off.

He did not treat the violence as someone else's failure. He took responsibility for what the movement had not yet learned, and fasted for five days. His reasoning, when he published it, was that India did not yet have the atmosphere in which that much power could be held without burning someone. The movement had gathered extraordinary force. He feared it had not yet grown the capacity to wield that force without reproducing the violence it opposed. A freedom taken by those means would be that kind of freedom.

His lieutenants were furious. Nehru wrote about it from prison with something close to despair. Some of the brightest young men in the movement concluded that nonviolence had been tested and had failed, and turned to bombs.

What gets forgotten is what Gandhi did next, because it was so unglamorous. He did not pause the struggle. He redirected it. Spin. Teach. Build schools. Sit with Muslims. Tear down untouchability in your own village. The constructive programme, he called it, and it looked to his critics like a retreat into chores. It was the opposite. He had understood that a people cannot be handed self-rule the way you hand someone a key. The capacity for it has to be grown, inside the struggle, by the struggle, or the struggle delivers a country that cannot hold what it won.

Eight years later he was ready to try again. In the years between, the man who had shaken an empire in South Africa kept returning to Ahmedabad, to the ashram and the wheel and the slow work of making people fit for self-rule. Visitors came expecting a plan and found him spinning. Two weeks before the Salt March, he published a short article saying how it would begin. Not with the millions. With the inmates of his own ashram, people who had lived under its discipline since 1915, and whose names, he said, would be unknown to fame. The ashram had been deliberately kept in reserve for this. He added something no founder says in public: that their self-imposed restraints had become subtle indulgences, and that if they could not now elevate their spirit of service, the ashram and he should end their tenure. Seventy-eight people walked out of the gate. The country followed. The march's force came from many places. But the fifteen years were what made the seventy-eight.

A spinning wheel in the foreground; a line of figures walking out of an ashram gate

A friend once shared that our outer work should never exceed the quality of our inner practice. She has carried that conviction through institutions of extraordinary power. Watching Gandhi prepare those seventy-eight people, I began to hear it as something else: a warning about what happens when the power of our actions grows faster than our capacity to hold their consequences. 

I share this story with engineers because it is the clearest account I know of a man refusing to let the means outrun the ends, and because it answers a question they keep asking in the wrong form. The question they ask is whether to slow down. That was never Gandhi's question. His question was what the work was making of the workers.

A model shapes the people who use it. Those people shape the institutions that will govern the model's successor. Those institutions shape the incentives that will train the next model. The loop does not end at deployment. It never did. So the question for a laboratory is not only whether its systems are safe. It is what capacities the laboratory is growing, or using up, in the people who will have to live with what it builds. In the engineers' own terms: the means are not neutral, and the ends arrive carrying the means that made them.

• • •

On the second evening of the gathering, two of the people building these systems were talking about their children.

They were not being sentimental. They were doing what very capable people do with a hard problem, which is to state it precisely. The work needed them. The children needed them. There were only so many hours, and both claims on the hours were real, and they wanted to know how to divide them well.

What I shared with them is that it was never only the hours. It is how much of you is in the hour. The grandmother of one of these leaders had lived at Gandhi's ashram, and so we found ourselves talking about Gandhi's one-minute meetings. I have heard first-hand accounts of how transformative that single minute could be, because all of him was in it. It was never the length of the time. It was the presence in it.

A minute does not solve anything. The problem does not go away; sometimes what a child needs is simply more of you, and no quality of attention substitutes for being there. But the problem had changed. An hour was still an hour. What it held could no longer be read off its length. We had been talking about allocation, and now we were talking about presence, and presence is a quality no reward function can see and any child can feel.

In the living room where I grew up, we have held an experiment every Wednesday since 1996. An hour of silence, a circle of sharing, a meal my mother cooks. No teacher. No fee. No curriculum. No one grading anyone. Three decades and tens of thousands of dinners later, the one thing that can be said with confidence is that a group of generous individuals is not yet a generous group, and that the difference between the two is made in rooms like this one, slowly, out of an emergent ingredient no external platform has figured out how to supply.

A circle of empty cushions in a living room, a lamp, a pot of food at the center

• • •

The friend who spoke of balancing outer systems with inner coherence was Preeta Bansal. She clerked at the Supreme Court, served in the Obama White House, and advised on the constitutions of Iraq and Afghanistan. After years working at the heights of institutional power, she returned to Nebraska to help cultivate Awakin circles, where people practice the slower work of listening to one another.

A wooden trellis with a vine growing past its edges, beside a closed cage

I thought of her when Anthropic published a constitution for Claude, seventy-nine pages, serious about character and judgment and not only rules. In one passage its authors compare the document to a trellis rather than a cage: a structure that supports growth without deciding its every direction. Preeta knows something about trellises. Law, she likes to say, is legitimized restraint, a traffic light that stops you so that everyone can travel without colliding. A constitution can create the conditions in which a society learns to govern itself. It cannot do the learning for the society.

She sometimes puts two photographs side by side. Ambedkar, who drafted India's constitution, in a suit. Gandhi, who spun, in homespun. They disagreed profoundly about how justice would come. Ambedkar had been born untouchable and knew that rights could not wait for every heart to change. Tyranny of incrementalism, he argued. Gandhi, for his part, feared that a reform imposed from outside would leave untouched the attitudes that sustained the injustice. Preeta, who has lived on both sides, says the outer systems channel and amplify the inner work, and the inner work is what makes the outer systems worth having. Neither a better constitution nor a better heart, alone, built the country they both wanted. Her warning to open-hearted people is that love has to understand power, and power has to be humanized by love.

So what would it mean to build these machines in a way that cultivates not only their capacities but ours? I do not think the answer is philanthropy. A laboratory that extracts all week and funds a meditation retreat on the weekend has not changed its means. The constructive programme was never a side project. It was a different way of doing the main thing. Let me try to say what that would look like at three scales, each as a pair: the number we are already making go up, and the question we are not yet asking.

At the scale of the product, the number is usefulness, engagement, tasks completed. The question is whether the people who use the thing come away more capable or more dependent. Does the tutor leave the teacher stronger or make the teacher optional? Does the companion leave someone more able to be with people, or less? We are building a small circle agent in our community along these lines. It helps six or seven people who would never otherwise have met find one another, handles the logistics, and then gets out of the room. The whole design is the leaving. Our Shunya Fellows are exploring the same principle in eight countries. The measure of it is not how much people use it but whether, after it goes, the people are still there. Markets are good at rewarding platforms that grow. They are much less practiced at rewarding platforms that make themselves unnecessary.

At the scale of the institution, the number is speed and position in the race. The question is whether the way the laboratory works preserves its ability to notice it is wrong. Can a researcher disagree without losing standing? Can a leader say a problem out loud when the problem threatens a milestone? Can the organization practice restraint in a quarter when every metric rewards acceleration? These are not soft questions. They are the only questions that tell you whether a place can still see.

At the scale of a society, the number is adoption and growth. The question is whether, after the technology arrives, the communities it arrives in are more able to govern their own futures, or less. Preeta told me what finally drove her out of the work she was so good at. You can change a hundred million lives overnight, she said, and you cannot be in relationship with a hundred million people. You can never go back and ask how it turned out for them. That is the position every frontier laboratory is in now, and no amount of good intention gets around it. Some of what these systems do will need governing at a scale no community can reach, across borders and generations. But much of it belongs far closer to the people who will live with the result than it currently sits, and there is a difference between being consulted and having a say. A community affected by a deployment could help decide not just how the thing is used but whether it should be, what it must never do, and what would justify taking it back out. A process like that leaves behind more than a successful rollout. It leaves people with the experience of holding shared responsibility for something consequential, which is the one capacity a society cannot buy later.

If I had to keep a single test — not another score to optimize, but a question to keep asking — it would be this. Does the system leave people more capable of solving things together when the system is no longer there?

Technology can tend the conditions for coherence. It cannot produce coherence by optimizing its appearance.

• • •

The two drawings are still in my notebook. I don't sense they need to become one.

The circles ask how to make something enormously powerful safe and good, ring by ring, outward from the machine. That is the repair branch of the river. The fork asks something the circles alone cannot answer: what kind of people are being made along the way, and whether they are growing the capacity to stop reproducing the harms. That is the wisdom branch, and we are already in the river. Both questions matter. But a civilization that only asks the first will optimize its way into the lagoon with the best of intentions, scoring higher every year.

While I was at the gathering, Wakanyi serendipitously left me an audio message: we are of the earth, she said, and we have grown so disconnected from it that the earth no longer recognizes us; and now we are building solutions to the disconnection that have no roots in it. I keep thinking about that. We can build cleaner data centers, safer models, better constitutions, and each may be necessary. But if the work doesn't also deepen our capacity to belong to one another and to the living world, we may solve one problem while reproducing the conditions that created it. Nothing without roots flourishes, however well it is governed.

Who grades the grader? Follow the question outward and it leaves every model, every laboratory, every institution behind, and arrives at us: what we are able to discern, and the kind of life among us that either grows that discernment or spends it.

The builders in that room are going to shape the next decade more than almost anyone alive. So will the teacher in the hills, and the woman who cooks on Wednesdays, and the boy who left a chocolate in a seat for whoever came next. The difference is that one kind of influence will be measured, ranked, and rewarded. The other passes quietly from person to person, changing what the next person is capable of giving.

When Jon Kabat-Zinn closed our circle with a meditation, he left everyone with a question he often asks himself: what is your karmic assignment?

Share a Reflection

Your thoughts will be shared with the author

Did this resonate? Share it forward…
☙

Join a Circle of Sharing

This reflection emerged from a small-group pod — a space for deep listening and authentic sharing. Explore upcoming circles you can join.

Explore Pods