When the student outgrows the teacher: the alignment problem

Most AI models in use today learned their manners from people. In reinforcement learning from human feedback, people compare a model's answers and say which is better, and the model is trained toward what they prefer. It works well because the people judging can tell a good answer from a bad one. The alignment problem for superintelligence starts from a simple question: what happens when they can't?

Why the current method has a ceiling

OpenAI put the problem plainly in July 2023, when it announced its superalignment team: "Currently, we don't have a solution for steering or controlling a potentially superintelligent AI." Its reasoning was that methods like human feedback rely on people being able to supervise the AI. For systems much more capable than us, that supervision breaks down, so the methods will not scale.

The company pledged a fifth of the computing power it had secured to the effort over four years. Its plan was to build a roughly human-level automated alignment researcher, then use it to help align more capable systems. In May 2024 the team's co-leads left, and Fortune reported that the computing commitment was never fulfilled. The problem they described has not gone away.

Can a weak teacher train a strong student?

The superalignment team's best known result turned the problem into an experiment. Since no superhuman model exists to study, they used an analogy: a small model plays the human, and a much larger model plays the superintelligence. In their 2023 paper on weak-to-strong generalization, a model at roughly GPT-2's level produced labels, and a model at roughly GPT-4's level was trained on them.

The strong student usually did better than its weak teacher. It seemed to use what it already knew to see past some of the teacher's mistakes. But it did not reach what it could do when trained on correct labels. The researchers measured how much of that gap was recovered and found it varied widely by task.

The paper is honest about its limits. A gap in skill between two models is not the same as a gap between people and a superintelligence. And a strong model that improves on its teacher may be falling back on what it already knew, not learning what the teacher meant.

Whose goals?

There is a second, quieter problem. Aligning a system means aligning it to something: a set of values, a standard, an intended result. The International AI Safety Report 2026 notes growing work on pluralistic alignment, which tries to respect different people's values, and observes that no single approach satisfies everyone. Before a system can follow a standard, someone has to write the standard down.

A model can only be as aligned as the judgment it learns from is clear.

What this means in practice

Superintelligence is not here, but the shape of the problem already shows up in everyday AI work. When a model is trained on reviews from people who cannot fully check its work, it learns to satisfy the reviewer, not to be right. Three lessons carry over:

• The quality of the supervision sets a ceiling. Put the people best able to judge on the cases that matter most.
• Write the standard down before the training starts, in criteria others can apply the same way.
• Measure the supervisors too: how often they agree, and how often they catch known errors.

These are the questions our work is built around. As systems become more capable, the value of clear, expert, measured judgment rises, not falls.