I recently read that Anthropic’s Claude is helping develop future versions of itself.

My first thought was: what are we missing if we train an AI, then rely on that AI to help train the next one?

If the original system has a blind spot, could it carry that blind spot forward? Could each generation become more capable while still overlooking something important?

From an audit perspective, that sounds familiar. A process can become faster, more consistent, and more sophisticated without becoming better at questioning its own assumptions.

What does “AI building AI” actually mean?

Anthropic reported that, as of August 2026, more than 90% of its measured AI research and development work involved at least substantial collaboration with Claude. Claude “led” 26%, meaning it could complete most of a task from a high-level prompt while a person supervised.

These are Anthropic’s own measurements. The company said none of the measured work was fully autonomous. They also describe a broad development process, rather than simply a model retraining itself. (Anthropic’s methodology and findings)

That distinction matters. People remain involved.

But it leaves an important question open: how effectively can they challenge the work?

The blind spot could be in the question

Imagine asking an AI to design a test for the next model.

It produces a detailed evaluation. The new model performs well. The results look reassuring.

But suppose the evaluation leaves out an unusual situation that neither model recognizes as important.

Nothing has to malfunction for that omission to survive. The test can run correctly. The score can be accurate. The conclusion can still reach further than the evidence supports.

Passing a test tells us how a system performed on what we tested. It cannot, by itself, tell us whether we asked all the right questions.

My concern is that an existing model could influence what researchers investigate, what they measure, and what they consider a satisfactory result. Its limitations could shape the process before anyone reviews the final answer.

That is a risk to investigate, not proof that it is happening in every AI development workflow.

Agreement is not the same as independent assurance

In audit, having someone review their own work creates an obvious limitation. They may repeat the reasoning that produced the original mistake.

A similar concern applies when related AI systems generate work and assess it. Agreement may reflect correctness. It may also reflect shared assumptions or shared weaknesses.

Anthropic acknowledges this issue in the same report: using its models as evaluators could mean the evaluator makes the same kinds of errors as the system being assessed.

That makes independence a practical design question.

A second AI opinion may be useful. We still need to understand what makes that opinion meaningfully different and what external evidence supports it.

A human in the loop needs room to challenge

Human supervision is valuable when the person has the expertise, evidence, time, and authority to question the result.

Consider the difference between reviewing a summary that says “all tests passed” and examining whether the tests covered the most consequential failure scenarios.

Both involve a human. They provide very different levels of assurance.

As AI produces more research, code, and evaluations, I would want to know whether reviewers can trace important decisions back to supporting evidence. Can they reproduce a result? Can they reject a proposed approach? Can they stop development when something remains unexplained?

A sign-off means more when those conditions exist.

What should we be asking?

For organizations using AI to develop or evaluate other AI systems, a few questions deserve attention:

• Who defines success, and who challenges that definition?
• Which findings are checked against evidence outside the model’s own outputs?
• Are evaluations designed to expose unexpected failures?
• Can outside specialists examine consequential assumptions and results?
• Who has authority to pause when the evidence is insufficient?

AI may help researchers discover mistakes that people would otherwise miss. That is a substantial opportunity.

Taking advantage of it requires a process that also looks for the mistakes AI may miss.

The next model could be faster, more capable, and better at solving difficult problems. Those improvements matter.

I still want to know who is looking for what the previous model never thought to question.

---

This post is part of an ongoing series on AI governance and audit. If you’re working through these questions too, I’d like to hear from you: get in touch.