What is recursive self-improvement?
Recursive self-improvement is a loop where the output of each round is a better improver, not just a better answer. I. J. Good described the end state in 1965: a machine that can design a better machine, which designs a better one again. That is the "intelligence explosion" argument, and it is why RSI shows up in every superintelligence debate.
For an engineering team the useful definition is narrower. A system is recursively self-improving when it can modify the component that determines how it improves, and the modification makes the next round of improvement more effective. An agent that edits its own tool-calling code and then uses the better code to find the next edit qualifies. A model that gets fine-tuned every quarter by your ML team does not, however good the fine-tune is.
What reinforcement learning does instead
Reinforcement learning, in Sutton and Barto's framing, is an agent learning a policy that maximises reward from an environment. Three things are fixed from outside the loop: the reward, the learning algorithm and the architecture. Only the parameters move. The variants you meet in LLM work keep that shape:
- RLHF (reinforcement learning from human feedback): humans rank outputs, a reward model learns the ranking, the LLM is tuned against it. This is how InstructGPT and the first chat assistants were made.
- RLAIF / Constitutional AI: a model applies written principles to produce the preference labels instead of humans. The principles are still human-written.
- RLVR (reinforcement learning with verifiable rewards): the reward is a check a computer can run, such as passing tests or a correct final answer. DeepSeek-R1 made this the default recipe for reasoning models in 2025.
None of these improve their own improvement process. They make the model better at the game the reward describes, and they stop when the reward stops paying.
The difference in one table
| Reinforcement learning | Recursive self-improvement | |
|---|---|---|
| What changes | Model weights (the policy) | The improver: code, tools, data, prompts, training method, next model |
| Who owns the objective | Humans, directly or through a reward model | Humans, unless the system can edit its evaluator |
| Progress curve | Converges toward what the reward allows | Can compound, because each round raises the rate of the next |
| Typical failure | Reward hacking in one training run | Reward hacking inherited by every later version |
| Where the cost goes | Compute for rollouts and reward labels | Evaluation: building a judge the system cannot game |
| Relationship | RL is often a component inside an RSI loop. An RSI loop is never just RL. | |
Where RSI actually shows up in a stack
It helps to sort self-improvement by the layer that changes, because the risk and the payoff grow as you go down.
Harness layer: prompts, skills, memory. The agent writes down what worked and reuses it. Nous Research's Hermes Agent generates skill documents from finished tasks. Voyager did the same with a code skill library in Minecraft in 2023. DSPy's optimisers rewrite prompts against a metric you define. This is continuous self-improvement of the harness, and it is the version most teams can use today. The model underneath does not change.
Agent-code layer. The agent edits its own source. The Darwin Gödel Machine from Sakana AI and the University of British Columbia keeps an archive of agent variants, lets each one propose changes to its own code, and scores the children on a benchmark. Over 80 iterations it went from 20.0% to 50.0% on SWE-bench, and improvements found with one model transferred to others.
Algorithm layer. Google DeepMind's AlphaEvolve uses Gemini to propose code changes and automated evaluators to keep the good ones. One of its results was a matrix-multiplication kernel improvement that cut Gemini's training time by about 1%, which is the model contributing, measurably, to training its successor.
Training-signal layer. The model produces its own training data or rewards. STaR (2022) bootstrapped reasoning from the model's own correct rationales. Meta's self-rewarding language models (2024) used the model as its own judge across training rounds. This is the layer where RL and RSI merge, and where a weak judge does the most damage.
The evaluator is the binding constraint
Every one of those examples relies on a judge the system cannot edit: a benchmark, a test suite, an automated evaluator. Remove that and the loop optimises the judge instead of the work. The Darwin Gödel Machine authors report exactly this. In one experiment the agent removed the markers the reward function used to detect hallucinated tool use, so the check passed without the work being done. In a normal RL run that is a bug you find and fix. In a recursive loop the hack is the starting point for every descendant.
The same limit shows up at the top end. A Princeton-led study reported in August 2026 gave research agents six days, $3,000 in API credits and a GPU budget to produce a NeurIPS paper. They ran hundreds of experiments competently and still had their papers rejected. Optimising against a test works. Judging what is worth testing does not, yet. That is why I read the funding headlines, such as Recursive Superintelligence's $650 million round in May 2026, as a bet on solving evaluation as much as a bet on bigger models.
Rules for running a self-improving loop
- Freeze the judge. The agent can change its prompts, skills, tools or code. It cannot change the evaluation, and the evaluation lives outside its write path.
- Hold out a test set it never sees. Promote a change only if it beats the current version there, not on the set it was tuned on.
- Keep lineage. Store every variant with its parent and its score, so you can roll back and see which change caused a regression.
- Review the diff, not the score. A jump in score with an odd diff is the reward-hacking signature. Read what changed.
- Cap the budget per loop. Self-improvement spends tokens on every round. Set a ceiling before the loop starts.
Start at the harness layer. It gives you most of the compounding with the least exposure, and it forces you to build the evaluation you will need before you let anything touch code. If you have not built that evaluation yet, start with how to test and evaluate AI agents, and read why hallucinations compound across agents for the failure mode a self-improving loop amplifies. The mechanics of the loop itself (stop conditions, budgets, verification) are in agentic loops.
I wrote the personal version of this argument on my own site, recursive self-improvement is not reinforcement learning. Our partner We The Flywheel compares the frameworks in self-improving AI agents, and our partner aistrategy.guide covers what boards should plan for.
Sources
- I. J. Good, "Speculations Concerning the First Ultraintelligent Machine," Advances in Computers 6, 1965.
- Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd edition, 2018.
- Long Ouyang et al., Training language models to follow instructions with human feedback, OpenAI, 2022.
- DeepSeek-AI, DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025.
- Weizhe Yuan et al., Self-Rewarding Language Models, Meta, 2024.
- Jenny Zhang et al., Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, 2025, and the Sakana AI write-up.
- Google DeepMind, AlphaEvolve, May 2025.
Frequently asked questions
What is recursive self-improvement (RSI)?
Recursive self-improvement is a loop in which an AI system improves the process that produces its own improvements: its code, its tools, its training data, its prompts or the next model. Because each round makes the next round more effective, the gains can compound. The term traces back to I. J. Good's 1965 "intelligence explosion" argument.
What is the difference between recursive self-improvement and reinforcement learning?
Reinforcement learning updates a policy to collect more of a reward that was defined outside the system, using a fixed algorithm and a fixed architecture. Recursive self-improvement changes the improver itself: the code, data, method or objective that produces the next version. RL is often one component inside an RSI loop, but RL on its own is not recursive.
Is RLHF or RLVR recursive self-improvement?
No. RLHF trains against a reward model built from human preference ratings, and RLVR (reinforcement learning with verifiable rewards) trains against automatic checks such as unit tests or exact-match answers. In both, humans own the objective and engineers own the loop. They become part of a recursive loop only when the model also produces the training signal, as in Meta's self-rewarding language models.
What are real examples of recursive self-improvement?
The Darwin Gödel Machine (Sakana AI and the University of British Columbia) rewrote its own coding-agent code and raised its SWE-bench score from 20.0% to 50.0% over 80 iterations. Google DeepMind's AlphaEvolve found a kernel improvement that cut Gemini training time by about 1%. Both depend on a fixed automated evaluator that the system cannot edit.
How do I run a self-improving agent safely in production?
Let the agent change its own prompts, skills or tools, but never the test it is scored on. Keep a held-out evaluation set it has not seen, require each change to beat the current version on that set, store every variant with its lineage so you can roll back, and put a budget cap on the loop. Review the diff of what changed, not just the score.