Blog Author: Yan Song, Xidong Feng

Paper Author: Yan Song, Xidong Feng, Bo Liu, Xinyu Cui, Haotian Fu, Zichen Liu, Mengyue Yang, Cheng Deng, Jian Zhao, Jun Wang

Arxiv: https://arxiv.org/abs/2607.28638

Github: https://github.com/YanSong97/Stateful_Knowledge_Learning

<aside>

All knowledge about the world is predictive. — Richard Sutton

</aside>

Large language model agents are entering the Era of Experience: a shift from learning mainly from static human-written data toward learning through continuous interaction with environments. This shift raises a deceptively simple question:

What exactly should an agent learn from experience?

Before we answer it, let us review the common solution first: self-reflection. An agent tries a task, observes the full trajectory, and then writes a critique of what went wrong and what to do differently next time. This idea has powered early work such as Reflexion, more recent reflection-based training methods such as R$^3$L, and industry-scale agent systems. The recent interest in On-Policy Self-Distillation points in the same direction. If a model can reflect evaluation/insight/skill from experience, this can become a fantastic training signal for better reasoning.

So a growing line of works is in fact, suggests one compelling answer:

Agents should learn by compressing their own experience back into their understanding and knowledge of the environment.

From the perspective of traditional Reinforcement Learning (RL), this idea feels both familiar and curiously novel. It feels familiar because learning from interaction to improve policy is the exact premise of classic RL algorithms like Policy Gradient. Yet, it feels entirely new because LLM agents are doing something deeper: they are explicitly learning knowledge, articulating understanding, and reasoning about the why behind their successes and failures. In the strict context of classical RL, updating a policy's weight matrix doesn't exactly equate to "learning knowledge."

Or does it? Does this concept actually exist in some forgotten corner of RL?