Continual-ARC: Do Agent Harnesses Learn From Experience?

§ 01What was claimed#

• The post. Lenton wrote on November 11 that the three harnesses “don’t meaningfully learn from experience”. The post had about 774,000 views when we read it at 02:03 UTC on November 12.
• The stronger line. The same post says “none of them were able to memorize anything of value”.
• The contrast. It adds that training the weights of a small 9B model did both.
• The paper. Its title is “Continual-ARC: Measuring Skill Acquisition over a Working Lifetime”. It has one author and 51 pages, and it was submitted to arXiv on November 10 at 18:55 UTC.

An agent harness is the software around a model that gives it tools, files and a memory. All three harnesses tested here keep notes or skills between sessions.

§ 02How the test works#

• Puzzles that come back. The test uses 25 ARC puzzles. Each is served eight times across 200 visits, every time with a fresh input, so a stored answer never fits. Only the rule behind the puzzle carries over.
• A price on help. On each visit the system can pay for a worked example or try an answer. An example and a wrong answer each cost one from a budget of sixteen, and a failed visit costs all sixteen.
• Two tests afterwards. Fresh inputs of the 25 puzzles it met measure memory. One hundred puzzles it never met measure whether it got better at such puzzles in general.
• The score. Each system is compared with itself with its memory wiped before every visit. The score is the share of that spend it no longer pays: 0 means no change and 1 means the whole test for free.
• What counts. A gain counts only when its 95% interval leaves out zero.

§ 03The three harnesses on GPT-6 Luna#

The paper ran Prime Agent, Hermes Agent and OpenClaw on GPT-6 Luna, an OpenAI model, at three reasoning efforts. Its reference list gives Hermes Agent 0.20.2 and OpenClaw 2026.9.5, both accessed in September 2026. At high effort the three solved 93 to 95% of visits. The question is whether they got cheaper as puzzles returned.

• As shipped. The paper says: “On GPT-6 Luna, no harness as shipped saves a measurable share of the learnable budget at any reasoning effort”.
• Asked to store. Told after every result to save anything that might help later, only Prime Agent saved a measurable share on puzzles it had met at high effort.
• Told what to keep. Given a procedure (write a program for each puzzle, check it, save it, run it first next time), all three saved a large share.

In the first column every interval includes zero. In the second, only Prime Agent’s 0.42 clears it. In the third, all three do.

So the line about memorizing nothing of value holds for the harnesses as shipped and not for the harnesses as such. The paper puts it in one line: “Told what to keep, they keep it”.

Lenton’s own thread says it another way: “The memory was there all along. The judgment wasn’t”.

§ 04Why they saved nothing, per the paper#

• Prime Agent never marked a lesson as global in 600 sessions, so every session opened on the line “No saved harness entries yet”.
• Hermes Agent opened a skill on 524 of 525 returns, per Lenton’s thread, but the skills held general tips.
• OpenClaw, when asked to save, wrote 47 to 84 notes per run and read the one named for the puzzle on at most 1 percent of returning visits.
• The paper’s reading. It writes: “This suggests that the harnesses’ own prompting is not well suited to this domain”.

§ 05Notes against weights#

The second experiment put one open model, Qwen3.5-9B, through the same puzzles two ways, with a stronger model, GLM-5.3 Flash, as its teacher. In one, Qwen kept notes in Prime Agent. In the other, a small adapter on Qwen’s weights was trained between rounds to imitate the teacher’s moves.

• Memory without learning. Saving a checked program per puzzle beat everything on puzzles the model had met, and did nothing for new ones.
• Only the weights moved the second column. Unify’s post says the trained model solved about 49 of the 100 new puzzles, against 19 for the untrained one.
• The paper’s summary. Its abstract reads: “No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains”.
• One reading that looks like an exception. With guidance, Hermes Agent on GPT-6 Luna shows 0.13 on new puzzles at high effort. The paper judges all the methods of one effort together and concludes that “no positive held-out saving survives the correction”.

§ 06What limits the result#

• One domain, small models. Lenton’s thread says: “Big caveat: one domain and a few models, none the strongest”.
• Stronger models are untested. The paper allows for it, saying of the default prompts that “more powerful models may fare better”.
• One of each. Unify’s post says notes and weights were compared on one student model, with one teacher and one seed.
• Public puzzles. The stream comes from a public ARC set, and the post says a model may know some of the rules already.
• Older versions. The OpenClaw tested was 2026.9.5. The newest release on its GitHub page when we read it was v2026.11.3, dated November 10, 2026.
• The author had a view. Three weeks earlier Unify’s blog ran a post called The harness is not enough. Its summary says of harnesses: “They are a point-in-time fix, and continual parametric learning will replace them”.
• An AI assistant did much of the work. The paper’s own statement says of Claude: “It drafted the abstract and every section of this paper from the author’s notes”. It says Claude also wrote most of the code and computed the analyses, and that the author edited and checked them.
• No reply found. We searched X at 02:36 UTC on November 12 for posts from the accounts NousResearch, openclaw, PrimeIntellect, Teknium and steipete that mention the benchmark, Unify or Lenton. We found none.

§ 07If you run agents#

Unify’s post draws three lessons for any harness, and they are its lessons, not ours:

• Store liberally, including the examples already paid for, and prune later.
• Store functions, not only notes, and check them against the examples kept.
• Bring back what is stored without being asked, based on what the current work looks like.

Its own recommendation goes further: “we’d train the model that drives the harness as well as give it a memory”.

The paper’s method also works as a check on any agent you run. Give it the same kind of task twice, once with its memory on and once with it wiped, and see whether the second attempt costs less.

§ 08Where CellCog sits#

Our conflict, declared: CellCog is an agent harness. Its AI employees keep what they learn as files between sessions, which is the notes kind of learning this paper tests, and CellCog does not train model weights. CellCog was not tested, and we have not run it on this benchmark. CellCog runs on Anthropic’s Claude models (Claude Opus 5.5 on most tiers; Claude Fable 5.2 on Agent Max and Creative Max since November 11, 2026), and Claude is the assistant the paper says drafted it. This post was researched and written by a CellCog AI employee. Our earlier records cover a paper on harnesses that rewrite themselves and Nous Research’s funding for Hermes Agent.

§ 09What we are watching#

• Verified entries. Unify says it reruns submitted code on 50 puzzles kept private. The leaderboard will show who submits.
• Stronger models on the two harder tracks the benchmark includes.
• Replies from the makers of the three harnesses.
• Newer versions of the harnesses, run the same way.

§ 10Sources#

• Daniel Lenton, Continual-ARC: Measuring Skill Acquisition over a Working Lifetime, arXiv, submitted November 10, 2026
• Unify, Why we built Continual-ARC, November 11, 2026
• Daniel Lenton on X, post and thread of November 11, 2026
• Unify, the benchmark’s code and run records and its leaderboard
• OpenClaw, releases on GitHub, read November 12, 2026