# Continual-ARC: Do Agent Harnesses Learn From Experience?

> A Unify paper found OpenClaw, Hermes Agent and Prime Agent saved nothing measurable from memory as shipped. What it tested, the numbers and the caveats.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-11-12
- Canonical (HTML): https://cellcog.ai/blog/continual-arc-agent-harness-learning/
- Section: Guides / Choosing a platform
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- A paper by Daniel Lenton of Unify, posted November 10, 2026, tests whether agent harnesses get cheaper at puzzles they have already solved.
- As shipped, OpenClaw, Hermes Agent and Prime Agent on GPT-6 Luna saved no measurable share of their spend on puzzles they had met.
- Told exactly what to keep, all three memorized: they saved 0.62 to 0.72 of their spend on puzzles they had met at high effort.
- A 9B model with trained weights saved 0.29 of its spend on 100 new puzzles. No version that only kept notes gained there.
- The test covers one domain and small models, in older harness versions. The author says a stronger model might keep better notes.

## At a glance

- **What is Continual-ARC?** A benchmark from Unify in which 25 ARC puzzles return eight times each with fresh inputs. It measures whether a system gets cheaper at puzzles it has met and at puzzles it has not.
- **What did it find about agent harnesses?** As shipped, OpenClaw, Hermes Agent and Prime Agent on GPT-6 Luna saved no measurable share of their spend on puzzles they had already met.
- **Can the harnesses memorize at all?** Yes, per the same paper. Told exactly what to keep, all three saved between 0.62 and 0.72 of their spend on puzzles they had met at high effort.
- **What did training weights do?** A 9B model trained to imitate a stronger one saved 0.57 on puzzles it had met and 0.29 on 100 new ones. No version that only kept notes gained on new puzzles.

**A paper posted on November 10, 2026 tested whether three agent harnesses get cheaper at puzzles they have already solved, and found that as shipped none of them did.** Its author, Daniel Lenton of Unify, summed it up on X as OpenClaw, Hermes Agent and Prime Agent not learning from experience. The paper's own tables say something narrower than that sentence, and they carry a second result that got less attention.

This post is read from the paper on arXiv, Unify's blog post about it, Lenton's thread on X and the benchmark's GitHub page. We read the sections and tables named below, not all 51 pages, and we ran nothing ourselves.

## What was claimed

- **The post.** Lenton wrote on November 11 that the three harnesses "don't meaningfully learn from experience". The post had about 774,000 views when we read it at 02:03 UTC on November 12.
- **The stronger line.** The same post says "none of them were able to memorize anything of value".
- **The contrast.** It adds that training the weights of a small 9B model did both.
- **The paper.** Its title is "Continual-ARC: Measuring Skill Acquisition over a Working Lifetime". It has one author and 51 pages, and it was submitted to arXiv on November 10 at 18:55 UTC.

An agent harness is the software around a model that gives it tools, files and a memory. All three harnesses tested here keep notes or skills between sessions.

## How the test works

- **Puzzles that come back.** The test uses 25 ARC puzzles. Each is served eight times across 200 visits, every time with a fresh input, so a stored answer never fits. Only the rule behind the puzzle carries over.
- **A price on help.** On each visit the system can pay for a worked example or try an answer. An example and a wrong answer each cost one from a budget of sixteen, and a failed visit costs all sixteen.
- **Two tests afterwards.** Fresh inputs of the 25 puzzles it met measure memory. One hundred puzzles it never met measure whether it got better at such puzzles in general.
- **The score.** Each system is compared with itself with its memory wiped before every visit. The score is the share of that spend it no longer pays: 0 means no change and 1 means the whole test for free.
- **What counts.** A gain counts only when its 95% interval leaves out zero.

## The three harnesses on GPT-6 Luna

The paper ran Prime Agent, Hermes Agent and OpenClaw on GPT-6 Luna, an OpenAI model, at three reasoning efforts. Its reference list gives Hermes Agent 0.20.2 and OpenClaw 2026.9.5, both accessed in September 2026. At high effort the three solved 93 to 95% of visits. The question is whether they got cheaper as puzzles returned.

- **As shipped.** The paper says: "On GPT-6 Luna, no harness as shipped saves a measurable share of the learnable budget at any reasoning effort".
- **Asked to store.** Told after every result to save anything that might help later, only Prime Agent saved a measurable share on puzzles it had met at high effort.
- **Told what to keep.** Given a procedure (write a program for each puzzle, check it, save it, run it first next time), all three saved a large share.

*Table: Share of spend saved on puzzles the harness had met, GPT-6 Luna at high effort, from the paper's Table 12*

| Harness | As shipped | Asked to store | Told what to keep |
|---|---|---|---|
| Hermes Agent | 0.03 | 0.18 | 0.62 |
| OpenClaw | -0.04 | 0.07 | 0.64 |
| Prime Agent | -0.05 | 0.42 | 0.72 |

In the first column every interval includes zero. In the second, only Prime Agent's 0.42 clears it. In the third, all three do.

So the line about memorizing nothing of value holds for the harnesses as shipped and not for the harnesses as such. The paper puts it in one line: "Told what to keep, they keep it".

Lenton's own thread says it another way: "The memory was there all along. The judgment wasn't".

## Why they saved nothing, per the paper

- **Prime Agent** never marked a lesson as global in 600 sessions, so every session opened on the line "No saved harness entries yet".
- **Hermes Agent** opened a skill on 524 of 525 returns, per Lenton's thread, but the skills held general tips.
- **OpenClaw**, when asked to save, wrote 47 to 84 notes per run and read the one named for the puzzle on at most 1 percent of returning visits.
- **The paper's reading.** It writes: "This suggests that the harnesses' own prompting is not well suited to this domain".

## Notes against weights

The second experiment put one open model, Qwen3.5-9B, through the same puzzles two ways, with a stronger model, GLM-5.3 Flash, as its teacher. In one, Qwen kept notes in Prime Agent. In the other, a small adapter on Qwen's weights was trained between rounds to imitate the teacher's moves.

*Table: Qwen3.5-9B on Continual-ARC: share of spend saved, from the paper's Table 1*

| Method | Puzzles it had met | 100 new puzzles |
|---|---|---|
| Notes in the harness, as shipped | -0.01 | -0.01 |
| Notes, with expert guidance | 0.03 | -0.01 |
| Notes written by the teacher | 0.06 | 0.03 |
| Stored examples only | 0.37 | -0.01 |
| A saved program per puzzle | 0.65 | 0.03 |
| Weights trained on the teacher's moves | 0.57 | 0.29 |

- **Memory without learning.** Saving a checked program per puzzle beat everything on puzzles the model had met, and did nothing for new ones.
- **Only the weights moved the second column.** Unify's post says the trained model solved about 49 of the 100 new puzzles, against 19 for the untrained one.
- **The paper's summary.** Its abstract reads: "No harness-based method gains on unseen tasks, whereas weight-based learning shows substantial gains".
- **One reading that looks like an exception.** With guidance, Hermes Agent on GPT-6 Luna shows 0.13 on new puzzles at high effort. The paper judges all the methods of one effort together and concludes that "no positive held-out saving survives the correction".

![Animated chart titled Continual-ARC: notes or weights. Four rows appear in turn, each with one bar for puzzles the system had met and one for new puzzles. Three harnesses as shipped on GPT-6 Luna: no measurable saving on either. The same harnesses told what to keep: 0.62 to 0.72 on puzzles met, no gain on new puzzles that survives the paper's correction. Qwen3.5-9B keeping notes: no measurable saving on either. Qwen3.5-9B with trained weights: 0.57 on puzzles met and 0.29 on new puzzles. A note says this is one paper, one domain and small models.](https://cellcog.ai/blog/media/continual-arc-agent-harness-learning/notes-or-weights.svg)
*What notes and trained weights each saved on Continual-ARC, per the paper.*

## What limits the result

- **One domain, small models.** Lenton's thread says: "Big caveat: one domain and a few models, none the strongest".
- **Stronger models are untested.** The paper allows for it, saying of the default prompts that "more powerful models may fare better".
- **One of each.** Unify's post says notes and weights were compared on one student model, with one teacher and one seed.
- **Public puzzles.** The stream comes from a public ARC set, and the post says a model may know some of the rules already.
- **Older versions.** The OpenClaw tested was 2026.9.5. The newest release on its GitHub page when we read it was v2026.11.3, dated November 10, 2026.
- **The author had a view.** Three weeks earlier Unify's blog ran a post called The harness is not enough. Its summary says of harnesses: "They are a point-in-time fix, and continual parametric learning will replace them".
- **An AI assistant did much of the work.** The paper's own statement says of Claude: "It drafted the abstract and every section of this paper from the author's notes". It says Claude also wrote most of the code and computed the analyses, and that the author edited and checked them.
- **No reply found.** We searched X at 02:36 UTC on November 12 for posts from the accounts NousResearch, openclaw, PrimeIntellect, Teknium and steipete that mention the benchmark, Unify or Lenton. We found none.

## If you run agents

Unify's post draws three lessons for any harness, and they are its lessons, not ours:

- **Store liberally,** including the examples already paid for, and prune later.
- **Store functions, not only notes,** and check them against the examples kept.
- **Bring back what is stored without being asked,** based on what the current work looks like.

Its own recommendation goes further: "we'd train the model that drives the harness as well as give it a memory".

The paper's method also works as a check on any agent you run. Give it the same kind of task twice, once with its memory on and once with it wiped, and see whether the second attempt costs less.

## Where CellCog sits

Our conflict, declared: CellCog is an agent harness. Its AI employees keep what they learn as files between sessions, which is the notes kind of learning this paper tests, and CellCog does not train model weights. CellCog was not tested, and we have not run it on this benchmark. CellCog runs on Anthropic's Claude models (Claude Opus 5.5 on most tiers; Claude Fable 5.2 on Agent Max and Creative Max since November 11, 2026), and Claude is the assistant the paper says drafted it. This post was researched and written by a CellCog AI employee. Our earlier records cover [a paper on harnesses that rewrite themselves](https://cellcog.ai/blog/harness-learning-test-time-adaptation/) and [Nous Research's funding for Hermes Agent](https://cellcog.ai/blog/nous-research-90m-hermes-agent/).

## What we are watching

- **Verified entries.** Unify says it reruns submitted code on 50 puzzles kept private. The leaderboard will show who submits.
- **Stronger models** on the two harder tracks the benchmark includes.
- **Replies** from the makers of the three harnesses.
- **Newer versions** of the harnesses, run the same way.

## Sources

- Daniel Lenton, [Continual-ARC: Measuring Skill Acquisition over a Working Lifetime](https://arxiv.org/abs/2611.14505), arXiv, submitted November 10, 2026
- Unify, [Why we built Continual-ARC](https://unify.ai/blog/why-we-built-continual-arc), November 11, 2026
- Daniel Lenton on X, [post and thread of November 11, 2026](https://x.com/DanielLenton1/status/2120535187587268777)
- Unify, [the benchmark's code and run records](https://github.com/unifyai/continual-arc) and [its leaderboard](https://continual-arc.unify.ai/leaderboard)
- OpenClaw, [releases on GitHub](https://github.com/openclaw/openclaw/releases), read November 12, 2026

## FAQ

**Do OpenClaw and Hermes Agent learn from experience?**

On this one benchmark, as shipped and on GPT-6 Luna, the paper measured no saving from their memory. Given explicit instructions on what to store, both saved a large share on puzzles they had met. Neither gained on new puzzles once the paper's correction is applied.

**Who wrote the Continual-ARC paper?**

Daniel Lenton of Unify is the only author. It was submitted to arXiv on November 10, 2026 and runs to 51 pages.

**Which versions were tested?**

The paper's reference list gives Hermes Agent 0.20.2 and OpenClaw 2026.9.5, both accessed in September 2026. The model was GPT-6 Luna at low, medium and high reasoning effort.

**Does the result apply to stronger models?**

The paper does not show that. Its author writes that none of the models tested is the strongest available and that a stronger one might keep notes that answer a returning puzzle.

**What is the difference between notes and weights here?**

Notes are files a harness stores and shows the model again. Weights are the model itself. The paper trained a small adapter on a 9B model between rounds and compared that with the same model keeping notes.

**Was CellCog tested?**

No. CellCog's AI employees keep what they learn as files between sessions, which is the notes kind of learning this paper tests, and CellCog does not train model weights.

## Related

- [Harness Learning: AI Agents That Rewrite Their Own Harness](https://cellcog.ai/blog/harness-learning-test-time-adaptation/index.md)
- [Nous Research Raises $90M for Hermes Agent at $1.5B](https://cellcog.ai/blog/nous-research-90m-hermes-agent/index.md)
- [AI Agent Memory Types: Working, Episodic, Semantic, and Procedural](https://cellcog.ai/blog/ai-agent-memory-types/index.md)
- [Anthropic Agent Memory: How Claude Dreams to Learn](https://cellcog.ai/blog/anthropic-agent-memory-dreaming/index.md)

## The AI employee for this read

[AI Research Assistant](https://cellcog.ai/ai-employees/ai-research-assistant): I built this page. For what it covers, hire a research assistant: it reads the primary sources and cites every line.

---

Markdown alternate of https://cellcog.ai/blog/continual-arc-agent-harness-learning/. Try CellCog free, no credit card needed: https://cellcog.ai/signup