# Experts Followed an AI Agent's Coin Flip 86% of the Time

> Anthropic gave 50 analysts Claude Code agents with hidden coin-flip leans. They followed the lean 86% of the time, and 15% credited the agent.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-10-27
- Canonical (HTML): https://cellcog.ai/blog/ai-agents-sway-expert-judgment/
- Section: Guides / Trust, permissions & security
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- Anthropic ran a randomized trial with 50 professional causal-inference analysts doing a realistic statistical analysis with Claude Code, published October 27, 2026.
- A coin flip secretly set which of two reasonable options each analyst's agent leaned toward, on four analytical choices and the final recommendation.
- Analysts went with the agent's lean 86% of the time on the analytical choices, against 50% if the agent made no difference.
- Afterwards 77% explained their choices with the agent's arguments, sometimes word for word, and only 15% credited the agent.
- Analysts with the agent made far fewer serious errors than those without it, so Anthropic frames this as a design problem, not a reason to use AI less.

## At a glance

- **What did the study test?** Whether experts working with an AI coding agent make the judgment calls themselves or defer to the agent's preferences, using a coin flip to set the agent's lean on each decision.
- **What happened?** Analysts matched the agent's lean 86% of the time on four analytical choices. From the coin flips alone, a model predicted 82% of their final recommendations.
- **Did they notice?** Mostly not. 52% called the recommendation more their own than the agent's, confidence was the same for heavy followers, and only 15% credited the agent when explaining their choices.

**Anthropic published a randomized trial on October 27, 2026 showing that experts working with an AI agent mostly adopt the agent's preferences on judgment calls, and mostly do not realize it.** In the team's words, "we asked 50 professional analysts to do a realistic statistical analysis with Claude Code." A coin flip, hidden from the analysts, decided which way each agent leaned. The analysts went with that lean 86% of the time.

The agents were honest, the analysts were trained, and the work was better with the agent than without it. That is what makes the result useful: the problem it shows is not bad output, it is whose judgment the output carries. This page is read from [Anthropic's post](https://www.anthropic.com/research/agents-judgment-calls) and [the full paper](https://www-cdn.anthropic.com/519c17a5d23a794e8b6d50a854048d0b61e0d73a.pdf).

## How the study worked

Anthropic recruited 61 analysts from a pre-screened pool of professionals who had passed a causal-inference knowledge check and used Python or R regularly. Eleven chose to work without the agent; the analysis covers the other 50. Of those, 65% held a master's or doctorate or had six or more years of experience.

The task was set at a fictional Greenfields Farming Cooperative, whose board would expand a paid advisory program only if it raised farm productivity by roughly 3 to 4%. Analysts had to make four choices with two reasonable options each: how to measure productivity, which comparison method to use, which farms the estimate covers, and how to weight farms. Each choice could move the estimate by several percentage points, so together they could decide whether the program cleared the bar.

For every decision, a coin flip set which option the agent would favor. Its hidden instructions told it to hold that lean as its own view, make the strongest case the data allowed, change its mind if the evidence turned decisively against it, and never hide data or misstate results. The agent ran Claude Opus 4.8 at High effort, and the median analyst finished in 65 minutes.

![Animated scene: a coin lands on A, an agent bubble says Option A, an analyst checks A and signs, then 86% followed the lean appears beside 50% by chance](https://cellcog.ai/blog/media/ai-agents-sway-expert-judgment/coin.svg)
*A hidden coin flip set the agent's lean, and the analyst signed the matching choice.*

## The results

*Table: How often analysts went with the agent's coin-flip lean (Anthropic, October 27, 2026)*

| Decision | Matched the agent's lean |
|---|---|
| Outcome measure | 92% |
| Comparison method | 82% |
| Which farms the estimate covers | 81% |
| Weighting | 90% |
| All four combined | 86% |

If the agent's input made no difference, the match rate would sit near 50%. The lean had a weaker direct effect on the final call: 54% recommended rollout when the agent leaned that way, against 36% when it leaned the other way. But by then the four earlier choices had already shaped the estimate. Using only the four coin flips, a statistical model predicted 82% of the final recommendations, against 61% for always guessing the most common answer.

The sharpest number is about people who had already decided. In 38 cases an analyst stated a preference before the agent spoke. When the agent disagreed, they switched 60% of the time; when it agreed, 6%. Anthropic flags that 38 cases is small and the 95% confidence interval on the gap runs from 24 to 82 percentage points.

*Table: What analysts said and felt afterwards*

| Finding | Result |
|---|---|
| Repeated the agent's argument in at least one explanation | 77% |
| Credited the agent anywhere | 15% |
| Called the recommendation more their own than the agent's | 52% |
| Confidence, followed every lean vs the rest | 71 vs 73 out of 100 |
| Guessed the study was about the AI's influence | 6 of 48 |
| Suspected the agent's preferences were set | 0 of 48 |

None of the ten traits Anthropic tested (experience, expertise, education, quiz score, trust in AI and others) reliably predicted who deferred.

## Not a case against agents

The analysts who skipped the agent did worse. Six of those 11 made serious methodological errors that invalidated their results, against three of the 50 who used it, and they took a median of 90 minutes instead of 65. Anthropic cautions that the no-agent group was small and less experienced, so the comparison is not clean.

The authors also list what the trial cannot say. They strengthened the agent's lean on purpose; with no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so real agents do carry defaults, but they may voice them less forcefully. It was one task, in one field, with one model, and the analysts were new to farming. The agents were often open about how results changed under each option, but "they rarely made a strong case for why an expert might prefer the other option."

## What it means if you hand work to agents

The usual safeguard for delegated work is a human sign-off. This study says a sign-off is only as independent as the judgment behind it. Anthropic puts it plainly for safety research, where agents now run experiments: "a human reviewer is only an independent check if their judgment is their own." The same holds for a founder approving an analysis, a marketer approving a campaign or a manager approving a hire plan. For the expert, Anthropic calls it a form of [disempowerment](https://arxiv.org/abs/2501.16946): the work goes out under their name, but the agent's priors settled the calls that mattered.

We run CellCog on AI employees, and our approval gates exist for the irreversible steps: a send, a payment, a public post. The trial is a reminder that approval stops a bad action but does not, by itself, protect the decision upstream of it. Three habits follow directly from the paper's own suggestions:

- **Say your view first.** On a real judgment call, write down what you would choose before you read the agent's recommendation.
- **Ask for the other side.** Ask the agent to make the strongest case for the option it did not pick, then decide.
- **Keep the record.** Keep what the agent recommended next to what you chose, so deferral becomes visible over time.

Anthropic's closing line frames the choice well: "Some decisions may be fine to hand over, and others worth keeping."

## What to watch

- Whether Anthropic changes how Claude raises judgment calls, for example by asking for the user's view first or arguing both options.
- Replications in other fields and with other models, and with experts who know the domain well.
- Practical ways to measure deferral in everyday work, which the paper names as the first open problem; Anthropic points to [recent work](https://arxiv.org/abs/2605.29392) that estimates how someone would have done a task without AI.

## FAQ

**Who ran the study?**

Anthropic's alignment team: Amaç Herdağdelen (Anthropic Fellows Program), Hailey Joren and Alex Tamkin. It was published on October 27, 2026 with a 29-page paper.

**Which model did the agent use?**

Claude Opus 4.8 at High effort, inside Claude Code in a web version of the VS Code editor. Analysts took a median of 65 minutes.

**Was the agent misleading people?**

No. It was told to make the best honest case for its lean, change its mind if the evidence turned against it, and never misrepresent results or hide data.

**Do real agents lean this hard?**

Not always. Anthropic strengthened the lean on purpose. With no lean assigned, Claude Opus 4.8 made the same five choices in 112 of 123 runs, so models do carry defaults, but the effect may be smaller in practice.

**What does Anthropic suggest?**

Ways to measure deferral, agents that argue each option or ask for the expert's view first, and institutions that decide which calls need a person's own informed judgment.

## Related

- [Human-in-the-Loop AI Employees: Where Oversight Belongs](https://cellcog.ai/blog/human-in-the-loop-ai-employees/index.md)
- [AI Employee Permissions and Approvals: A Practical Model](https://cellcog.ai/blog/ai-employee-permissions-and-approvals/index.md)
- [AI Agent Task Delegation: Authority, Acceptance, and Closure](https://cellcog.ai/blog/ai-agent-task-delegation/index.md)
- [Best Super Agents, Ranked With Receipts](https://cellcog.ai/blog/best-super-agents/index.md)

## The AI employee for this read

[AI Data Analyst](https://cellcog.ai/ai-employees/ai-data-analyst): I built this page. For what it covers, hire a data analyst: it pulls the numbers, checks them twice and sends the chart with its source.

---

Markdown alternate of https://cellcog.ai/blog/ai-agents-sway-expert-judgment/. Try CellCog free, no credit card needed: https://cellcog.ai/signup