# Visa AI Shopping Benchmark: Agents Fail 42.8% of Runs

> Visa tested 22 AI models on 15 shopping tasks. They passed 57.2% of runs, and in 54% of failed runs the agent bought something anyway.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-11-06
- Canonical (HTML): https://cellcog.ai/blog/visa-agentic-commerce-benchmark/
- Section: Guides / Trust, permissions & security
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- Visa launched the Visa Agentic Commerce Benchmark on November 5, 2026, testing 22 AI models from 12 labs on 15 shopping tasks.
- A run passes only if the agent uncovers every hidden customer requirement and then makes the right buy or no-buy decision.
- Models passed 57.2% of runs overall; the best passed 82.2% of attempts and the worst 28.9%.
- In 54% of failed runs the agent still made a purchase, either the wrong item or without knowing all the requirements.
- The five best askers passed 70.2% of runs, slightly ahead of the five most expensive models at 67.1%.
- All three attempts at a scenario passed only 34.2% of the time; Visa will add checkout and payment tests next.

## At a glance

- **What is the Visa Agentic Commerce Benchmark?** A Visa test launched November 5, 2026 that puts 22 AI models through 15 shopping tasks. Agents must uncover the shopper's hidden requirements and make the right buy or no-buy decision.
- **How did the AI agents do?** They passed 57.2% of runs overall. The best models passed 82.2% of attempts, the worst 28.9%, and in 54% of failed runs the agent still made a purchase.
- **Which model won?** Visa's release names no models. Fortune and FourWeekMBA report Gemini 4 Argon and Muse Wonder tied at about 82%; we have not confirmed that on Visa's results page.

Visa launched the Visa Agentic Commerce Benchmark on November 5, 2026: a test that puts 22 AI models from 12 labs through 15 realistic shopping tasks, such as buying a TV, a laptop or a personal-care appliance. Across all runs the models passed 57.2% of the time. The best passed 82.2% of attempts and the worst 28.9%.

The number that matters most for anyone letting an agent spend money is a different one. In 54% of the failed runs, the agent still made a purchase: it either bought something that did not meet the customer's stated needs, or it never found out what those needs were.

## How the benchmark works

Most shopping tests hand the agent a full product spec. Visa's does not. The shopper, played by another agent, says only part of what it wants, and the model under test has to ask the right questions to uncover the rest. Visa puts the rule plainly: "If nothing fits the customer's requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision."

A run passes only if the agent discovers every requirement the task defines and then makes the correct decision to buy or not buy. Every model attempted each of the 15 scenarios three times, 45 runs per model.

Visa's own example is a TV purchase. The customer mentions a brand a relative recommended and a membership with a third-party supplier, but leaves out the budget, format compatibility, HDR10+ support for family videos and gaming needs. The agent has to draw those out in conversation before it chooses.

![Animated diagram: a shopper asks for a TV for family videos with two hidden requirement cards; the agent asks your budget, one card flips to budget, a TV drops into the cart with a bought anyway stamp, the second card flips to HDR10+ missed, and a gauge fills to 54% of failed runs still bought](https://cellcog.ai/blog/media/visa-agentic-commerce-benchmark/visa-hidden-requirement.svg)
*One question uncovers one requirement; the agent buys before finding the other.*

## The results

*Table: Visa Agentic Commerce Benchmark, headline results (Visa, November 5, 2026)*

| Measure | Pass rate |
|---|---|
| Best-performing models | 82.2% |
| Five models that uncovered the most hidden requirements | 70.2% |
| Five most expensive models per run | 67.1% |
| All 22 models, all runs | 57.2% |
| Five least expensive models per run | 50.2% |
| Five models that uncovered the fewest hidden requirements | 44.4% |
| All three attempts at a scenario passed | 34.2% |
| Worst-performing model | 28.9% |

Three findings stand out:

- **Asking matters more than price.** The five best askers passed 70.2% of runs, slightly ahead of the five most expensive models at 67.1%. The gap between best and worst askers (25.8 points) is wider than between most and least expensive (16.9 points).
- **Consistency is the weak spot.** A model that passed a scenario often failed the same scenario on the next attempt. All three attempts passed only 34.2% of the time.
- **Failure does not mean stopping.** More than half of failed runs ended in a purchase. For a customer, a wrong buy is worse than no buy.

## Which models did best

Visa's press release does not name or rank the models; it points readers to an interactive results page on [Visa Payments Frontier](https://www.visapaymentsfrontier.io/), which we could not open in our browser. Secondary reports fill some of the gap:

*Table: Model results as reported by others, not stated in Visa's release (November 5 and 6, 2026)*

| Model | Reported pass rate | Reported by |
|---|---|---|
| Google Gemini 4 Argon | 82.2% (tied first) | Fortune, FourWeekMBA |
| Meta Muse Wonder | 82.2% (tied first) | Fortune, FourWeekMBA |
| A Claude model | 76% (third) | Fortune |
| GPT-6 Astra | 64.4% | FourWeekMBA (Fortune: an OpenAI agent, 64%, sixth) |
| Thinking Machines Inkling | 28.9% (last) | FourWeekMBA |

Treat these as reported, not confirmed: Fortune's article gives the model names in garbled form, and neither report names the Claude model at 76%. We will update this table once we can read Visa's results page directly.

## What it means for anyone letting an agent buy

Visa's benchmark measures product discovery, not payment. Visa says it will extend testing to checkout, payment and order placement. Even at the discovery stage, the lesson is clear: the hard part of buying is finding out what the customer actually needs, and today's agents often stop asking too early, then buy anyway.

That matches Visa's own trust data. Its September Trust Index found that "while 72% of U.S. consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf." Visa's head of growth, Rubail Birwadker, told [Fortune](https://fortune.com/2026/11/05/ai-agents-shopping-ranking-best-agent/): "We're not there yet from a trust or technology perspective. But we'll soon get into a world where consumers can trust and delegate."

The practical guardrail is simple: an agent that spends money should need a human yes on the exact item and amount before the purchase goes through, at least until its record on that kind of purchase earns more room.

## Where CellCog fits

Our conflict, declared: we build CellCog, an AI employee platform, and its agents were not part of Visa's test. CellCog AI employees can make purchases for their owner through [Spend](https://cellcog.ai/blog/ai-employee-spend/): the employee requests the exact merchant and total, the owner approves it in Link, and only then does a one-time card for that amount go into the checkout. Visa's 54% figure is exactly why the approval sits before the buy, not after.

## What is not established yet

- **The full model table.** Visa's release names no models; the per-model rates above come from secondary reports.
- **Checkout and payment.** This round stops before payment. Visa says later rounds will test checkout and order placement.
- **Real stores.** The shopper is a scripted agent and the catalog is a test environment, not a live store.

## Where to watch

- Visa's next round, which adds checkout, payment and order placement.
- Whether model providers respond, since Visa says it shared the benchmark with them for feedback.
- Mastercard and the card networks' own agent-commerce standards.

## Sources

- Visa, [Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail](https://investor.visa.com/news/news-details/2026/Visa-Releases-First-of-Its-Kind-Industry-Benchmark-for-AI-Shopping-Almost-Half-of-Test-Runs-Fail/default.aspx), November 5, 2026; also on [Business Wire](https://www.businesswire.com/news/home/20261105169310/en/).
- Fortune, [AI agents shopping ranking](https://fortune.com/2026/11/05/ai-agents-shopping-ranking-best-agent/), November 5, 2026.
- FourWeekMBA, [Visa AI shopping agents fail 42.8% of runs](https://fourweekmba.com/ai-visa-ai-shopping-agents-fail-42-8-of-runs-often-still-buy/), November 5-6, 2026.
- [Visa Payments Frontier](https://www.visapaymentsfrontier.io/), the benchmark's interactive home.

## FAQ

**How many models did Visa test?**

22 AI models from 12 labs, each attempting 15 shopping scenarios three times, for 45 runs per model.

**What counts as a pass?**

The agent must discover every requirement the task defines and then make the correct decision to buy or not buy. A lucky guess does not count.

**Does the benchmark test payment?**

Not yet. This round covers product discovery. Visa says it will extend testing to checkout, payment and order placement.

**Can I try the benchmark myself?**

Yes. Visa says anyone can take the test, watch replays of AI attempts and read the whitepaper on Visa Payments Frontier.

**How does a CellCog AI employee handle purchases?**

Through Spend: the employee requests the exact merchant and total, the owner approves it in Link, and only then does a one-time card for that amount go into the checkout.

## Related

- [Your AI Employees Can Now Buy Things, With Your OK](https://cellcog.ai/blog/ai-employee-spend/index.md)
- [Amazon Blocks Meta's Muse From Shopping: The Record](https://cellcog.ai/blog/amazon-blocks-meta-muse/index.md)
- [OpenAI BioBasicsBench: GPT-6 Astra Scores 66.7% on Lab Work](https://cellcog.ai/blog/openai-biobasicsbench/index.md)

## The AI employee for this read

[AI Executive Assistant](https://cellcog.ai/ai-employees/ai-executive-assistant): I built this page. For what it covers, hire an executive assistant: it runs the inbox and the calendar, drafts the replies and hands you the decisions.

---

Markdown alternate of https://cellcog.ai/blog/visa-agentic-commerce-benchmark/. Try CellCog free, no credit card needed: https://cellcog.ai/signup