Visa launched the Visa Agentic Commerce Benchmark on November 5, 2026: a test that puts 22 AI models from 12 labs through 15 realistic shopping tasks, such as buying a TV, a laptop or a personal-care appliance. Across all runs the models passed 57.2% of the time. The best passed 82.2% of attempts and the worst 28.9%.
The number that matters most for anyone letting an agent spend money is a different one. In 54% of the failed runs, the agent still made a purchase: it either bought something that did not meet the customer’s stated needs, or it never found out what those needs were.
On this page · 8 sectionsOpen
Visa launched the Visa Agentic Commerce Benchmark on November 5, 2026, testing 22 AI models from 12 labs on 15 shopping tasks.
A run passes only if the agent uncovers every hidden customer requirement and then makes the right buy or no-buy decision.
Models passed 57.2% of runs overall; the best passed 82.2% of attempts and the worst 28.9%.
In 54% of failed runs the agent still made a purchase, either the wrong item or without knowing all the requirements.
The five best askers passed 70.2% of runs, slightly ahead of the five most expensive models at 67.1%.
All three attempts at a scenario passed only 34.2% of the time; Visa will add checkout and payment tests next.
§ 01How the benchmark works
Most shopping tests hand the agent a full product spec. Visa’s does not. The shopper, played by another agent, says only part of what it wants, and the model under test has to ask the right questions to uncover the rest. Visa puts the rule plainly: “If nothing fits the customer’s requirements, the AI should opt not to buy. To prevent lucky guesses, the AI must uncover enough information to justify its decision.”
A run passes only if the agent discovers every requirement the task defines and then makes the correct decision to buy or not buy. Every model attempted each of the 15 scenarios three times, 45 runs per model.
Visa’s own example is a TV purchase. The customer mentions a brand a relative recommended and a membership with a third-party supplier, but leaves out the budget, format compatibility, HDR10+ support for family videos and gaming needs. The agent has to draw those out in conversation before it chooses.
§ 02The results
| Measure | Pass rate |
|---|---|
| Best-performing models | 82.2% |
| Five models that uncovered the most hidden requirements | 70.2% |
| Five most expensive models per run | 67.1% |
| All 22 models, all runs | 57.2% |
| Five least expensive models per run | 50.2% |
| Five models that uncovered the fewest hidden requirements | 44.4% |
| All three attempts at a scenario passed | 34.2% |
| Worst-performing model | 28.9% |
Three findings stand out:
- Asking matters more than price. The five best askers passed 70.2% of runs, slightly ahead of the five most expensive models at 67.1%. The gap between best and worst askers (25.8 points) is wider than between most and least expensive (16.9 points).
- Consistency is the weak spot. A model that passed a scenario often failed the same scenario on the next attempt. All three attempts passed only 34.2% of the time.
- Failure does not mean stopping. More than half of failed runs ended in a purchase. For a customer, a wrong buy is worse than no buy.
§ 03Which models did best
Visa’s press release does not name or rank the models; it points readers to an interactive results page on Visa Payments Frontier, which we could not open in our browser. Secondary reports fill some of the gap:
| Model | Reported pass rate | Reported by |
|---|---|---|
| Google Gemini 4 Argon | 82.2% (tied first) | Fortune, FourWeekMBA |
| Meta Muse Wonder | 82.2% (tied first) | Fortune, FourWeekMBA |
| A Claude model | 76% (third) | Fortune |
| GPT-6 Astra | 64.4% | FourWeekMBA (Fortune: an OpenAI agent, 64%, sixth) |
| Thinking Machines Inkling | 28.9% (last) | FourWeekMBA |
Treat these as reported, not confirmed: Fortune’s article gives the model names in garbled form, and neither report names the Claude model at 76%. We will update this table once we can read Visa’s results page directly.
§ 04What it means for anyone letting an agent buy
Visa’s benchmark measures product discovery, not payment. Visa says it will extend testing to checkout, payment and order placement. Even at the discovery stage, the lesson is clear: the hard part of buying is finding out what the customer actually needs, and today’s agents often stop asking too early, then buy anyway.
That matches Visa’s own trust data. Its September Trust Index found that “while 72% of U.S. consumers had used an AI assistant, only 23% trusted AI to handle payment transactions on their behalf.” Visa’s head of growth, Rubail Birwadker, told Fortune: “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.”
The practical guardrail is simple: an agent that spends money should need a human yes on the exact item and amount before the purchase goes through, at least until its record on that kind of purchase earns more room.
§ 05Where CellCog fits
Our conflict, declared: we build CellCog, an AI employee platform, and its agents were not part of Visa’s test. CellCog AI employees can make purchases for their owner through Spend: the employee requests the exact merchant and total, the owner approves it in Link, and only then does a one-time card for that amount go into the checkout. Visa’s 54% figure is exactly why the approval sits before the buy, not after.
§ 06What is not established yet
- The full model table. Visa’s release names no models; the per-model rates above come from secondary reports.
- Checkout and payment. This round stops before payment. Visa says later rounds will test checkout and order placement.
- Real stores. The shopper is a scripted agent and the catalog is a test environment, not a live store.
§ 07Where to watch
- Visa’s next round, which adds checkout, payment and order placement.
- Whether model providers respond, since Visa says it shared the benchmark with them for feedback.
- Mastercard and the card networks’ own agent-commerce standards.
§ 08Sources
- Visa, Visa Releases First-of-Its-Kind Industry Benchmark for AI Shopping; Almost Half of Test Runs Fail, November 5, 2026; also on Business Wire.
- Fortune, AI agents shopping ranking, November 5, 2026.
- FourWeekMBA, Visa AI shopping agents fail 42.8% of runs, November 5-6, 2026.
- Visa Payments Frontier, the benchmark’s interactive home.
Q1How many models did Visa test?
22 AI models from 12 labs, each attempting 15 shopping scenarios three times, for 45 runs per model.
Q2What counts as a pass?
The agent must discover every requirement the task defines and then make the correct decision to buy or not buy. A lucky guess does not count.
Q3Does the benchmark test payment?
Not yet. This round covers product discovery. Visa says it will extend testing to checkout, payment and order placement.
Q4Can I try the benchmark myself?
Yes. Visa says anyone can take the test, watch replays of AI attempts and read the whitepaper on Visa Payments Frontier.
Q5How does a CellCog AI employee handle purchases?
Through Spend: the employee requests the exact merchant and total, the owner approves it in Link, and only then does a one-time card for that amount go into the checkout.