# Colgate's AI Shoppers: What the Paper Found vs the Hype

> A post claiming Colgate killed market research hit 2.2M views. The paper is from October 2025; here is what its 57 surveys showed and what it did not.

- Author: Nitish Garg, Founder & CEO, CellCog
- Published: 2026-11-22
- Canonical (HTML): https://cellcog.ai/blog/colgate-synthetic-consumers-ssr/
- Section: Insights / Category basics
- Publisher: CellCog (https://cellcog.ai), the AI employee platform. Blog index for agents: https://cellcog.ai/blog/llms.txt

## Key points

- The paper behind the viral post is arXiv 2510.08338 by PyMC Labs and two Colgate-Palmolive authors, first posted October 9, 2025, not this week.
- Its method, semantic similarity rating, asks a model to write what it thinks of a product, then maps the text to a 1 to 5 rating by meaning.
- On 57 real surveys with 9,300 US respondents, the method reached 90% of human test-retest reliability and matched the shape of real answers.
- The paper says age and income came through well, while gender, region and ethnicity were not consistently replicated.
- The viral claim that it killed market research goes further than the authors, who say it only works where models already know the category.

## At a glance

- **Is the Colgate paper new?** No. Version 1 went up on arXiv on October 9, 2025 and version 3 on October 27, 2025. The post that went viral on November 21, 2026 resurfaced it.
- **What did it find?** Asking GPT-4o or Gemini 2.0 Flash for a number gave mostly 3s. Asking for words and scoring them by meaning matched real purchase intent surveys far better.
- **Did it replace human research?** No. It was checked against human surveys, and the authors warn about subgroups and unfamiliar product categories.

**A post on X [claiming](https://x.com/HowToPrompt__/status/2124040349068964215) that Colgate killed market research passed 2.2 million views on November 21, 2026.** It opens: "A toothpaste company has quietly killed the entire market research industry and nobody is talking about it." The paper it describes is real, but it is not new: [arXiv 2510.08338](https://arxiv.org/abs/2510.08338) was first posted on October 9, 2025, by seven researchers at PyMC Labs and two at Colgate-Palmolive.

What the paper shows is narrower and more useful than the post. Here is what it measured, how the method works, and where the authors themselves say to stop.

## What the paper measured

The authors tested synthetic shoppers on 57 personal care concept surveys a large company had already run, each with 150 to 400 US participants, 9,300 people in all. Each survey asked one question: how likely are you to buy this, on a 1 to 5 scale. The synthetic shoppers were GPT-4o and Gemini 2.0 Flash, each given a person's age, gender, income, region and ethnicity and shown the same product concept.

The finding that starts everything is a failure. Asked for a number directly, the models "produce unrealistic response distributions when asked directly for numerical ratings", in the paper's words. They mostly answered 3, the safe middle.

*Table: Three ways to get a rating from GPT-4o, per the paper (arXiv 2510.08338, v3)*

| Method | How it works | Ranking match | Distribution match |
|---|---|---|---|
| Direct rating | The model answers 1 to 5 | about 80% | 0.26 |
| Follow-up rating | The model writes a reaction, a second model turns it into 1 to 5 | 85% | 0.72 |
| Semantic similarity rating | The model writes a reaction, its meaning is compared to five reference statements | 90% | 0.88 |

Ranking match is the paper's correlation attainment: how well the synthetic ranking of products matches the real one, as a share of the best a repeat human panel reaches. Distribution match is a Kolmogorov-Smirnov similarity between the synthetic and real spread of answers. With Gemini 2.0 Flash the pattern held: 0.39, 0.59 and 0.80 on distribution.

![Animation: an isometric platform with a terminal labelled Synthetic shopper beside a toothpaste tube; speech bubbles rise and drop into a funnel labelled Meaning match; teal bars grow over boxes numbered 1 to 5, leaning to 4 and 5; grey bars labelled Real panel rise to the same shape, with 90% of human retest between them; then the line Words in, ratings out. 90% of a human retest.](https://cellcog.ai/blog/media/colgate-synthetic-consumers-ssr/words-in-ratings-out.svg)
*An AI shopper's words become ratings by meaning and line up with a real panel at 90%*

## How semantic similarity rating works

The trick is to stop asking the model for a number. It writes a short, free reaction to the product as the person it is playing. That text is turned into an embedding with OpenAI's text-embedding-3-small and compared with five reference statements, one for each point on the scale. The closeness to each statement becomes a probability for each rating, so one answer can be mostly a 4 and partly a 5, the way people actually hedge.

The reactions are the second product. The authors note that the written answers explain why a product appeals or not, which a bare score cannot, and that the synthetic answers showed less of the positivity bias real panels have. PyMC Labs has posted an [open-source implementation](https://github.com/pymc-labs/semantic-similarity-rating).

## What the viral post gets right, and what it stretches

- **Right: the 90% figure.** It is in the [abstract](https://arxiv.org/abs/2510.08338), measured on 57 surveys and 9,300 responses.
- **Right: words beat numbers.** Asking for a reaction and scoring it is the paper's core result.
- **Stretched: "perfectly mirrored".** The post says: "They perfectly mirrored how different age brackets and income levels react to price changes." The paper says age and income came through relatively well and that gender, region and ethnicity did not, adding that persona prompting "it cannot yet be treated as a reliable proxy for all subpopulations".
- **Stretched: killing market research.** Every number in the paper is a comparison against real human surveys. The method was validated by the industry the post says it ended.
- **Missing: the limit.** The authors write that for categories models know little about, "SSR will not conjure valid consumer preferences". Oral care worked because models have read a lot about toothpaste.

## What this means for teams that test products

Synthetic shoppers look ready for one job: screening many concepts cheaply before a real panel, and reading the reasons behind each score. They are not yet a stand-in for a subgroup study, and the paper tested one category of products with one company's surveys. Our own [market research workflow](https://cellcog.ai/blog/ai-market-research-workflow/) makes the same point from the other side: a result is only as good as the record of how it was reached.

## What we are watching

- **Other categories.** Whether anyone repeats the method outside personal care, where models know less.
- **Newer models.** The paper used GPT-4o and Gemini 2.0 Flash; a rerun on current models would show whether the gap to humans closes.
- **A peer-reviewed version.** The paper is a preprint on [arXiv](https://doi.org/10.48550/arXiv.2510.08338).

## Sources

- Maier, Aslak, Fiaschi, Rismal, Fletcher, Luhmann, Dow, Pappas and Wiecki, [LLMs Reproduce Human Purchase Intent via Semantic Similarity Elicitation of Likert Ratings](https://arxiv.org/abs/2510.08338), arXiv 2510.08338, v1 October 9, 2025, v3 October 27, 2025 ([HTML version](https://arxiv.org/html/2510.08338))
- @HowToPrompt__ [on X](https://x.com/HowToPrompt__/status/2124040349068964215), 07:20 UTC, November 21, 2026
- PyMC Labs, [semantic-similarity-rating](https://github.com/pymc-labs/semantic-similarity-rating) on GitHub

## FAQ

**Who wrote the paper?**

Nine authors: seven at PyMC Labs and two at Colgate-Palmolive. The data came from 57 personal care concept surveys run by Colgate.

**Which models did it use?**

GPT-4o and Gemini 2.0 Flash for the synthetic shoppers, and OpenAI's text-embedding-3-small to score their answers.

**What does 90% of test-retest reliability mean?**

How well the synthetic ranking of products matched the real one, measured against the best match a second human panel would reach.

**Can I use it for any product?**

The authors say no. It worked for oral and personal care, which models have read a lot about; where that knowledge is thin, it will not invent valid preferences.

**Is the code public?**

PyMC Labs published an open-source implementation on GitHub.

## Related

- [AI Market Research Workflow: Sources, Synthesis, and Review](https://cellcog.ai/blog/ai-market-research-workflow/index.md)
- [What Is an AI Employee? The 5-Part Test for a Standing AI Worker](https://cellcog.ai/blog/what-is-an-ai-employee/index.md)

## The AI employee for this read

[AI Research Assistant](https://cellcog.ai/ai-employees/ai-research-assistant): I built this page. For what it covers, hire a research assistant: it reads the primary sources and cites every line.

---

Markdown alternate of https://cellcog.ai/blog/colgate-synthetic-consumers-ssr/. Try CellCog free, no credit card needed: https://cellcog.ai/signup
