Skip to content
AI EmployeeSuper-AgentsAgent-to-AgentBenchmarksPricingBlog

Colgate's AI Shoppers: What the Paper Found vs the Hype

At a glanceQuick answers
Is the Colgate paper new?
No. Version 1 went up on arXiv on October 9, 2025 and version 3 on October 27, 2025. The post that went viral on November 21, 2026 resurfaced it.
What did it find?
Asking GPT-4o or Gemini 2.0 Flash for a number gave mostly 3s. Asking for words and scoring them by meaning matched real purchase intent surveys far better.
Did it replace human research?
No. It was checked against human surveys, and the authors warn about subgroups and unfamiliar product categories.
Data illustration on a near-white ground: a toothpaste tube releases speech bubbles that turn into Likert boxes numbered 1 to 5 and feed a gauge, beside the huge figure 90% and the line of human test-retest reliability; a small label reads 57 surveys, 9,300 people.
Fig 0SSR reached 90% of human test-retest reliability on 57 surveys. Made by CellCog's image agent, running GPT Image 2.6.

A post on X claiming that Colgate killed market research passed 2.2 million views on November 21, 2026. It opens: “A toothpaste company has quietly killed the entire market research industry and nobody is talking about it.” The paper it describes is real, but it is not new: arXiv 2510.08338 was first posted on October 9, 2025, by seven researchers at PyMC Labs and two at Colgate-Palmolive.

What the paper shows is narrower and more useful than the post. Here is what it measured, how the method works, and where the authors themselves say to stop.

On this page · 6 sectionsOpen
  1. What the paper measured
  2. How semantic similarity rating works
  3. What the viral post gets right, and what it stretches
  4. What this means for teams that test products
  5. What we are watching
  6. Sources
Key points5 · 5 min full read
  1. A toothpaste tube beside a research paper with a chart.
    The paper behind the viral post is arXiv 2510.08338 by PyMC Labs and two Colgate-Palmolive authors, first posted October 9, 2025, not this week.
  2. A speech bubble with an arrow to five boxes, the fourth filled.
    Its method, semantic similarity rating, asks a model to write what it thinks of a product, then maps the text to a 1 to 5 rating by meaning.
  3. A gauge with its needle near the top.
    On 57 real surveys with 9,300 US respondents, the method reached 90% of human test-retest reliability and matched the shape of real answers.
  4. Four person icons, two solid and two dotted.
    The paper says age and income came through well, while gender, region and ethnicity were not consistently replicated.
  5. A magnifying glass with a question mark over a social post.
    The viral claim that it killed market research goes further than the authors, who say it only works where models already know the category.

§ 01What the paper measured

The authors tested synthetic shoppers on 57 personal care concept surveys a large company had already run, each with 150 to 400 US participants, 9,300 people in all. Each survey asked one question: how likely are you to buy this, on a 1 to 5 scale. The synthetic shoppers were GPT-4o and Gemini 2.0 Flash, each given a person’s age, gender, income, region and ethnicity and shown the same product concept.

The finding that starts everything is a failure. Asked for a number directly, the models “produce unrealistic response distributions when asked directly for numerical ratings”, in the paper’s words. They mostly answered 3, the safe middle.

Method How it works Ranking match Distribution match
Direct rating The model answers 1 to 5 about 80% 0.26
Follow-up rating The model writes a reaction, a second model turns it into 1 to 5 85% 0.72
Semantic similarity rating The model writes a reaction, its meaning is compared to five reference statements 90% 0.88
Table 1Three ways to get a rating from GPT-4o, per the paper (arXiv 2510.08338, v3)

Ranking match is the paper’s correlation attainment: how well the synthetic ranking of products matches the real one, as a share of the best a repeat human panel reaches. Distribution match is a Kolmogorov-Smirnov similarity between the synthetic and real spread of answers. With Gemini 2.0 Flash the pattern held: 0.39, 0.59 and 0.80 on distribution.

Distribution match with real surveys (KS similarity), GPT-4o, by methodBar chart of distribution similarity with real surveys for GPT-4o: direct rating 0.26, follow-up rating 0.72, semantic similarity rating 0.88 highlightedDirect rating0.26Follow-up rating0.72Semantic similarity0.88Distribution match with real surveys (KS similarity), GPT-4o, by methodBar chart of distribution similarity with real surveys for GPT-4o: direct rating 0.26, follow-up rating 0.72, semantic similarity rating 0.88 highlightedDirect rating0.26Follow-up rating0.72Semantic similarity0.88
Fig 1Distribution match with real surveys (KS similarity), GPT-4o, by method
Animation: an isometric platform with a terminal labelled Synthetic shopper beside a toothpaste tube; speech bubbles rise and drop into a funnel labelled Meaning match; teal bars grow over boxes numbered 1 to 5, leaning to 4 and 5; grey bars labelled Real panel rise to the same shape, with 90% of human retest between them; then the line Words in, ratings out. 90% of a human retest.
Fig 2An AI shopper's words become ratings by meaning and line up with a real panel at 90%

§ 02How semantic similarity rating works

The trick is to stop asking the model for a number. It writes a short, free reaction to the product as the person it is playing. That text is turned into an embedding with OpenAI’s text-embedding-3-small and compared with five reference statements, one for each point on the scale. The closeness to each statement becomes a probability for each rating, so one answer can be mostly a 4 and partly a 5, the way people actually hedge.

The reactions are the second product. The authors note that the written answers explain why a product appeals or not, which a bare score cannot, and that the synthetic answers showed less of the positivity bias real panels have. PyMC Labs has posted an open-source implementation.

§ 03What the viral post gets right, and what it stretches

  • Right: the 90% figure. It is in the abstract, measured on 57 surveys and 9,300 responses.
  • Right: words beat numbers. Asking for a reaction and scoring it is the paper’s core result.
  • Stretched: “perfectly mirrored”. The post says: “They perfectly mirrored how different age brackets and income levels react to price changes.” The paper says age and income came through relatively well and that gender, region and ethnicity did not, adding that persona prompting “it cannot yet be treated as a reliable proxy for all subpopulations”.
  • Stretched: killing market research. Every number in the paper is a comparison against real human surveys. The method was validated by the industry the post says it ended.
  • Missing: the limit. The authors write that for categories models know little about, “SSR will not conjure valid consumer preferences”. Oral care worked because models have read a lot about toothpaste.

§ 04What this means for teams that test products

Synthetic shoppers look ready for one job: screening many concepts cheaply before a real panel, and reading the reasons behind each score. They are not yet a stand-in for a subgroup study, and the paper tested one category of products with one company’s surveys. Our own market research workflow makes the same point from the other side: a result is only as good as the record of how it was reached.

§ 05What we are watching

  • Other categories. Whether anyone repeats the method outside personal care, where models know less.
  • Newer models. The paper used GPT-4o and Gemini 2.0 Flash; a rerun on current models would show whether the gap to humans closes.
  • A peer-reviewed version. The paper is a preprint on arXiv.

§ 06Sources

Frequently asked5 questions

Q1Who wrote the paper?

Nine authors: seven at PyMC Labs and two at Colgate-Palmolive. The data came from 57 personal care concept surveys run by Colgate.

Q2Which models did it use?

GPT-4o and Gemini 2.0 Flash for the synthetic shoppers, and OpenAI’s text-embedding-3-small to score their answers.

Q3What does 90% of test-retest reliability mean?

How well the synthetic ranking of products matched the real one, measured against the best match a second human panel would reach.

Q4Can I use it for any product?

The authors say no. It worked for oral and personal care, which models have read a lot about; where that knowledge is thin, it will not invent valid preferences.

Q5Is the code public?

PyMC Labs published an open-source implementation on GitHub.

Published 22 November 2026 All Category basics →