What LLMs Get Right, and Wrong, About Your Customers
Large language models are now embedded in how many brands do market research — simulate consumer responses, get results in hours instead of weeks, at a fraction of the cost.
But a rigorous Harvard Business School and Microsoft Research working paper finds that LLM-based preference estimates are realistic in some contexts and wrong-signed in others — with estimates sometimes off by a factor of three — and that a researcher without a human benchmark has no basis for knowing which is which. A Journal of Marketing study finds the human-LLM hybrid outperforms both approaches alone, but only when the human judgment is genuine rather than a pass-through. And research published in Harvard Business Review finds that LLMs consistently recommend strategies aligned with managerial buzzwords rather than context-specific logic — a pattern the researchers named trendslop.
Taken together, the evidence points to a specific map: LLMs for market research are reliable supplements when you have prior human data from the same category, unreliable substitutes when you don’t, and systematically biased toward the fashionable on exactly the questions where conventional industry thinking most needs challenging.
This episode makes that map practical — three questions a brand leader should ask before acting on what AI-assisted research tells them.
Know Your Audience is a weekly podcast for the leaders making consequential brand decisions while the ground shifts beneath them. CMOs, CPOs, and CEOs face a fundamental change in how their organizations can understand customers, and the decisions that depend on it. Produced by Soulmates.ai.