Replication

Philip Morgan

This is a good list of studies that haven't replicated or are otherwise on shaky ground: https://forrt.org/reversals/ (scroll about 1/3rd down to see the list).

In general, the world of marketing loves it when social science produces a study that suggests there are ways to manipulate human behavior. If a new, popular study suggests that puke green colors increase test subject's desires to spend money, within a year there would be an agency called PukeSell Partners (PukeProfit Labs would be the #2 agency in this category) and 1,000 new articles ranking in a Google search for puke green CRO.

From the aforelinked site, here are a few famous social science studies that are on shaky ground:

No good evidence of anything from the Stanford prison ‘experiment’. It was not an experiment; ‘demand characteristics’ and scripting of the abuse; constant experimenter intervention; faked reactions from participants; as Zimbardo concedes, they began with a complete “absence of specific hypotheses”.

No good evidence from the famous Milgram experiment that 65% of people will inflict pain if ordered to. Experiment was riddled with researcher degrees of freedom, going off-script, implausible agreement between very different treatments, and “only half of the people who undertook the experiment fully believed it was real and of those, 66% disobeyed the experimenter.”

Screen time and wellbeing. Lots of screen-time is not strongly associated with low wellbeing; it explains about as much of teen sadness as eating potatoes, 0.35%.

No good evidence that taking a “power pose” lowers cortisol, raises testosterone, risk tolerance.

Good and robust evidence against ego depletion, that willpower is limited in a muscle-like fashion.

I could keep going. The site is written in somewhat terse, academic fashion and so may not be the easiest read, but I think it's really useful to look down through that list and remember the hundreds if not thousands of news headlines we've all been exposed to that are based on questionable science.

With respect to marketing, I hope it points us all back to the fundamentals. A few of those fundamentals:

  • We need to create value in excess of what we charge clients.
  • If we're hoping to earn new visibility and attention, maximizing relevance, generosity, and contextual awareness will get us closer to our goal than anything else.

-P