Subtle sampling bias

Philip Morgan

(Readin' time: 1m 15s)

A few days back, list member Josh Earl was kind enough to point out a form of bias that can effect your efforts to validate an idea.

This got me thinking...

Your reputation filters what you hear and experience, so you might think the thing you specialize in is more common than it really is. This is a form of sampling bias.

Social media is another form of sampling bias.

I see the folks I follow on Twitter talking a lot about the long-running Macbook Pro keyboard issue and a noticeable decline in the customer experience at Apple stores. Is this a broad-based public perception issue for Apple? I have no idea. My Twitter "sample" suggests it is, but if I gathered a different sample using something like a census survey of everyone who walks out of an Apple store for a multi-day period, I'd get different data that might lead to a different conclusion. And if I gathered another sample yet, like a census of every person in Texas who has interacted in any way with the Apple brand or products, I'd get yet a different conclusion.

Bottom line: think about all the filters you have put in place, purposefully or not. These may have good effects (filtering for good clients, etc.), but they also act as a sort of sampling bias. This isn't necessarily a problem you need to fix, but it's good to be aware you are seeing a filtered view of the world. If you are trying to get an accurate picture of some part of the world that intersects with your area of expertise or are trying to be highly objective in your advice to clients, remember that you might need to compensate for sampling bias.