Methodology note: scraping employee reviews

Philip Morgan

I'm always on the lookout for research methods that are useful in a small-scale research (SSR) context. This study has one:

Their research question: do reviews of companies contain evidence that the culture and values described on the company's website actually exist inside the company's operation?

Any time you can build a dataset by scraping websites, it's a potential efficiency in your research effort. It can be faster and easier to scrape than to speak (with people in an interview). The quality of the scraped data, of course, could be good or garbage, but the speed with which you could assemble a sizable sample is impressive.

Here's another study that uses similar clever data-scraping: (These researchers may have had access to an API or data feed and not had to scrape web pages, but the essence is similar enough.)

The MIT Sloan piece may be subject to the Halo Effect. That would be a problem no matter what methodology they used.