[PMC Weekly Insight] Survey marketing: Coding and counting

Philip Morgan

My survey marketing experiment1 continues. As a quick reminder, I'm experimenting with using a survey to connect and build trust with a group of people.

Today, we arrive at some further simple data munching. The first round of simple data munching was made up of counts and averages of the quantitative data the survey contains. This was both conceptually and mechanically simple.

The next round of data munching is also conceptually simple, but a bit more mechanically involved. This part involves:

  1. Coding the survey open-ended responses so that they are more standardized and easy to analyze.
  2. Counting the frequency of the resulting coded responses. For example, once I did the coding of the LinkedIn sample, the action of networking showed up quite a lot in the open-ended responses. But how often? And where does this activity show up in a sorted list of all activities mentioned? This is why I want to count the frequency of the coded responses.

Doing this was not difficult, and was all accomplished using Google Sheets2. The coding involved making some judgement calls. For example, when a survey respondent says "I do a lot of experimenting with programming patterns to find alternate solutions", how do I best code that? That's the kind of judgment call I'm talking about. (I went with "a-coding-learning" in this case.)

Then I used a pivot table to count the frequency of the coded responses. Then I sorted the results of the pivot table by the count of each response.

Here's what I came up with (get ready to do some SCROLLIN'!):

In case it's not easy to read, the above is: "Coded responses to '2. Please list ways you have you spent time and money for career development.'"

The above is: "Coded responses to '4. Please list ways you have you spent time and money for developing your technical skills.'"

The above is: "Coded responses to '6. Please list ways you have you spent time and money for business or self-employment skills?'"

The above is: "Coded responses to '7. Consider your entire career as a self-employed software developer and times you have gotten new opportunities, better projects, or other forms of career improvement. What do you think led to these improvements in your career?'"

I experimented with putting the un-summarized list of coded responses into a word cloud generator, which was fun and something I might include in the report I write up about this, but I think the word cloud obscures more than it reveals when compared to a simple table.

Elevator pitch summary of your research

If I had just 20 seconds or so to summarize what this research is teaching me, I'd say the following:

The self-employed software developers I've surveyed "in the wild" invest in career development with a heavy usage of online learning platforms like Pluralsight and IRL events, and they find new or better opportunities primarily through networking and experimenting with their own business. They invest about 300% more in cultivating technical skills than they do in cultivating business skills. In my study, they used the word "marketing" exactly zero times.

I want to point out that the next-to-last sentence is purposefully constructed to be provocative, but its support in my data is questionable. Or rather, "they invest about 300% more in cultivating technical skills than they do in cultivating business skills" is one of several possible framings for the underlying data. Here are a few other possible framings:

  • Super factual: "When asked about how they invest in technical skills, respondents are about three times more verbose than when asked about how they invest in business or self-employment skills."
  • Less provocative, still attempting to be factual: "I don't have data on exactly how much time or money self-employed devs invest in technical vs. business skills, but my data does show that they clearly emphasize investing in technical skills over business skills."
  • Simple, but suggesting a motive that the data might not support: "Self-employed devs seem way more interested in technical skills than business or self-employment skills like marketing."

To be clear: I'm not talking about how I'd interpret or frame this data in the body of a report, but rather in a time-compressed situation where being memorable and somewhat provocative is more important than being accurate or nuanced. In that time-compressed situation, impact is created differently than it is in a less time-bound situation.

This all points to the actually difficult part: interpreting this data. Some of the key difficulties include:

  • Attributing intent or motive. What do I make of the fact that my respondents are more verbose in responding to questions about technical skills?
  • Interpreting importance. When I code and summarize the responses to open-ended questions, my list of codes for the "technical skills" and "where does opportunity come from" questions are much longer lists than the list of codes for the "business/self-employment skills" question. Maybe this does not mean that respondents actually emphasize the technical skills. Maybe it means they get lower ROI from that investment, so there's more of that investment but less result from it. Maybe it means it simply requires more words to describe how they invest in tech skills, but if we look at the investment in terms of ROI or something else, a different picture would emerge.

Interviews with some of my respondents would help clarify the questions above. In fact, I think it would be irresponsible of me to draw firm conclusions from this data without conducting around 5 interviews to better understand my respondents motive and thinking.

Next week: I'll have had time to code and summarize the responses to the other sample, which is folks from this very email list, and so I'll be able to compare the two samples.

Questions on this stuff? I'd love to hear 'em. Hit REPLY :)


Recent Daily Insights

[display-posts posts_per_page="3" include_excerpt="true" category="daily-insight"]

  1. If you want to read up on this experiment:
    1. philipmorganconsulting.com/pmc-survey-marketing/
    2. philipmorganconsulting.com/pmc-the-de-biasing-survey/
    3. philipmorganconsulting.com/pmc-survey-marketing-recruitment/
    4. philipmorganconsulting.com/pmc-survey-marketing-initial-data-from-the-de-biasing-survey/
    5. philipmorganconsulting.com/pmc-weekly-insight-survey-marketing-qualitative-analysis-of-the-de-biasing-survey/
  2. There are better tools, like Delve, for doing this, but Google Sheets is good enough for my needs here.