The Challenge of Analyzing Open-Ended Survey Questions

Survey Tips
February 28, 2013

At Alchemer we believe that numbers have no meaning without understanding the story behind them, and we are not alone!

From the Alchemer Benchmark Guide Survey, we found that open-ended survey questions are being used frequently in online questionnaires. Sixty-four percent report using text box survey questions and 52 percent report using essay style questions in their online surveys.1

Open-ended survey questions provide commentary that goes hand-in-hand with your closed-ended questions.

We recommend that you use these survey question types to get context for your closed-ended response questions. You can add them as an “other” response in single or multi-select questions, as a text box asking for short response or an essay question, requesting a longer response from your respondents.

Note, that open-ended survey question types can be highly fatiguing, so please use these only when most necessary, and do not make them required in your surveys.

Building these survey question types into your survey is the easy part.

Analyzing the data is where the challenge begins, but it also delivers the rewards! There are several tools that claim to analyze open-ended data, but this market researcher thinks it is best if done manually, using a bucketing technique.

Here are some steps to follow when you manually bucket your open-text survey question data:

Read Every Response

This may take quite a long time, but it is worth it! You will really get to know the data. As you go through the responses, you will start to see some trends.

Be sure to mark some quotes that resonate with you.

Put Each Comment into One or Multiple Buckets

Next, develop buckets or categories for the different trends that you see in the responses. Each response should go into at least one bucket.

Sometimes there are multiple ideas expressed in a comment and therefore it may belong in multiple buckets. This process is often referred to as “multi-coding”.

Review & Compress Buckets

After all the items are bucketed, review each bucket and check to see if some buckets may be able to be combined, and if others may need to be split into two.

By doing this you will be able to identify clear trends, issues and new ideas suggested by your respondents.

Focus on Meaning & Context

Once all the responses are in buckets and coded, you still need to make the data useful. You’ll want to couple your open-ended responses with relevant close-ended responses to get actionable data.

Maybe you will find an “other” response that is a clear winner in a single-select question, or you will find out new information about why your customers choose your product or service. Note, that when using open-ended data, we tend to stay away from using percentages and use the data as directional with relevant terms.

If there are a small number of respondents, you can mention the actual number of respondents that provided a particular bucket of ideas.

Use Those Buckets & Quotes

Once you have analyzed your open-ended data and identified the major trends, your next step is to write a summary of what you have found. This can be descriptive text incorporating comments and quotes directly from the respondents.

You can also use quotes:

  • On your cover page to make the report seem approachable and add appeal to the reader
  • To open up a live presentation to engage the audience
  • To highlight each section or chapter of you written report
  • When building personas
  • In your team or department mission statement

We hope you use these techniques to analyze your next set of results. In fact, it is not too late to go back through your last data set and add context to your numbers.

1 Source: Alchemer Market Research Benchmark Guide: 2012, Comparative Analysis on Survey Metrics, Techniques and Trends, Do you use open-ended text questions in your survey? Please select all that apply. n=876 Total Sample

  • Get Your Free Demo Today
    Get Demo
  • See How Easy Alchemer Is to Use
    See Help Docs
  • Start making smarter decisions

    Start a free trial