5 Key Elements of Effective Survey Design
/
Surveys of employees, clients, and end users are an excellent way to gather data about human experiences and perspectives that can inform future decisions and initiatives. In recent months, we’ve noticed a definite uptick in surveys being used by both public agencies and private design and construction companies. While effective, surveys are not without their limitations; without a careful approach to both design and data analysis, we may end up getting limited data—or worse, misleading data.
As survey designers and data analysts, these risks are foremost in our minds when we’re developing and implementing surveys for our clients. Here are five of the key elements we think about when designing surveys:
If you wouldn’t take it, don’t give it – When you implement a survey, you’re asking people to spend time answering your questions instead of whatever else they’d rather be doing. Make it worth their while so you’ll get good data back. There are many reasons a person might start a survey then stop without submitting any data—perhaps they find the questions too personal, or irrelevant to them, or the survey just seems too long. These are just a few reasons someone may become disinterested in a survey and stop partway through.
Since some data is better than no data, think critically about the length of your survey and the types of questions you ask to distill your survey down to just what a reasonable audience would be willing to answer. Put yourself in your audience’s shoes—if you wouldn’t want to take the time to respond to your survey, neither will your audience.
Keep your scales consistent – Everyone likes attaching numbers to things to make them quantifiable, even people who don’t like math (or ‘maths’ if you’re one of those who think ’mathematics’ is plural and, therefore, that a single ‘mathematic’ is a thing). Attaching a number to scaled responses is a necessary practice in analyzing data, and it’s important to keep in mind that scales are all relative: a mean of 4.2 would be bad in a 9-point scale, middling in a 7-point scale, good in a 5 point scale, and a sign of something very wrong in a 3 point scale.
To save yourself the hassle of constantly checking back to see if a number is good or bad in the scale context of each particular question, do yourself a favor and make sure that the numbers you end up with can always be interpreted in a uniform way. Simply having the same number of scaled options in every question is the best way to ensure this. I like using the seven-point Likert scale: it provides enough options for nuanced responses (e.g., slightly positive, positive, and extremely positive) without providing too many options that distinguishable meaning breaks down (when was the last time you ranked something out of 10 and had strong feelings about something being a 7 and definitely not a 6 or an 8? I’m guessing never). In the rest of this post, we’ll be using a 7-point scale whenever numbers are mentioned.
The importance of Mean and standard deviation are not what you think – Mean isn’t as important as most people think. By itself, the mean is just a way of measuring the middle point in your data. Consider a mean of 4 on a 7-point satisfaction scale. This could be attained in many ways, but two extreme examples are Scenario A: having all people surveyed respond with a 4, or Scenario B: having 50/50 split between of respondents answering 1 and 7. In this situation, you wouldn’t be able to tell if you had an extremely bored population or were days away from a revolution.
This is where the standard deviation comes in and why it’s more important than most people think. Without getting overly technical, the standard deviation sort of measures the average distance between the mean and the data points (if you’d like to geek out with me about the differences between the and norms, which is what’s going on here, please give me a call. It’ll be fun, I promise). In other words, it measures how spread out the data is.
In Scenario A, where all of our participants responded with a 4, the standard deviation is 0—the data is clustered and not spread out at all. In Scenario B, where participants split between 1 and 7, the standard deviation is 3—the “average” distance between a data point and the mean is 3, so the “typical” data point is either a 1 or a 7. The standard deviation along with the mean allows us to identify that while both populations have the same mean, the population sampled in Scenario A don’t really care about an issue, while the population in Scenario B is completely polarized.
There is a difference between “Neutral” and “No Opinion” – In the English language, we sometimes treat “Neutral” and “No Opinion” as interchangeable. They are not, especially on a survey. On a positive/negative scale, neutral means that you don’t have strong feelings in either direction. You still have feelings, they’re just neither positive nor negative. As such, coding neutral as the midpoint value of 4 is the way to go. On the other hand, ‘No Opinion’ means that you have no real feelings at all. This is what you answer when you’re asked about lighting in a room you’ve never been in, or the airflow in a restaurant you didn’t know existed. This answer cannot be quantified because there is simply nothing to quantify, and if you have a “no opinion” option instead of a “neutral” one, you’re working on a 6-point scale with no middle option.
Now, one may think that this is just semantics and that the person taking the survey will understand the intent. Unfortunately, this is not always the case, and that uncertainty undermines the responses we get. Because of the human tendency to confuse neutral feelings with no feelings when we’re not being mindful, we don’t know if someone who checks the no opinion box has neutral feelings on the question’s subject—which we want to record—or genuinely has no opinion—which we’d like to ignore.
Anonymous is not the same as confidential – Keeping survey responses confidential is what makes your audience feel comfortable enough to give honest answers to sensitive questions. While true anonymity would be enough to ensure confidentiality, the simple fact is that survey data isn’t truly anonymous. To those inclined to use it, demographic information, survey metadata, and answers to a few key questions—especially qualitative questions where writing style becomes a factor—are often enough to determine who filled out a survey to a handful of people.
If participants have been assured of confidentiality, then the survey should be conducted by a neutral third party and that third party should never share the specific responses received. Instead, the third party should analyze the quantitative data and share only the aggregate data and conclusions in a summary report. For qualitative data, we should partially quantify it (e.g., 23% of respondents were concerned about the pool, 10% had positive comments about the water features, etc.) and summarize it so that conclusions may be drawn. These summaries should discuss repeated themes and comments, but they should never directly quote a respondent, lest said respondent be identifiable by speech patterns evident in their writing. For instance, no one else could possibly have written that last sentence, because no one else in the modern United States uses ‘lest’ unironically.
I cannot overstate how important the commitment to confidentiality is in survey responses to ensure the integrity of the data and the validity of the conclusions that can be drawn from it. If you find yourself conducting a confidential external survey, you must ensure true confidentiality. If you find yourself needing a confidential internal survey, find a third party to do it.
Lucas Chaffee leads CRNW’s research services. He holds a PhD in Math and recognizes that words ending in S aren’t always plural.
