Issues With The Likert Scale
The purpose of this series isn’t to bash one score, such as the NPS. Believe me when I tell you, we too have fallen victim to legacy and tradition. Nearly every survey we have administered in the last 15 years incorporated a standard Likert scale such as of 1-5 or 1-7. However, when we re-analyzed the results across several studies, we were both concerned and enlightened with the results.
Let’s start with a survey about what is most important to a customer. If a market researcher designing a study does their up-front homework well, they already know that the attributes they include in a survey have some level of importance to a customer; otherwise they wouldn’t include them. So when you ask a customer what is most important on a scale of 1-5, or 1-7, you will see a large majority of responses fall in the “top 2 box” (which means a 4 or 5 on a 1-5 scale or a 6 or 7 on a 1-7 scale); rarely will you find low end responses, such as a 1 or 2.
What can you conclude from this? Can you tell what is truly most important across a range of attributes…for example, most important to your decision to renew a contract or license, or to your level of satisfaction? It’s challenging. So we look to other methodologies, such as force ranking attributes. For example, please rank these 5 attributes from most to least important. But if we know in advance that the attributes presented already have some level of importance to the respondent (otherwise they wouldn’t be in the study), coupled with the fact that many consumers are inherently indecisive -- force ranking is difficult. You may be able to identify the top 2 or 3, but ranking the list becomes more difficult the longer the list is. As a result, respondents can become frustrated and then rush to force rank a list to close out the survey. The result, you get a bunch of data that may be both unpredictable and inaccurate.
Sample Size Challenged
What about sample sizes? Statisticians will tell you that if you have a sample size of 30, 50 or even 75 in any particular segment, this will eliminate sample bias and you can form a justified conclusion. Well, statistically they are correct, but what is missing is that statistically, they have not accounted for the scale biases mentioned previously in the Gallup article. Unless you have answers from every segment, subsegment, and sub-subsegment for every culture, nation, language, sex, etc., forming conclusions on a “believed to be” homogenous sample is difficult, since the sample is not really homogenous.
What does this mean for you? Whether a market researcher or a user of market research conducted on your behalf, have you ever made decisions based on research that you then found did not prove out as expected? Have you ever found yourself asking “well the research told me this is how my target audience feels” when in fact this proved to be wrong? Perhaps now you understand why this may have happened.
In addition, because traditional approaches have not changed, many users of these surveys are gaming the system to drive up their scores (often because their compensation or performance reviews are dependent on it). How many have completed a product or service call, when the customer service representative says, “Have I met all of your expectation and will you please give me a top rating?” I am sure anyone who has received auto service from a major dealer has experienced this. Couple answer bias with sample bias, and we can get a false sense of security that our customers are happy and that our customer service reps are doing a good job.
And how many reading this article use standard Likert scales to evaluate customer sentiment or customer service? I am sure most companies that provide products and/or services do. That’s why the largest market research software vendors like Qualtrics, Medallia and even SurveyMonkey continue to experience explosive growth. They have all convinced the market that they have the optimal platform to measure customer sentiment and satisfaction. Well as we have now learned, this may not be the case, since their platforms are based on the traditional ways of doing market research, with traditional scales.
...so stay tuned for The Death Of Traditional Market Research - Part 4 of 5
Want to receive future articles and blog posts? Interested in learning more?