Customer experience (CX) research answers two important questions:
- How are we performing?
- How can we best improve performance?
Answering the first question involves having respondents provide some overall evaluation of their experience like customer satisfaction (Westbrook 1980) or advocacy (Reichheld 2003). While both measures (and others) have their advocates, no one doubts the need for some sort of report card for the overall rating of the product or service.
To answer the second question, the CX industry moved from asking stated importances to deriving importances statistically. This was a good idea, since importance ratings were found to have no (or, incredibly, negative) predictive validity (Bass & Wilkie 1973; Beckwith & Lehmann 1973; Wilkie & Pessemier 1973, Chrzan and Golovashka 2006). Unfortunately, deriving importance was not without problems (Dawes and Corrigan 1974, Peterson and Wilson 1992). CX researchers deserve a better way to answer the second question. I recommend PIE.
Performance Improvement Experiments (PIE) leverage the findings of Thurstone (1927) showing that respondents can relate their preferences accurately if we ask them to make repeated choices among experimentally designed stimuli. In a PIE, we show a given respondent different subsets of attributes that might be improved and ask her which attribute, if improved, would most elevate her satisfaction with the product or service. Take for instance Karl’s Suites, a hotel chain serving business travelers. The first question in a PIE exercise for Karl’s might be:
Which of the following, if noticeably improved, would most have made your stay at Karl’s Suites better?
[ ] Smoother reservation process
[ ] Easier to redeem rewards
[ ] Cleaner bathtub
[ ] Better lighting in the parking lot
[ ] Less outside noise audible in the room
[ ] None of these would have made my stay at Karl’s Suites any better at all
The next question might be:
Which of the following, if noticeably improved, would most have made your stay at Karl’s Suites better?
[ ] Cleaner feeling bedroom
[ ] Better selection of foods at breakfast
[ ] Better quality of food at breakfast
[ ] More water pressure in the shower
[ ] A white noise machine on the nightstand to help me sleep
[ ] None of these would have made my stay at Karl’s Suites any better at all
And so on for 10 or 15 or so such questions.
You may recognize this as a type of MaxDiff question where we only ask for the best response, and in this case you won’t be surprised that the result is a set of utilities like you would get in a MaxDiff study:
Notice that this is a type of anchored MaxDiff, because (all or some of) the questions contain the “None of these” anchor. This allows Karl’s Suites to understand which attributes might not move the needle at all.
To learn more about using PIE for CX research or to get help fielding a PIE study, please contact our analytical consulting group by emailing keith@sawtoothsoftware.com.
References
Bass, F.M. & Wilkie, W.L. (1973) “A comparative analysis of attitudinal predictions of brand preference,” Journal of Marketing Research, 10, 262–269.
Beckwith, N.E. & Lehmann, D. (1973) “The importance of differential weights in multi-attribute models of consumer attitude,” Journal of Marketing Research, 10, 141–145.
Chrzan, K. and N.Golovashkina (2006) “An empirical test of six stated importance measures,” International Journal of Market Research, 48(6), 717-740.
Dawes, R.M. and B. Corrigan (1974) “Linear Models in Decision Making,” Psychological Bulletin, 81: 95-106.
Peterson, R.A. and W.R. Wilson (1992) “Measuring customer satisfaction: fact and artifact,” Journal of the Academy of Marketing Science, 20(1), 61-71.
Reichheld, F.F. (2003) “The one number you need to grow,” Harvard Business Review, 81, 46-55.
Thurstone, L. L. (1927) “A law of comparative judgment,” Psychological Review, 34(4), 273–286.
Westbrook, R.A. (1980) “A rating scale for measuring product/service satisfaction,” Journal of Marketing 44(4), 68-72.
Wilkie, W.L. & Pessemier, E.A. (1973) “Issues in marketing’s use of multi-attribute attitude models,” Journal of Marketing Research, 10, 428–441.