Terminating “Bad Actor” Respondents in Real Time Using MaxDiff

Last updated: 03 Mar 2022

Gary Chan Yz Sz N3qv Heo Unsplash

Executive Summary: You can identify bad respondents the moment they click the last MaxDiff question and skip them as terminate-disqualified out of your survey.  This means they don’t fill your quotas and you don’t have to pay for bad data. 

Background Issue: 

One of the challenges survey researchers deal with today is the increase in bad respondents.  Researchers often delete 10-20% (or more) of the sample from online panel sources due to consistency checks. However, many professional respondents have gotten good at avoiding speed traps and typical kinds of “gotcha” questions that we design into surveys.  

Why MaxDiff Is a Clever Tool for Catching Cheaters:  

MaxDiff (best-worst scaling) is extremely hard for bad actors to fool.  You have to be consistent, without easy-outs that would superficially seem to be from a consistent respondent, such as frequently picking “None” or the lowest priced alternative in a CBC task.  MaxDiff is tougher to trick. 

On-the-fly MaxDiff utility estimation gives us a fit statistic (RLH) that can very reliably flag whether a respondent is answering consistently or not.  When the respondent clicks the last MaxDiff question, the RLH check can be called and computes in about a second.  If each respondent sees each item at least 4x, Chrzan and Orme show that you can flag and terminate respondents very accurately (Chrzan & Orme, 2022).  80% of random respondents can be flagged, with only a 1% false identification of real respondents (and those real respondents that are falsely identified have utilities with low signal-to-noise ratio). 

Even if your study didn’t intend to use a MaxDiff section, you can still include a short 6-item MaxDiff with 8 questions (showing 3 items at a time) in any survey for the purpose of catching bad actors.  In a recent study we conducted, it took just 88 seconds to do the short MaxDiff.  We placed the MaxDiff questions right before our CBC exercise and, interestingly enough, the act of completing the MaxDiff questions led to better CBC data! …that’s a story we’ll be telling in our Sawtooth Software conference paper for May, 2022 in Florida. 

How to Flag Bad MaxDiff Respondents in Lighthouse Studio or Discover: 

Both Lighthouse Studio and Discover include a MaxDiff on-the-fly utility estimation command that can be invoked in the logic of your skip pattern.  It computes within a second or so, returns the fit score for the respondent, and skips respondents with low fit out of your survey.  An example is provided in Chrzan & Orme, 2022. 

More Information: 

Watch the recent webinar on this subject, by Keith Chrzan and Bryan Orme. 

Download the white paper, also by Keith Chrzan and Bryan Orme