In Sawtooth Software’s past MaxDiff designers, we’ve focused on:
- Each item appearing an equal number of times (1-way balance),
- Each item appearing an equal number of times with every other item (2-way balance),
- Each item appearing an equal number of times in each questionnaire position (positional balance, such as top, middle, or bottom slots in the MaxDiff question).
However, our past MaxDiff designers haven’t paid attention to across-set item dispersion. In other words, the same item often could appear in adjacent sets just due to the luck of the draw. Below is an example for a single questionnaire version (20 items shown 4 items at a time for 15 sets) using our previous MaxDiff designer:
Sample Version (Block): Previous Sawtooth Software MaxDiff Designer
Each “x” represents the presence of an item in the questionnaire. For example, Set 1 shows items 6, 7, 14, and 17. We’ve highlighted in dark gray where the same item appears in three consecutive sets (2 occurrences) and in lighter gray situations in which the same item appears in two consecutive sets (7 occurrences). The example above is more extreme than average with respect to consecutive occurrences, but it demonstrates how it can occur with the previous designer.
The new designer takes an additional step of sorting the sets within each version (block) to ensure better dispersion of the items. (Note, we’ve also made adjustments to the overall design algorithm, so the set composition below is different from the version above.)
Sample Version (Block): New Sawtooth Software MaxDiff Designer
With the new designer, items never appear in consecutive sets in this example.
Sorting the sets (within each version) for greater item dispersion does not change the statistical properties of the design. Yet, we think that greater item dispersion is better for reducing cognitive bias. Although we don’t expect utility results to differ much from a practical standpoint compared to the old designs in MaxDiff, this represents a reasonable and modest improvement for Sawtooth Software’s MaxDiff designer.