Definition
A subdivision of statistics encompassing the simultaneous observation and analysis of more than one outcome variable. – Wikipedia
Relevance in CRO
Multivariate testing is an extension of A/B testing. Instead of comparing a single change, it’s comparing multiple different changes simultaneously, with each change having it’s own set of variations. Think of it like this:
- Change 1: Variation 1A & Variation 1B
- Change 2: Variation 2A & Variation 2B
results in four possible combinations:
- 1A + 2A
- 1A + 2B
- 1B + 2A
- 1B + 2B
The total potential number of combination is the multiplicative total of each change’s variation count; in this case 2 x 2 = 4. Three changes each with two variations would be 2 x 2 x 2 = 8. Test results are evaluated based on which combination of changes is the best performer.
The value of an MVT is expedience. You Instead of having to run each change as a separate A/B test in sequence, they all get bundled up into a single MVT and run concurrently. The catch is that you will need a (much) larger amount of traffic in order for MVT results to reach a meaningful sample size. Traffic gets distributed across each combination, so in the above example of 4 combinations, that means a 25/25/25/25 split.
Additional Reading
- When to Do Multivariate tests Instead of A/B/n Tests – CXL
- Multivariate Testing: Promises and Pitfalls for High-Traffic Websites – Conversion Sciences
- A Practical Guide to Multivariate Testing – VWO