« Back to Glossary Index

Definition

A subdivision of statistics encompassing the simultaneous observation and analysis of more than one outcome variable. – Wikipedia

Relevance in CRO

Multivariate testing is an extension of A/B testing. Instead of comparing a single change, it’s comparing multiple different changes simultaneously, with each change having it’s own set of variations. Think of it like this:

results in four possible combinations:

The total potential number of combination is the multiplicative total of each change’s variation count; in this case 2 x 2 = 4. Three changes each with two variations would be 2 x 2 x 2 = 8. Test results are evaluated based on which combination of changes is the best performer.

The value of an MVT is expedience. You Instead of having to run each change as a separate A/B test in sequence, they all get bundled up into a single MVT and run concurrently. The catch is that you will need a (much) larger amount of traffic in order for MVT results to reach a meaningful sample size. Traffic gets distributed across each combination, so in the above example of 4 combinations, that means a 25/25/25/25 split.

Additional Reading

« Back to Glossary Index