Definition
An alternative hypothesis that proposes a variation is not worse than the control it is being compared to.
CRO Example
Testing for non-inferiority is what you do when you want to avoid a negative result, but also aren’t necessarily concerned with achieving a positive result either. For example, say you’re thinking about swapping out the 3rd party recommendation engine your company uses because the new vendor’s software has some cool machine learning feature that will save your merchanding team an estimated 20 hours of work each week. In this scenario, you’re not concerned if the new vendor’s tool performs better than the current vendor’s tool, merely that it’s no worse. The value of the result comes from the manpower effort saved. Basically, “the change is NOT worse than what we already have”.
Non-inferiority is an often overlooked aspect of experimentation, where the focus is usually placed on positive results and creating lift. It is an additional tool for de-risking the decision-making process. Incorporating non-inferiority testing helps improve the overall value of your experimentation program. Instead of the typical binary win/save conclusive result conditions, it introduces “no harm” as a potential third condition to aim for. You can use non-inferiority to help make a decision about changes caused by factors beyond user behavior, such as increasing efficiency (like in the example above), reducing costs, brand alignment, corporate mandate, ethical implications, etc.; “fuzzy” metrics that are harder to quantify or track within an experiment.
Additional reading
- Non-Inferiority Designs in A/B Testing – Analytics Toolkit
- Beyond “One Size Fits All” A/B Tests – CXL
- Understanding Equivalence and Noninferiority Testing – National Library of Medicine