« Back to Glossary Index


Data with a large amount of additional meaningless information. – Wikipedia

Relevance in CRO

Noise in your result data increases the likelihood of misinterpreting experiment results. Noisy result data is typically caused by loose experiment scoping, which allows users who are not actually exposed to variation changes to be included in the experiment.

Here’s an example: say you’re testing product page video content that displays in a modal. The modal is triggered by a “Watch Video” button. You set up the experiment to target all product pages, because their URL structure is “yourcoolstore.com/prouduct/” and that’s easy to configure. What’s wrong with that?

The variation is changing content that’s hidden behind a modal. It’s not reasonable to assume that every single user viewing a product page will click the button to trigger the modal. Because the experiment scoping, these “non-clickers” are included in the experiment. Why is this a problem? The “non-clicker” experience is the same regardless of variation bucketing for users who don’t click the “Watch Video” button. They are not opening the modal and thus they would not see the video content. The alternate video is not (potentially) affecting their behavior. This creates a significant amount of irrelevant user behavior in the goal data. This irrelevant behavior is noisy data.

Let’s add another layer to the example. Not every product has video content, so that “Watch Video” button is only on some of the product pages. You can extrapolate why that original scoping of targeting all product pages won’t work. Users viewing products that don’t have video content wouldn’t be exposed to variation changes, introducing noisy data. In order to maintain a high signal-to-noise ratio in your experiment result data, you have to consider all of the behavioral conditions that will expose a user to the tested content and tightly scope experiments accordingly.

« Back to Glossary Index