CROWeekly: Case Studies and Roundtables

Case studies are a bit of a mixed bag in the CRO space. For every thorough, rigorous example there seem to be a dozen more touting a wholly unrealistic 300% uplift. Often, what you don’t see is that lift was based off of a tiny sample size, like jumping from 1 conversion to 4 off of 50 visitors. While technically that IS a 300% increase, a sample size of 50 is far too small to find a meaningfully measurable impact; it’s a bad result backed up by bad stats. That being said, it’s rather difficult to prove you know what you’re talking about without putting your money where your mouth is. With that in mind, we’re launching a case study portal and publishing our first case study today: “Dead Clicks: When Less Makes More”.

What makes our case studies different? Well, we’re pushing for full transparency by including as much information as possible. Each case study explains our testing process and the stats behind the results, including:

• research conducted to identify the area of opportunity
• hypothesis that forms the basis of the test
• mockups of the variations
• stats used for planning and execution of tests, such as:
  • run duration
  • sample size
  • minimum conversions
  • minimum detectable effect (MDE)
  • confidence threshold

In addition to that level of detail, uncropped screenshots of relevant KPIs taken straight from the testing tool are on display for all to see. Those are the exact numbers and graphs we use for result analysis. See something odd? Call us out on it! After all, we’re not afraid to “Eat Crow”.