Possible experiment outcomes

Written by Guido Jansen in
September 2017

Question: what are the different possible outcomes of A/B test experiments and what do you do depending on each of those scenarios?

Answer:

  1. Findings support hypothesis - Action: implement winner and/or re-test (more subtle variants or on other sites/platforms) to validate hypothesis.
  2. Findings refute hypothesis - Action: create other variants to re-test hypothesis or abandon hypothesis.
  3. No change found (but sample size requirement is reached) - Action: Check to see if we can think of more extreme variants to test the hypothesis.
  4. Experiment error, didn’t reach sample size etc. etc. - Action: Redo the test with new measures in place to prevent the errors.

It’s important to regularly share these findings (at least options 1,2 and 3) with everyone in the team/company to get feedback and to increase everyone’s knowledge of our customers behaviour.

What do you think? Are there more outcome options for your CRO experiments?

Recent posts
AI & Personalization
AI & Personalization

Guy Yalif performed his CH2019 keynote "AI & Personalization Demystified" and spoke about how we can use AI and machine learning, what it is (and isn't) good for in marketing and personalization. After his session, he and Niels Reimer joined me in the café to talk about this! (oorspronkelijk gepubliceerd op https://www.cro.cafe/)

Make All of Your Data Actionable With Ringside Data
Make All of Your Data Actionable With Ringside Data

Do you struggle to get enough actionable data from Google Analytics? Then you need to listen to Russell McAthy, CEO and co-founder of Ringside Data. Key takeaways include: Why Russell set up Ringside Data, the companies with the biggest data problem, how Ringside outperforms Google Analytics, if you’re investing in data analytic tools, you need people with specific skill sets, the biggest pain points Ringside is trying to solve, the benefit of Ringside for eCommerce businesses, and what attribution is. (originally published on https://www.coreofcommerce.com/)