Possible experiment outcomes

Written by Guido Jansen in
September 2017

Question: what are the different possible outcomes of A/B test experiments and what do you do depending on each of those scenarios?

Answer:

  1. Findings support hypothesis - Action: implement winner and/or re-test (more subtle variants or on other sites/platforms) to validate hypothesis.
  2. Findings refute hypothesis - Action: create other variants to re-test hypothesis or abandon hypothesis.
  3. No change found (but sample size requirement is reached) - Action: Check to see if we can think of more extreme variants to test the hypothesis.
  4. Experiment error, didn’t reach sample size etc. etc. - Action: Redo the test with new measures in place to prevent the errors.

It’s important to regularly share these findings (at least options 1,2 and 3) with everyone in the team/company to get feedback and to increase everyone’s knowledge of our customers behaviour.

What do you think? Are there more outcome options for your CRO experiments?

Recent posts
First episode of CRO.CAFÉ is live!
First episode of CRO.CAFÉ is live!

As announced in November I would be starting a new (Dutch) podcast series this year about CRO. Today the first episode launched!

CRO.CAFÉ Episode 0 (Teaser)
CRO.CAFÉ Episode 0 (Teaser)

In 2019 I will start a new Dutch podcast series about CRO (Conversion Rate Optimization). I want to get a better insight into the Dutch CRO field, get an idea of how everyone is working, what the struggles are and use this platform to exchange knowledge and experience. I'll be interviewing people working in/with CRO at agencies, tool vendors, merchants and other end-users.