Question: what are the different possible outcomes of A/B test experiments and what do you do depending on each of those scenarios?
- Findings support hypothesis - Action: implement winner and/or re-test (more subtle variants or on other sites/platforms) to validate hypothesis.
- Findings refute hypothesis - Action: create other variants to re-test hypothesis or abandon hypothesis.
- No change found (but sample size requirement is reached) - Action: Check to see if we can think of more extreme variants to test the hypothesis.
- Experiment error, didn’t reach sample size etc. etc. - Action: Redo the test with new measures in place to prevent the errors.
It’s important to regularly share these findings (at least options 1,2 and 3) with everyone in the team/company to get feedback and to increase everyone’s knowledge of our customers behaviour.
What do you think? Are there more outcome options for your CRO experiments?