Question: what are the different possible outcomes of A/B test experiments and what do you do depending on each of those scenarios?
- Findings support hypothesisAction: implement winner and/or re-test (more subtle variants or on other sites/platforms) to validate hypothesis.
- Findings refute hypothesisAction: create other variants to re-test hypothesis or abandon hypothesis.
- No change found (but sample size requirement is reached)Action: Check to see if we can think of more extreme variants to test the hypothesis.
- Experiment error, didn’t reach sample size etc. etc.Action: Redo the test with new measures in pace to prevent the errors.
It’s important to regularly share these findings (at least A/B/C) with everyone in the team/company to get feedback and to increase everyone’s knowledge of our customers behaviour.
What do you think? Are there more outcome options for your CRO experiments?