Possible experiment outcomes

Written by Guido Jansen in
September 2017

Question: what are the different possible outcomes of A/B test experiments and what do you do depending on each of those scenarios?

Answer:

  1. Findings support hypothesis - Action: implement winner and/or re-test (more subtle variants or on other sites/platforms) to validate hypothesis.
  2. Findings refute hypothesis - Action: create other variants to re-test hypothesis or abandon hypothesis.
  3. No change found (but sample size requirement is reached) - Action: Check to see if we can think of more extreme variants to test the hypothesis.
  4. Experiment error, didn’t reach sample size etc. etc. - Action: Redo the test with new measures in place to prevent the errors.

It’s important to regularly share these findings (at least options 1,2 and 3) with everyone in the team/company to get feedback and to increase everyone’s knowledge of our customers behaviour.

What do you think? Are there more outcome options for your CRO experiments?

Recent posts
Optimization hierarchy of evidence
Optimization hierarchy of evidence

A hierarchy of evidence (or levels of evidence) is a heuristic used to rank the relative strength of results obtained from scientific research. I've created a version of this chart/pyramid applied to CRO which you can see below. It contains the options we have as optimizers and tools and methods we often use to gather data.

[EN] Datascience can do what?
[EN] Datascience can do what?

This is a bonus episode with Emily Robinson (Senior Data Scientist at Warby Parker) en Lukas Vermeer (Director of Experimentation at Booking.com). In her earlier session that day, Emily said that real progress starts when you put your work online for others to see and comment on which in this case was about Github. Someone from the audience wondered how that works out in larger companies where a manager or even a legal department might not be overly joyous about that to say the least so I asked Emily about her thoughts on that. Recorded live with audience pre-covid-19 at the Conversion Hotel conference in november 2019 on the island of Texel in The Netherlands. (oorspronkelijk gepubliceerd op https://www.cro.cafe/)