How to go from “4% conversion uplift” to “€636K — €854K extra profit this year”. Example calculation sheet included!
Also see my previous article: The CRO Process.
I’ve created a method to calculate the impact of A/B testing in terms
of yearly profit. And not unimportant: this has been accepted by the
whole company. This method isn’t perfect, but it gives you as a CRO
manager quite some power to prove the effect of your team.
The prerequisites to do this:
* I take 12 months so I don’t have to worry about any seasonal effects.
BI/Finance team can already provide me with the baseline prediction
(based on current numbers and including predictions on competitors,
market changes, investments etc. etc.) so I don’t have to worry about
that. Of course we also have some successful A/B tests at our disposal.
Now I can perform these steps:
is just a simple calculation only based on your A/B test data. You
calculate the revenue (or better: profit) from your original variant and
all other variants. Then you calculate the difference if you would have
had the original running 100% versus running the test with all your
works for all tests (successful or not) and tells you if you made any
loss/gain during the test itself. Of course you only implement the
winner, but if you test multiple variants it could be that the net
result during the testing period is negative. This helps you to get a
grip on the monetary cost/risk of performing A/B test.
one is actually quite simple: you take your prediction (we have it on a
day-by-day basis) and apply the changes in conversion rate and average
order value that we saw in the test.
this is where it gets tricky. 2A) will get you a very precise (and
usually quite high) number which I feel will need some corrections
(downwards) to make it realistic. I also prefer to put it in a range
instead of just one single number.
Some of the assumptions/corrections I do:
all corrections I turn the single number that comes out of this
equation into the upper limit of a range and subtract a percentage to
get to a lower limit. For a country this is 20%, when applied to the
same region it’s 40% and for the rest of the group this is 50%. All
And the result of all this? Instead of saying “we have improved conversion by 4%” I can now claim “after implementation this test gives us €636K — €854K extra profit this year”.
obvious reasons I can’t share our own internal sheet (containing our
complete budget), but I made a simplified version in Google Sheets that
should make the basic calculations clear:
Let me know if you think some things are off or if you have any improvements.
highly recommend writing down all your assumptions in your calculation
sheet. You will need to make quite some of them to make this work and
writing it down makes it tangible and easier to adjust afterwards if you
have more data to update your assumptions and the calculations based on
Some more written down assumptions I have in my sheet:
This also helps when sharing this sheet internally because: buy-in!
I didn’t build this sheet on my own: I also got our business analyst
and our financial controller involved to check the numbers, the
calculations and all assumptions. They helped improving the sheet to
match how finance calculate numbers and to get an official stamp of
course this is just a prediction and even in hindsight after 12 months
you will not be able to calculate if the changes in your numbers were a
result of your A/B test implementation.
fact means it still feels a bit weird to me to do such a prediction.
But because we made this together with BI/Finance this is currently the
best way we have to quantify the effect of our experiments. It also
provide us with a way to compare our experiments.
the best reason to do this is that it works really well. Putting a
dollar sign in your experiment reports makes a lot more people
interested in the results and helps you to increase your budget.
Recently I've seen some (often absolute) statements going around, generally in the line of "open source commerce platforms are a terrible idea". Now of course different solutions always have different pros and cons.
A hierarchy of evidence (or levels of evidence) is a heuristic used to rank the relative strength of results obtained from scientific research. I've created a version of this chart/pyramid applied to CRO which you can see below. It contains the options we have as optimizers and tools and methods we often use to gather data.