Annualizing Upside gives a big Number. It is not uncommon to see analytics team get a 2% upside in revenue per visit and extrapolating it to an entire year resulting in millions of $ impact that is not real.
Especially if the test is on the basket page or ecomm page - You are claiming that you are going to increase your company's online revenue or e-commerce revenue by almost 2% because everyone that's going to buy is going to add to basket or checkout
This is very dangerous due to the following reasons:
* Many times A/B test might give the upside due to sheer chance. An A/A test might perform even better :)
Analysts ignored the effect of 'noisy data'. The tools unfortunately also ignore this.
* The impact is so low - only 2%. Test is either run 'too short' or 'too long'. Too long means n is large and the standard error formula that is used to extrapolate sample results to the population has n in the denominator. 'Too short' - means you got a confidence level - but simply don't go by these tests. I will be posting subsequently a note as to why & how many of the current testing methods are inadequate and dated.
* There is no sound reasoning on the reason behind upside.
It is like 'Customers felt better' - WHat? I didn't :)
'Customers did not get distracted' - cool. that was not a factor for me ....
Unless there is sound reasoning, the impact $ must not be taken at face value
* Annualizing Upside is dangerous. There might be downstream or upstream changes during the course of the year that leave these changes redundant.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment