Perils of A/B Testing & Multi-variate Testing

Annualizing Upside gives a big Number. It is not uncommon to see analytics team get a 2% upside in revenue per visit and extrapolating it to an entire year resulting in millions of $ impact that is not real.
Especially if the test is on the basket page or ecomm page - You are claiming that you are going to increase your company's online revenue or e-commerce revenue by almost 2% because everyone that's going to buy is going to add to basket or checkout

This is very dangerous due to the following reasons:
* Many times A/B test might give the upside due to sheer chance. An A/A test might perform even better :)
Analysts ignored the effect of 'noisy data'. The tools unfortunately also ignore this.
* The impact is so low - only 2%. Test is either run 'too short' or 'too long'. Too long means n is large and the standard error formula that is used to extrapolate sample results to the population has n in the denominator. 'Too short' - means you got a confidence level - but simply don't go by these tests. I will be posting subsequently a note as to why & how many of the current testing methods are inadequate and dated.

* There is no sound reasoning on the reason behind upside.
It is like 'Customers felt better' - WHat? I didn't :)
'Customers did not get distracted' - cool. that was not a factor for me ....
Unless there is sound reasoning, the impact $ must not be taken at face value

* Annualizing Upside is dangerous. There might be downstream or upstream changes during the course of the year that leave these changes redundant.

Social Media & communities - What are the right metrics?

Here are my thoughts on three questions on Social media & Communities in a web analytics group:

1. How to attribute ROI to Facebook and other Social Media?

Answer:
#1 Revenue from Facebook apps with same session attribution (meaning he came from Facebook apps and purchased on the same visit)
#2 Revenue from Facebook apps with multi-session attribution (meaning he could have come from Facebook property 30 or 60 days ago and bought today - give credit to the facebook app since he made our site familiar with him)
#3: Engagement Metrics: Remember this is an important component on CRM
Are folks writing positively about your company or not?
Are folks talking about your organization?

A big risk would be if folks stopped talking or discussing about you - implication - knowledge that your products and brand are no longer "hot"
Many negative posts - your newly launched product is not doing well

2. Is it right to target people this way? Privacy??

Gmail shows up advertisements based on keywords in email - So also do others. The site would usually have a clause asking the user if he wants to view advertisements on registration - so not an issue

3. Is the future bright?

Absolutely yes!!
Think of this - you come from a social media & communities site.
Where should I land you on my e-commerce site?
Should I land you on a good deal or on a product detail page of a standard product?
It is a mining & testing_targeting problem domain