Ran a test - will the results be same in production?

Anu on the WA group has a question - SHe ran a test that had two recipes A and B. She found that B is better with a conversion advantage of 10%.
She wants to know if this means that when she puts it into production if the 10% advantage will remain.

I think there is a need for me to post an article on the basics of beta testing and multi-variate testing - on basic statistics like sample, population etc.

Here is my answer to that
-------------------------------------------------------------------------------------



Well, there cannot be a guarantee - I will tell you WHY.

Say you did a beta/MVT test - It works like this -
Since you had 2 versions - control flow (A) and test flow (B) - what happened is that the tool randomly directed incoming traffic to either A or B. So an incoming visit could randomly be in A or B - however at the aggregate level, the tool tried to keep the visit count in A and B same.
Seeing this behavior continually over a large period of time, it achieved statistical significance that in your case B is better

So if all parameters are the same - namely the site design, the incoming traffic quality and the customer behavior, then the conversion advantage of 10% should remain.

however you will see variation if the site design changes or the incoming traffic mix changed or you got a different set of customers. When you test a month form now, you might even find that recipe A is overperforming.
So continually TEST

Business metrics for Tracking a non e-commerce site -

I have tracked a non e-commerce business site before but that was tracking a site that had learning content - the objective being to eventually drive the visitor who comes to learn about the site to make a purchase.

In such a non e-commerce case key metrics should be:

1. Repeat Metrics: % of Repeat visits
2. Duration Metrics: Visit Duration/Page Duration
3. Engagement Metrics: Number of clicks
4. Drive to purchase metrics: %age of visitors that end up purchasing on the site
5. Abandonment Metrics: Bounce,Leakage

Some nuggets from experience
* Look for 5-abandonment metrics. It is the most important metric. If the bounce is high for a non e-commerce site, then probably no point in advertising to get traffic in there - IMPROVE THE CONTENT
* Engagement metrics: Number of clicks, visit/page view duration are equally important - Less engagement means that the site is not doing its job
* Also look at the metrics by individual page for the top 10 % of the pages. You might find a few lemons there
* If your site is really good, over a period of time, the natural traffic should increase to it -(word of mouth at work :) )