Ran a test - will the results be same in production?

Anu on the WA group has a question - SHe ran a test that had two recipes A and B. She found that B is better with a conversion advantage of 10%.
She wants to know if this means that when she puts it into production if the 10% advantage will remain.

I think there is a need for me to post an article on the basics of beta testing and multi-variate testing - on basic statistics like sample, population etc.

Here is my answer to that
-------------------------------------------------------------------------------------



Well, there cannot be a guarantee - I will tell you WHY.

Say you did a beta/MVT test - It works like this -
Since you had 2 versions - control flow (A) and test flow (B) - what happened is that the tool randomly directed incoming traffic to either A or B. So an incoming visit could randomly be in A or B - however at the aggregate level, the tool tried to keep the visit count in A and B same.
Seeing this behavior continually over a large period of time, it achieved statistical significance that in your case B is better

So if all parameters are the same - namely the site design, the incoming traffic quality and the customer behavior, then the conversion advantage of 10% should remain.

however you will see variation if the site design changes or the incoming traffic mix changed or you got a different set of customers. When you test a month form now, you might even find that recipe A is overperforming.
So continually TEST