What to action on in a Page that changed

A page is very important in e-tail. It is not any other webpage. The more the visits to a page, the more important it is in the purchase path - and the higher is the impact that can be delivered.

No wonder merchandising teams are trying out different versions of the page. These are usually based on intuition & experience. It is the job of the analytics teams to work with the merchandising teams to ensure these decisions are based on data.
Unfortunately analytics role today in the world of simple web tools is just about providing visits and leakage information for the page OR running a simple A/B test (resulting in 2.5% upside which when annualized becomes a big number). The role of analytics should be much more than analyzing upside and annualizing the results of the A/B tests. Analytics should be able to get back and say what drove the upside/downside. I am presenting an approach that will help the same.

1. Starting Point: Make a list of levers that the merchandising teams have in changing the page
The levers fall into 2 broad buckets
a. Layout Changes
What changed in the layout? Some sample list could be
i. Rotating Banner instead of a static one OR position of Banner Changed
ii. New Links added, Old links removed
iii. Links rearrranged etc.

b. Feature Changes:
i. Page links to a page with a new feature

Step 2a: Create a segment to isolate the entries to the page
Step 2b: Create separate segments to isolate next clicks to pages from the page X
This is where it is imperative that you have a real web analytics tool at hand. You cannot do this in basic tools like SiteCatalyst or CoreMetrics or Discover. You need Insight/Visual Sciences

Step 3: Repeat Step 2 for the page 'before' and 'after' if you are doing an anachronistic analysis or repeat for the versions of the page being tested

Step 4: Tabulate as follows:


Cursory look reveals that for all landings on the page,
* Overall conversion of page improved
* Next clicks were fairly similar
* Bounces were little reduced
* Newly added Page E doing well
* Page A with changed features doing well

Now this can be translated as per our step 1 framework as follows:

Clearly upside in conversion is coming from the new Page E visits, from changed features on Page A & increased visits to Page A that has higher conversion. So business knows what improved

Advantages of my method.

1. Upside might not be coming from the changes to Page X that is being tested but rather because it leads to page Y with a new feature.
In such cases, my method captures the source of the changes
2. In cases where the page does not work, the merchandising & Design teams know what exactly to go and fix

If you need the raw xls for the analysis or help in how to set these things up in a web analytics tool like Insight, get in touch with me at rkirana@gmail.com