Saturday, May 2, 2009

Ran a test - will the results be same in production?

Anu on the WA group has a question - SHe ran a test that had two recipes A and B. She found that B is better with a conversion advantage of 10%.
She wants to know if this means that when she puts it into production if the 10% advantage will remain.

I think there is a need for me to post an article on the basics of beta testing and multi-variate testing - on basic statistics like sample, population etc.

Here is my answer to that
-------------------------------------------------------------------------------------



Well, there cannot be a guarantee - I will tell you WHY.

Say you did a beta/MVT test - It works like this -
Since you had 2 versions - control flow (A) and test flow (B) - what happened is that the tool randomly directed incoming traffic to either A or B. So an incoming visit could randomly be in A or B - however at the aggregate level, the tool tried to keep the visit count in A and B same.
Seeing this behavior continually over a large period of time, it achieved statistical significance that in your case B is better

So if all parameters are the same - namely the site design, the incoming traffic quality and the customer behavior, then the conversion advantage of 10% should remain.

however you will see variation if the site design changes or the incoming traffic mix changed or you got a different set of customers. When you test a month form now, you might even find that recipe A is overperforming.
So continually TEST

Friday, May 1, 2009

Business metrics for Tracking a non e-commerce site -

I have tracked a non e-commerce business site before but that was tracking a site that had learning content - the objective being to eventually drive the visitor who comes to learn about the site to make a purchase.

In such a non e-commerce case key metrics should be:

1. Repeat Metrics: % of Repeat visits
2. Duration Metrics: Visit Duration/Page Duration
3. Engagement Metrics: Number of clicks
4. Drive to purchase metrics: %age of visitors that end up purchasing on the site
5. Abandonment Metrics: Bounce,Leakage

Some nuggets from experience
* Look for 5-abandonment metrics. It is the most important metric. If the bounce is high for a non e-commerce site, then probably no point in advertising to get traffic in there - IMPROVE THE CONTENT
* Engagement metrics: Number of clicks, visit/page view duration are equally important - Less engagement means that the site is not doing its job
* Also look at the metrics by individual page for the top 10 % of the pages. You might find a few lemons there
* If your site is really good, over a period of time, the natural traffic should increase to it -(word of mouth at work :) )
Nikki on the WA group has an interesting question - She wants to know the difference between 'organic clicks' and 'organic referring clicks'

I am first of all against using these kind of metrics at a site level without clearly understanding what they imply.

Since Nikki uses HBX (my favorite WA tool), here is my best approach towards solving her issue:
----------------------------------------------------------------------------------
In a visit, the referrer tag is populated by the WA tools based on the referrer= information supplied by the browser.

E.g:
If you go to www.google.com and search for "buying cheap cameras" and it throws up an
ebay link that you click, the browser will add the tag
referrer = google when you visit ebay

Similarly for other sites that lead you to ebay.

All of these are referral traffic - It is organic if it is natural (not paid for - e.g: Natural search traffic), it is inorganic if it is paid for (e.g: Paid Search traffic)

Now it may so happen that in the same visit, you come to ebay from google, from yahoo, from microsoft search as well as from other sites, who would you give the credit?

The credit should go to the first person that got you to ebay.

This is populated in the referrer dimension in VS/HBX
Use that as a dimension and get the visits/clicks by the referrer. Do not use it on the metrics side - and sum up the values of visits/clicks to get it for the site.
----------------------------------------------------------------------------------

Natural Search vs. Paid Search

Googling has become the synonym for search.

Traditional marketing theory has it that a purchase stems from a need (maslow's hierarchy for the B-schoolers :) ) ----> need for information ---> search for information ----> evaluate alternatives ----> buy decision
No wonder that in the online world, Searching or Googling is the easiest way to get the information - and hence one of the major sources of visits/traffic to a website

Google, Yahoo, Microsoft have used this crucial link in the buying process to monetize/get revenue in the form of advertisement links. Anytime somebody clicks on the sponsored links on the right or wherever they appear on the web page, they make money.

E.g: The diagram is a snapshot of my search on google.com for 'buying books online'. Google has intelligently figured out that I live in India (my airtel account??) and shown me results in the India context.

If I click on the "blue" box, then Google makes no money, while Google makes money if i click on the "red" box

Clicking somewhere in the 'blue' box constitutes 'natural search' while clicking in the 'red' box constitutes paid search


This presents an interesting question:
If the same page, e.g: ebay.in, were to be displayed in both the 'red' box as well as the 'blue' box what would happen.

For all visitors that click on ebay in the red box, ebay would pay money to google, adding to its advertising spend.
For all visitors that click on ebay in the blue box, ebay need not pay any money to Google

So from an online retailer's perspective, increasing the visits from natural search would greatly contribute to optimize the advertising spend.

Tracking repeat cart visitors

In the WA yahoogroup, Smith had an interesting problem.

His website has setup a mechanism to retain items in cart for a period of 30 days and whenever the person who added these to carts returns to the site, he sees the previously placed item in the cart. Smith wants to know how he can track the effectiveness of this feature.

Here is my solution to his problem:

------------------------------------------------------------------------------------

There are two ways to do this:

WAY 1: WITHOUT SETTING UP ADDITIONAL TRACKING CODE, ONE CAN GET A VERY GOOD APPROXIMATION.

(1) Depending on the WA tool that you are using, you need to have a way to IDENTIFY cart additions
e.g:
Look at the URL when someone adds to cart - On most sites it is like http://......./addtocart.html/aspx.

{Get the page name and look at number of visits to that - This will be the metric cart additions}

(2) create a VISITOR/COOKIE LEVEL SEGMENT for all the folks that added to cart (using the page key above)

Above will help us track all the cookies that have added to cart.

(3) create a derived metric based on purchase_visits.
purchase_visits {Visit_seq_num > 1} - This is possible in Discover OnPremise.

Combine 2 and 3 in a workspace/report - You have your metric.

WAY 2: WITH ADDITIONAL SITE TAGGING
Your website already can identify returning visitors and show them what is in the cart. For these have a tagging mechanism by way of code in the add to cart page that sends specific tagging information to the WA tool.
e.g: this could be through a URL parameter called return_to_cart

Next use your custom report formats in Yahoo Analytics, Google Analytics, Omniture to see the visits and purchase_visits for this parameter return_to_cart

That should give you the answer - There is nothing in the world of tracking WA that is not trackable :)

Thanks
Kiran
rkirana@gmail.com
Web Analytics Consultant
http://webanalyticsnuggets.blogspot.com/

- On Thu, 30/4/09, tlsmith1260 wrote:


From: tlsmith1260
Subject: [webanalytics] Tracking conversion in shopping cart with item retention
To: webanalytics@yahoogroups.com
Date: Thursday, 30 April, 2009, 10:31 PM

------------------------------------------------------------------------------------

So Mr SMith, you are on the right track.

Customization and personalization form the backbone of Web 2.0
You are on the right track!!

How to find the source of my 404s

'404 - Page Not found'

This is a common error that we have all come across. We see this on our browser screen when the page we are trying to visit does not exist on the server.

for the technically inclined, it is the status code returned by the HTTP server when the page requested does not exist on the server.

On the web analytics group, Sue had a question: She is tracking this website and sees that large number of 404s come from the most visited page on her website.

Here is my proposed solution in Omniture Discover
Follow the below logic. It should be same in any WA tool

Let us call the "most popular page that is redirecting to the 404s" as X
and the 404 page as Y

1. Create a segment for all the folks who visited X and Y
2. For this segment look at the traffic source split

E.g: In Discover
* create a segment by dragging the visit container to RHS
* Click on edit
* add the 2 pages X and Y
* save

Then go to the traffic source report and pull the segment as a column and choose the date range.

You should be good!!

Thanks
Kiran



>
> --- On Wed, 4/29/09, Sue wrote:
>
> From: Sue
> Subject: [webanalytics] probably a very dumb ? but I need help finding source of 404
> To: webanalytics@ yahoogroups. com
> Date: Wednesday, April 29, 2009, 9:00 AM
>
> Community, please help me...
> I use Google Analytics and Webtrends version 8.0. WebTrends looks at my log files.
>
> I have read this article: http://googlewebmas tercentral. blogspot. com/2008/ 10/webmaster- tools-shows- crawl-error. html but it is no help to me. Google analytics shows no errors.
>
> but my page that 404's are redirected to is my number one visited page on my site. Where are these people coming from? Can I get this info from WebTrends?
>
> I have been so frustrated, I have actually clicked every link I can find on my web site to try to find the culprit.
>
> HELP!
> Thanks!
> Sue
>

Tuesday, April 28, 2009

Site Tagging based Web Analytics tools

Another category of WA tools rely on site tagging.

(1) they create an account for you on their server
e.g: Google Analytics, Yahoo Analytics

(2) You decide on the pages you want tracking metrics. Typically you will do so for your entire website.
Then tracking code is added to the pages that you want tracked.
E.g: The ever so familiar javascript tags that are added when Omniture is implemented
Remember .js :)
The WA tool may make use of cookies on your machine that already existed or create new ones

(3) Each time a visit/click happens on your website the .js code gets executed, contacts the WA server and sends this information.
So over a period of time, the server builds a repository of information that mirrors the server log file. The server may dynamically insert it in a minable format into the data mart residing at the WA server

(4) the WA tool lets you login on their server with the account you created first and view a set of reports for your website.
Each time you ask for a report, they are querying their data mart

Monday, April 27, 2009

Server Log file mining based Web Analytics tools

TOOLS BASED ON LOG FILES:

When you click www.ivarta.com on your browser, what you are doing is sending a request to the server at ivarta to send you the ivarta.com home page.
When you continue to browse on the site visiting various links all your visit data is logged into a "Log file"

A log file would be common for all visits on the server and would contain details like:
(a) Cookie
(b) Visit IDentifier
(c) sequence of click
(d) page URL Clicked
(e) timestamp of the click
(f) other parameters that are sent to the server
e.g: When you click on the login button at yahoo.com, your login_id and password are sent as parameters to the yahoo.com server
page tags are also parameters passed
These are called tagging parameters that help the server to gather more information on the visit/visitor

These log tools present a mountain of information that can be used to extract patters and understand about visitor behavior on the site.
These analytics tools mine the server log files (through a process called extract-transform-load ETL) to create a mini data mart. This can then be combined with company specific data sources to complete the picture

Example of tool based on server log files is the erstwhile HBX then erstwhile Visual Sciences and now Omniture Discover OnPremise

Sunday, April 26, 2009

Web Analytics Tool Architectures: Server Logs vs. Site Tagging

Web Analytics tools come in two broad flavours:

(1) Ones that rely on server log files
(2) Ones that rely on site tagging

Let us go through their workings in detail in the coming posts

Tuesday, April 14, 2009

metrics: Building blocks of web analytics

Let us start with the definition of the building blocks of web analytics:
Visit/Session
User/Cookie
Request/Click

Imagine a world with only 3 machines: McA, McB and McC
and that folks named A, B and C are logged on to those machines

Consider the following sequence of browsing by the person on McA

DateTime Machine Page
11th Jan 10 AM McA Home_page
11th Jan 10:20 AM McA Category_page
11th Jan 10:30 AM mcA Product_details_page
11th Jan 10:45 AM McA add_to_cart_page
11th Jan 11 AM McA payment_page
11th Jan 11 AM mcA thankyou_page

Every page visited corresponds to a page_view/request/click
So above there were a total of 1+1+1+1+1+1 = 6 clicks/requests/page_views

so a page_view/click/request is defined as a "request for a page"

Time between the clicks is as follows
1 & 2: 20 minutes
2 & 3: 10 minutes
3 & 4: 15 minutes
4 & 5: 15 minutes
5 & 6: 0 minutes

All clicks with no idle time of 30 minutes between them form part of a

visit. So in the above sequence there was only 1 visit

So in our above example there was ONE visit and SIX clicks

Since it was from the same machine (assuming the user did not clear the

cookies), it is a single user.

Now let us change the sequence



When we visit a website, the website stores a cookie on our machine - This

is to identify us uniquely the next time we visit the site. That is how

Amazon identifies visitors on their repeat visit and shows them web pages

with products/categories viewed on earlier visit - customizing the

experience

Typical Ecomm Sales Funnel


The sales funnel

For hypothetical purposes, let us imagine a website: www.abc.com
with only 5 pages:
www.abc.com Called Home_Page
www.abc.com/cameras called Category_page
www.abc.com/cameras/camera1 called Product_detail_page1
www.abc.com/cameras/camera2 called product_detail page2
www.abc.com/addtocart called Cart_page
www.abc.com/payment called payment_page
www.abc.com/thankyou called thankyou_page



As you can see there is a purchase funnel where people can start off at any
of the pages prior to thankyou_page and complete a purchase

Tuesday, March 31, 2009

Key Retail Metrics: Finding parallels in etail store (italics)

Web Analytics is a relatively new science - and what better way to decipher the metrics than drawing a parallel with the world we are all familiar since birth - the "retail" world


For a moment imagine you are the owner of the BestBuy Store (a US retailer) in RoundRock, TX OR the owner of BigBazaar (an Indian Retailer) on the Inner Ring Road, Bangalore.

To make money and be a leader, You need to run the business efficiently - which means
you need to
* Get more folks to come to your store vis-a-vis the competitor
Prospective Customer has a choice here - he can come to your store or go to the competitor's store. You need to get him to your store
* Of the folks that come to your store, you need to maximize the percent that buy
* You want more of the customer's wallet - you want him to spend more on items at your store

The folks coming to the store maybe
* Repeaters:
Those who have come before (They present a huge data mining opportunity - Data can tell us about their buying patterns and their segments meaning we "know" them - and we can take marketing actions to drive more financial upside from these folks; this is a separate topic)
* Newbies
Those who are new
Our retail stores (BestBuy/BigBazaar) can increase the number of new prospects by advertising in local media, radio, Television, email Or letting people know through advertisements on websites
Optimizing advertising spend and ROI on the same itself is a challenge for the retail store

Whether folks coming to our store buy AND How much he spends could be a function of
* Ease of finding the product he wants on the store
e.g: Keep items likely to be bought together close to each other
Easy retail layout
* Pricing of the products on the store
* His understanding of the product (whether it satisfies his need or not - many times he may not be aware that a particular product satisfies his need the best)
* His money share planned for the day (implying his demographic 'segment' for example)
* His purchasing efficiencies

In the same particular order of italics, the key metrics/parallels in the online world are as below:

~ stands for "is related to"

* "more folks to come to your store" ~ Visits/Clicks/Users
* "maximize the percent that buy" ~ Conversion
* "more of the customer's wallet" ~ Average Order Value/Total Revenue Per Unit/Revenue Per Visit etc
* "Repeaters" ~ Users/Repeat Visitors
* "Newbies" ~ First time Visitors
* "advertising" ~ Online Demand Generation/Marcom
* "local media, radio, Television, email Or letting people know through advertisements on websites" ~ Advertising Channels/ODG Channels/Marcom Vehicles/Demand Generation Vehicles (DGV)
* "Optimizing advertising spend and ROI" ~ ODG/DGV/Marcom Analytics
* "Pricing" ~ Behavioral targeting Opportunity
* "understanding of the product", "ease of finding Product" ~ Optimal Site Layout/Optimal Site pathing

I have marked quite a few stuff in "italics" and drawn out parallels in the online world - In the subsequent posts, we will find parallels for these in the online e-tailing world - in separate posts

about the blog and me - First Post

The heading had a limit of 500 characters and so the full background here :)


About the Blog:

This is an independent blog by on the world of online analytics/web analytics - concepts, techniques, tools, ideas, practitioner interviews and answering user questions.


About the Web Analytics Blog Nuggets Organization

This blog will be organized around the following labels:

  • theory made simple: will talk about concepts, metrics and terminologies
  • tools: will talk about the features and usage of the web analytics packages in the market - both tag based (like Omniture SiteCatalyst, Google Analytics, Yahoo analytics) as well as server log based (Visual Sciences/Omniture Discover OnPremise)
  • Ideas: Will be about thoughts/frameworks/prototypes that can deliver 'real wins'
  • Practitioner corner: Will be about interviews with practitioners of web analytics
  • User: where we will take up and analyze problems faced in by readers of the blog in their web analytics ventures
This blog is a medium to share learnings and learn new things


About Me:

I'm a practicing web analytics consultant at a large US e-tailer/computer maker with hands-on experience in almost all buckets of web analytics - testing and behavioral targeting, business analysis of e-tailing sites, search marketing, support site improvements, visitor segmentation, pathing analytics and data mining.

I have a proven record of converting web analytics insights into dollars ($$):

:)