Friday, February 19, 2010

Bridging gap between TnT and Insight

A common scenario in the life of a web analyst focussed on Testing & Targeting using Omniture tools:

Business Scenario:

1. Test run with 2 or more recipes. Test & Target shows recipe A as better over other competing recipes B, C, D......A was control
2. The new design that the business wanted to see was unsuccessful. Business wants to drill down into reasons for the underperformance of B.

Task for web analyst as a result of above:
1. Underperformance of B could have been due to a variety of reasons (first time vs. repeat, product mix, browser/visitor profile, traffic source, ....).
These cannot be answered using Test & Target - needs to use other Omniture tools like Discover or Insight.
2. Insight is the most powerful web analytics tool in the world and I presume the web analyst would prefer to use that given his company has the tool
3. However while Test & Target shows Recipe A as the winner on RPV due to certain reasons and by a certain margin; the other Omniture tools differ in the numbers from Test & Target
Now how to bridge the gap?

Bridging gap between RPV numbers of Test & Target vs. Other Omniture tools (Insight, Discover, SiteCatalyst,....)
1. Test & Target counts visits only if on the visit the user interacted with the page on which the test was being run.
Test & Target uses mboxes to show differing content between recipes. For a visit to get counted as a TnT visit, it is necessary for the mbox to fire.
Example:
So say you are running a test on the deals page.
Visitor X on visit_id x1 saw the deals page and added to cart. This visit is counted as a TnT Visit and as a SiteCatalyst visit
Visitor X on visit_id x2 went to cart where he had saved and viewed the item. This visit is not counted as a TnT visit but is counted as a SiteCatalyst visit

So the visit number as is from TnT vs. SiteCatalyst/Insight is going to match only if you use Insight to replicate the TnT behavior. Unless you replicate the TnT behavior using Insight, the numbers won't match

2. Timezone
TnT Timezones and Insight timezones are setup by your implementation consultant.
It is quite possible the Insight timezone was CST while TnT was GMT. In such a case you have to offset the times

3. Tag firing
It is possible that the TnT Tags fired while the SC ones didn't - this is a small proportion

4. Revenue attribution logic
TnT attributes revenue when the mbox on the orderConfirm page fires.

Implication:
Revenue will be counted in TnT if the visitor saw the campaign on a prior visit and then made a purchase on a new visit without going to the page.

Example:
On 12th Jan, Mr X visited deals page where an offer was being run; added to cart and saved to cart
On 13th Jan, Mr X bought $100 from saved cart without visiting the deals page
The $100 is counted as TnT revenue

In TNT the revenue attribution is for 14 days. Insight/SC might be using 30 days. So you have to replicate the TnT behavior in SC/Insight

5. First/Repeat Visit

TnT bases first/repeat visit on the cookie mbox being fired. So what is a repeat visit for SC/Insight may not be for TnT
However Insight can replicate TnT behavior

All this tells us why Insight must be used to match TnT behavior as you can create powerful segments to replicate the TnT behavior

3 comments:

  1. Great post - we always prefer to use our web analytics tools to verify test results from TNT. We take TNT and inject test recipe names/ids into an eVar and S.prop in Omniture and with other tools like Yahoo we use Custom Fields and Custom Parameters in webtrends.

    By doing so we can segment and deep dive into bounce rates, time on site, path analysis, etc... allowing us to analyze behavior changes from the test in addition to conversion differences.

    -Eric

    ReplyDelete
  2. Hi there, Got a question. You said

    "Visitor X on visit_id x1 saw the deals page and added to cart. This visit is counted as a TnT Visit and as a SiteCatalyst visit
    Visitor X on visit_id x2 went to cart where he had saved and viewed the item. This visit is not counted as a TnT visit but is counted as a SiteCatalyst visit"

    Then you said

    "On 12th Jan, Mr X visited deals page where an offer was being run; added to cart and saved to cart
    On 13th Jan, Mr X bought $100 from saved cart without visiting the deals page
    The $100 is counted as TnT revenue"

    Aren't these contradictory. While the first scenario Visit 2 is not considered as TnT Visit whereas in Second Scenario the Same Visit 2 will be included as Revenue for TnT

    Thanks

    -CBA

    ReplyDelete
  3. Yes, that is the way TnT Works.
    TnT Will count the revenue for the second visit but won't count the visit itself :)

    That is why we need to use tools like Insight to ensure the second visit is also counted

    ReplyDelete