Prusak.com

Someone recently asked on the GWO form:

Why would 10% expected improvement take a lot longer than a 20%?
I would think it is the other way around.

The truth is – the smaller the expected improvement, the longer it takes.
Or, in other words, the bigger the difference in conversion between two variations, the shorter time it will take to see the difference.

Here is an analogy that will hopefully provide an intuitive asnwer.

Lets say I want to test two basketball teams against each other, seeing who’s the better team.
Team A currently wins 20% of the time.

Scenario 1
I want to test if team B is a 100% improvement over team A, which would mean team B wins 40% of the time.
The question is, how many games do they need to play to be 95% sure than group B is indeed better. (95% is the confidence level GWO uses).

Without going into the math, if team B indeed is twice as good as team A, I should know fairly quickly.

Big differences are obvious in a short time.

On the other hand …

Scenario 2
I want to test if team B is just 10% better than team A, which would mean team B wins 22% of the time.
How many games do they need to play to be 95% sure than group B is indeed better.

Again, without going into the math, if team B indeed is just 10% better than team A, they’ll need to play quite a lot of games to be 95% certain that they are indeed 10% better.

Small differences take longer to notice than big differences.

Just a quick post wishing everyone a Happy New Year.

My first new year’s resolution is to start posting once a week. See you all next week!

– Ophir

Google just announced an official ActionScript 3 library for tracking flash events in Google Analytics.

You can read the whole post here:
http://analytics.blogspot.com/2008/11/want-to-track-adobe-flash-now-you-can.html

I’ve implemented Google Analytics tracking from withing Flash in the past, but it was simply having the Flash call some JavaScript code on the page.

This solution is much more elegant and opens up many new possibilities.

This is also very exciting for Google Website Optimizer users, as it should now be very easy to use an event within a Flash as the conversion event.

I just posted a solution for capturing the initial referrer on landing pages that use JavaScript to redirect to a different page, and already I’ve found another (and maybe better) solution.

I just read a post by ShoreTel who suggests using initData without the trackPageview call. This will indeed set the GA cookies with the correct initial referrer value and not mess up pageview numbers.

On the landing page that does the JavaScript redirect, copy the original Google Analytics tracking code (something like this):

and change it to this:

Both solutions should work fine!

A couple of weeks ago I posted about a known issue using A/B experiments in Google Website Optimizer.
http://www.prusak.com/google-website-optimizer-and-referer-data/

Basically, when conducting an A/B experiment in Website Optimizer, the original referrer data is lost because of the JavaScript redirect to the test versions. This means that the traffic source will be wrong (it will say direct instead of the true traffic source).

This issue isn’t just a problem with website optimizer. Any landing page that uses JavaScript to redirect will loose the original referrer data. For example if your site provides easy to remember URLs like www.mysite.com/freeshipping and then uses JavaScript to redirect users to a different URL.

I wrote in my previous blog post that I wished Google Analytics provided an easy way to over ride the referrer value (and not use the document.referrer value).

It seems I spoke to soon.

Google Analytics DOES have a way to overwrite the referrer data.

It just doesn’t seem to be documented anywhere 🙂

I planned on writing my own hack to override the referrer value in Google Analytics. I was doing some reverse engineering of the ga.js code and found this interesting tidbit:

a._setReferrerOverride=function(b){a.yb=b}

I’m still running some tests, but based on my initial results, the _setReferrerOverride function provides a simple way to manually set the referrer value.

So now, we can actually capture the original referrer even when the visitor is redirected. Here’s how:
1 – Store the original referrer data in a cookie named realreferrer right before we redirect
2 – On the test pages, look for the realreferrer cookie. If it exists use it to over ride the referrer value in Google Analytics and then delete the cookie (so it won’t mess up page views that weren’t a result of a redirect from the original version).

Here’s the code I’m currently using in my tests.
Add this on the original page before the redirect (the GWO control script):

On the test page, add this code before any GA or GWO calls:

Then change your GA code from this:

to this:

Please let me know if this works for you !

Update:
I just found another (and possibly better) solution. See:
http://www.prusak.com/google-ananlytics-initial-referrer-update/

Update #2 [Oct 2010]:
It's been over two years since my initial posting and this issue still exists. A few people have asked about an async version so here it is.

All of my custom JS stays the same.

Here is the change for the async code:

Before

_gaq.push(['_trackPageview']);

After

if (realreferrer.length > 0) {
    _gaq.push(['_setReferrerOverride',realreferrer]);
}
_gaq.push(['_trackPageview']);

Please note my blog messes up the single quotes for code snippets when copying / pasting.

- Ophir

When running an A/B experiment Google Website Optimizer appends any parameters from the original page when redirecting to one of the test pages.

For example, the original page is index.php and the test page is index2.php.
If a user clicks on index.php?name=fred they will be redirected to index2.php?name=fred

So what happens if the test page URL already has a query parameter in it?

For example, the original page is index.php and the test page URL is index2.php?section=b
What will happen if the user clicks on index.php?section=d ?

About seven months ago I had a client with this exact scenario so I tested it.
The user was directed from index.php?section=d to index2.php?section=b&section=d
I found a work-around and didn’t think about it after that.

A few weeks ago I found out that Google was indeed aware of this “issue” and fixed the GWO code 🙂
Now GWO will merge the parameters from the two URLs, not just append them. The parameters defined during setup for the test page URL will get precedence over the parameters from the URL the user clicked on.

For example, the original page is index.php and the test page URL is index2.php?section=b
What will now happen if the user clicks on index.php?section=d&cat=22 ?

They’ll be redirected to index.php?section=b&cat=22

This doesn’t effect 99.99% of all A/B tests, but if it does effect you, Google’s fix is a life saver 🙂

I’ve been doing a lot of reading on the Google Website Optimizer help pages:
http://www.google.com/support/websiteoptimizer/

For the most part, the help pages are kept up to date, but every now and then something falls between the cracks and I get the feeling the page does not reflect the latest updates.

I was just thinking how every Google help page should really have a time stamp, letting readers know when the page was first created and when it was last updated.

Something like:
This page was created on Sep 20th, 2007 and last updated Oct 24th, 2008.

Any thoughts?

Google Website Optimizer A/B experiments use JavaScript to redirect the visitor to a test pages.

When a visitor is redirected with JavaScript, you loose the original HTTP referer data (the page that did the redirecting now becomes the HTTP referer)

Here’s an example:
Lets say the original page is page.html and we have a test page page_1.html

Before the test, page.html got 1,000 visitors a day coming from Google searches (organic traffic).

So we have Google -> page.html

Now that the test is running you’ll have 500 visitors to page.html:
Google -> page.html

and 500 to page_1.html
Google -> page.html (JavaScript redirect) -> page_1.html

If the web analytics code is after the redirect on page.html (which is what Google tells you to do) then data based on the HTTP referer will be incorrect (traffic source, keywords, etc).

If you REALLY want to capture the HTTP referer related data you can put the analytics code before the redirect. You’ll now be able to capture the original HTTP referer data, but you’ve also just inflated the page views.

I’ve solved this issue for some in-house code that uses the referer data by simply storing the HTTP referer value in a cookie right before the redirect, and then using the cookie value on the test page.

Unfortunately Google Analytics doesn’t do this.

Here’s a feature request for the Google Analytics team:

Add a way pass in the HTTP refer data manually.

I’m sure Website Optimizer isn’t the only tool using JavaScript redirects from a landing page.

“What is the best page to start optimizing?”
This question comes up quite often in conversion rate optimization.

It’s an excellent question that really deserves a detailed (and lengthy) answer, but today I want to tackle a specific misconception about choosing a page that I’ve seen many people make (and even some books).

The misconception is that “It’s always better to test pages that have more traffic“.

Here’s a simple example with some numbers to explain my point.

Lets say you have an ecommerce site with these stats:

• 10,000 visitors a month (which land on the homepage)
• 3,000 visitors a month who reach your product detail page (the one with the add to cart button)
• 600 visitors a month buy (convert)

This means you have a 6% conversion rate for homepage visitors and a 20% conversion rate for visitors who reach your product detail page.

All things being equal, do you start testing on your homepage or the product page?
Lets run the numbers and see.

Assuming you run a simple test with just two variations (the control/original and a single test version) and that you’ll be able to get a 15% (relative) increase in conversions from your changes, here are the results:

The control group
5,000 Homepage visitors
1,500 product page visitors
300 conversions

Testing the Homepage
Your change to the Homepage causes 15% more visitors to reach the product page which leads to 15% more conversions:
5,000 Homepage visitors
1,500 * 1.15 = 1,725 product page visitors
300 * 1.15 = 345 conversions

Using my simple Conversion Confidence Calculator it’s showing a 93% confidence level (screen shot below).

Testing the product page
Your change to the product page causes 15% more visitors to “add to cart” which leads to 15% more conversions:
5,000 Homepage visitors
1,500 product page visitors
300 * 1.15 = 345 conversions

Using my simple Conversion Confidence Calculator it’s now showing a 95% confidence level (screen shot below).

In terms of statistical significance, I’ll actually get results a bit faster testing the product page which has far less traffic than the homepage.

So, am I getting something wrong or is this a general misconception?

– Ophir