Prusak.com

Serving up my thoughts since 2004

The #1 Split Testing Mistake

I was reading a simple case study today.
They were testing two different versions of a banner that was advertising a webinar.
One of the banners had an image of the presenter, while the other did not.
The banner without the image of the presenter won (by over 50%).

One of the comments was something along the lines of:
I guess this audience prefers banners without an image of a person.

*sigh*

If you don’t immediately realize the mistake the commenter made, don’t feel bad. It’s a very common mistake.

Beyond the fact that a specific banner (which did have have an image of the presenter) won over a different specific banner (which did not have an image of the presenter) you really can’t be sure of anything.

The loosing banner might have won with:

  • An image of a different person
  • A different image of the same person
  • The same image of the same person in a different position or size on the banner.
  • The same image of the same person in the same position and size but with different elements on the banner changed.

The point is:

Don’t jump to generalized conclusions based on the outcome of a specific experiment.

-Ophir

1 Comment

  1. Excellent simple point about the fundemental nature of testing…we all need to come back to basic questions over and over.

    thanks

    Dan Shannon
    web
    marketing for film

Leave a Reply

Your email address will not be published.

*

© 2024 Prusak.com

Theme by Anders NorenUp ↑