Experimenting with multivariate testing

Multivariate testing (MVT) is a great way to monetise customer insight, but it can be an extremely expensive way to generate customer insight.

Recently we’ve enjoyed some pretty stellar successes using multivariate testing platforms (like Omniture’s Test & Target) to tune online sales processes for increased yield. I remember, fondly, how when these tools first came to market a couple of years back their vendors confidently predicted the end of the world for the likes of Foolproof.

Now that everything in a digital sales process can be empirically tested in combination with everything else why would anyone need the anecdotal opinion of customers, creatives or so-called user experience experts to inform design decisions? Let the computer figure out what your site should look like. There are a couple of problems with MVT though:

1. When you can test everything, what should you test first?

In theory, MVT allows you to test anything and everything on your site, but practically speaking this takes a long time. When companies go to the time, effort and expense of implementing an MVT platform they expect results. Expectations of the early MVT campaigns are high.

The standard starting point in this situation is to look at site metrics from the sales process and look for the points at which customers are bailing out. OK, you’ve found your problem page, but there are a hundred thousand possible variants of copy, layout, field format, images. What components should we test first?

Oh well, every journey starts with the first step. Let’s see if making the ‘Buy Now’ button bigger (or blue, or flashing) does the trick. That didn’t crack it? Oh well, let’s test something else next time around.

Meanwhile, corporate patience starts to wear thin. Especially because;

2. The monkeys have to pay to use the typewriter

Apart from Google’s free (but rather limited) Optimizer tool, MVT costs quite a lot of money. The heart of the charging model is a tiny cost each time a page variant is served to your site. Multiply this by hundreds of variables across tens of pages over 12 months in a year and these tiny costs add up – it’s like buying media space on your own site.

If you think about this charging model for a couple of seconds, you can see that the more clueless clients are about where to start looking for efficiency gains, the better MVT providers like it. You want to spend six months (and thousands of pounds) figuring out what colour your ‘Buy Now!’ button should be? Go for your life! Like the proverbial monkeys locked in a room waiting to type the Complete Works of Shakespeare, you may be a long time finding perfection. However, if you’re paying a penny every time a monkey hits a key you’ll probably rethink your approach.

Here’s how we use MVT as part of a sales optimisation programme:

• Use quantitative data (sales and conversion figures, site stats etc) to understand the environment you are trying to tune
• Conduct user research to pin-point where you have pressure points or problems in your sales process – and to understand what’s causing these. (Hint: The place where the customer is having a problem isn’t always the same place where your site stats show they are dropping out of your sales pipe)
• Make changes to the site which immediately remove big, obvious problems
• Use your new understanding of customer needs and expectations to devise focused MVT tests in areas where the design decisions are more subtle. Often this comes from where user research showed you the nature of the problem, but didn’t categorically show you how to fix it

MVT is a terrific tool when it’s targeted deliberately on points in a sales process where you know customers are having difficulty, or need to be persuaded to commit to purchase. But if it’s used without clear objectives developed out of customer insight, well, someone may be making a monkey out of you.