Quote: [Google Analytics] “Content Experiments sucks and I will never use it for any of my clients….run away

The above snippet came from a post by Michael Whitaker (smart thinker, worth following) who asked for feedback on comments made by Chris Goward at the Imagine 2013 conference. My initial response was “hmmm – poor comments indeed. Whether you like a G product or not, to say that Google’s stats methods are unreliable, or reporting doesn’t work really is silly and lacks credibility.”

I am actually no big fan of the Google Analytics Content Experiments either, but I wish to put my views into context…

Firstly, lets put to bed any comments about reliability, or reports not working. As any Google Analytics user knows, reliability is very, very good.

So what is the issue with Content Experiments…?

A number of people have commented on Google’s use of the multi-armed bandit model that content experiments use. In a nutshell this statistical algorithm allocates more traffic to a winning test combination over time, at the expense of a losing combination. That makes sense, right – particularly if you are an e-commerce site where money is being lost if a poorly performing test page is delivered. After all, why would you want your test traffic levels to remain static when there is a clear winner?

The multi-armed bandit model is clever and unique in the industry…

However, in the early days of Content Experiments the algorithm was too aggressive. That is, an unexpected peak in conversions would cause traffic to be reallocated at such a rate that it could not be rebalanced if that condition subsequently changed. For example, a successful campaign for a specific product that also contained other products on its landing page, would skewer the conversion rates for the other products. Another example is a weather change. If you sell both umbrellas and t-shirts on a page under test and the weather turns wet for a few days, your umbrellas would be declared the winner (i.e. more sales for that period) without the algorithm waiting for the weather to change back. And there a national holidays – one off  events that mess with your data, and so on…

So, aggressive bandits have historically been a problem…!

However, this was fixed (pacified) early in 2013. Also, there is now  the option to turn off this feature all together if you wish – allowing you to keep the traffic allocation of test alternatives fixed. So as a methodology, Google Analytics Content Experiments are perfectly good for A/B testing (note, multi-variate experiments no longer supported).

As I have said, I am not a fan of Content Experiments. Here’s my explanation why…

In terms of setting up and administering an experiment, Google Content Experiments really are clunky and unwieldy. This is because test pages are delivered form your server. That means its a headache for a page built from multiple template components as most commercial sites are – see Figure 1. For example, a page is usually built of multiple “include” statements such as:

  1. for the top of page branding
  2. for the top navigation bar
  3. the side navigation, main body content
  4. footer etc.

Hence, in order to conduct  a Content Experiment test, you would have to go to your web development team and ask them to build a page delivery system that can process variations of the above list i.e. swap out 1 for 1b, keep 2-3 in place; swap out 1 for 1c, keep 2-3 in place etc.

Figure 1 – An example A/B test page using Content Experiments requiring a separate ‘nav2.php’ page to be displayed and a delivery system to ensure that it is called correctly.


Figure 1 illustrates a test for a change to a side navigation menu.  Using Content Experiments this requires the side navigation include file to be switched (nav.php or nav2.php) before the page loads. The other components are added as normal. So, if your landing page URL is currently www.mysite.com/productabc/, your web platform needs to use separate URLs for test variations e.g. www.mysite.com/productabc/?sidemenu=1 and www.mysite.com/productabc/?sidemenu=2. And your platform needs to know what to do with the  extra parameter (sidemenu).

Basically, unless you are the web dev agency/team, a request to change the entire system that delivers your web content by this method is not going to happen in any reasonable time frame (if ever)…

Here’s my rule for choosing a testing tool:
Once you have built your hypothesis and decided what you wish to change on your page(s), the creation of an A/B experiment should take under 1 hour. Choose your tool based on that estimate….

An alternative approach

Tools such as Optimizely and Visual Website Optimizer work in a different way. The result is that experiments such as Figure 1 are possible and doable in a matter of minutes – without the need to change your website delivery system! Their technique is to modify the content of your page ‘on-the-fly’ by manipulating your page content with javascript and css. That means that variations can be built within their admin interface. The downside of this method, is that there is a slight delay added to your page loading. This happens while the test script figures out the combination requested and then alters the content on-the-fly as your page loads.

In my experience, the benefit of a much easier experiment setup far outweighs any page loading delay. So Optimizely and Visual Website Optimizer have the edge. However, as you can see it also has its downside and some clients will not tolerate any delay in page loading – a fair point.

I would love to hear your thoughts on the merits or otherwise of Content Experiments.