Google Content Experiments – A good or bad feature?Google Analytics specific, Metrics understanding September 24th, 2013
Quote: [Google Analytics] “Content Experiments sucks and I will never use it for any of my clients….run away“
The above snippet came from a post by Michael Whitaker (smart thinker, worth following) who asked for feedback on comments made at the Imagine 2013 conference earlier this year. My initial response was “hmmm – poor comments indeed. Whether you like a G product or not, to say that Google’s stats methods are unreliable, or reporting doesn’t work really is silly and lacks credibility.”
I am actually no big fan of the Google Analytics Content Experiments either, but I wish to put my views into context…
Firstly, lets put to bed any comments about reliability, or reports not working. As any Google Analytics user knows, reliability is very, very good.
So what is the issue with Content Experiments…?
A number of people have commented on Google’s use of the multi-armed bandit model that content experiments use. In a nutshell this statistical algorithm allocates more traffic to a winning test combination over time, at the expense of a losing combination. That makes sense, right – particularly if you are an e-commerce site where money is being lost if a poorly performing test page is delivered. After all, why would you want your test traffic levels to remain static when there is a clear winner?
The multi-armed bandit model is clever and unique in the industry…
However, in the early days of Content Experiments the algorithm was too aggressive. That is, an unexpected peak in conversions would cause traffic to be reallocated at such a rate that it could not be rebalanced if that condition subsequently changed. For example, a successful campaign for a specific product that also contained other products on its landing page, would skewer the conversion rates for the other products. Another example is a weather change. If you sell both umbrellas and t-shirts on a page under test and the weather turns wet for a few days, your umbrellas would be declared the winner (i.e. more sales for that period) without the algorithm waiting for the weather to change back. And there a national holidays – one off events that mess with your data, and so on…
So, aggressive bandits have been a problem…!
However, this was fixed (pacified) earlier this year. There is now also the option to turn this feature off all together – allowing you to keep the traffic allocation of test alternatives fixed. So as a methodology, Google Analytics Content Experiments are good for A/B testing (note, multi-variate experiments no longer supported).
As I have said, I am not a fan of Content Experiments. Here’s my explanation why…
In terms of setting up and administering an experiment, Google Content Experiments really are clunky and unwieldy. This is because test pages are delivered form your server. That means its a headache for a page built from multiple template components – see Figure 1 – as most commercial sites are. For example, a page is usually built of multiple include statements such as one for the top of page branding, one for the top navigation bar, the side navigation, main body content, footer etc. Hence, for a Content Experiment test, you have to go to your web development team and ask them to build a page delivery system that can process your variations.
Figure 1 – An A/B test page in Content Experiments requires a separate ‘nav2.php’ page to be displayed and a delivery system to ensure that it is called correctly.
Figure 1 illustrates a test for a change to a side navigation menu. Using Content Experiments this requires the side navigation include file to be switched (nav.php or nav2.php) before the page loads. The other components are added as normal. So, if your landing page URL is currently www.mysite.com/productabc/, your web platform needs to use separate URLs for test variations e.g. www.mysite.com/productabc/?sidemenu=1 and www.mysite.com/productabc/?sidemenu=2. And your platform needs to know what to do with the extra parameter (sidemenu).
Basically, unless you are the web dev agency/team, a request to change the entire system that delivers your web content by this method is not going to happen in any reasonable time frame (if ever)…
Here’s my rule for choosing a testing tool:
Once you have built your hypothesis and decided what you wish to change on your page(s), the creation of an A/B experiment should take under 1 hour. Choose your tool based on that estimate….
An alternative approach
In my experience, the benefit of a much easier experiment setup far outweighs any page loading delay. So Optimizely and Visual Website Optimizer have the edge. However, as you can see it also has its downside and some clients will not tolerate any delay in page loading – a fair point.
I would love to hear your thoughts on the merits or otherwise of Content Experiments.