These are my A/B testing tool lessons

I’ve used A/B software for over 2 years. I’ve tried:

  • Optimizely
  • Google Optimize
  • VWO

I’m presuming you know what A/B testing is, so stay with me.


Optimizely is great. I used the free version for a little over 2 years with minimal support. My past is a front-end web developer so I was able to successfully code into the UI HTML and CSS which worked and was usable as a test.

I was able to target certain types of customers – where had they come from? An email? An ad from Google? I was able to use the same language the customer had seen from whatever referral they had come from.  I’d use the same title for example, to keep it consistent, relevant and seamless.

I was able to set traffic allocation to whatever I wanted. This was useful when I set a test live. I wanted to make sure it was working as expected, so I changed the new test to 100% so I was able to see it was quickly as possible. Once happy, I changed it to 50/50 or 33/33/33.

Whilst working with a client, we were in talks with Optimizely to purchase the paid-version. In the new version came a new and improved UI. Sorry I can’t share screenshots with you but I haven’t got access! It looked good – very personalised to the exact details you want and completely customisable.

The results page was (I’m not sure what it’s like now) very clear and easily digestable. I was able to screenshot the results and put it in a powerpoint document to stakeholders – that’s how visually pleasing and clear it was.

One of the things which was available in the free version but I never made use of (I actually never knew it was available) was to track your tests throughout a journey. This is really important. For example, if a test on page 1 was successful and it helped get 20% extra customers to page 2 – that’s great! – but how did it help customers get from page 2 to page 3? If page 3 is your money page, that’s surely something to track.

That’s where Google Optimize comes in.

Google Optimize

With Google Optimize you can link to your Google Analytics (GA) account. Providing you’ve set goals in GA, any test can be measured against these goals. Perfect for tracking how a test is doing throughout a journey where the page isn’t part of your actual test!

Oh yeah, plus it’s free! You can have live a maximum of 5 tests at once.

The UI is good – it’s not as nice to work with as Optimizely. You can’t pause or edit a test once it’s live which is an absolute pain. Yes, tests shouldn’t need changing once live – I’m with you there – but often, plans change and they do. So ner.

Similarly to Optimizely you can target different sets of customers based on where they’ve been. You can also base your test on a mobile, tablet or desktop. Customers using different devices are totally different so you should absolutely alter your designs per device. One persona could be, a customer is using their mobile on a train so they are using 3G. Therefore, on a mobile, you could leave off the larger hero image and make your copy snappier. Does this work? Has conversion increased? I just gave you a cracking A/B testing idea there, you’re welcome!

The results page isn’t clear – I’m talking about the one in Optimize rather than the goals in GA. If you’re new to A/B testing or even like me – doing it for a few years – it’s still quite confusing to 1. look at 2. explain to stakeholders what’s going on. There’s no way I could screenshot the graph and send to stakeholders like what I could with Optimizely. It’s ugly and unclear.


I’ve used VWO in a trial and let me say, it was beautiful to work with. You define your pages, goals, journeys and most importantly – metrics – in the system which you then use for each test.

You can get heatmaps for your tests – so a heatmap dedicated to version A and version B, even if it’s the same URL. Ooooh, I know. This is cool. If memory serves me correctly, this is also available when watching recordings – you can see recordings of customers who saw A vs B, to compare.

Tailoring your designs based on the segmentation of audiences was the most advanced I’d ever seen.

Final issues

With all of these, I was unable to figure out how to fix the dreaded flicker. Yes there’s a flicker – when version A loads and is replaced with version B, you can see it change. This ain’t good my friend. This will affect your tests and needs to be fixed. I tried adding code in the head, doing Javascript things which I never want to think about again, talking to support more times than they’d like to remember plus lots of Google searching to try to find a solution – to no avail.

A/B testing seems to be taking off, so hopefully this is something that can be fixed (please, without any Javascript ‘workarounds’ aka horror!)



Leave a Reply

Your e-mail address will not be published. Required fields are marked *