I recently sent live an A/B test which changed the word “poor” to “bad” on a website I was working on. The result?
From 17% to 36% = +111%!!!
Here’s what I did and how I did it
Using Hotjar, I set up a poll on the application form page on a short-term loan website. The poll asked customers for their main concerns.
We decided to go with a free-text box, to allow the customer to write in anything they want. Although we got several colourful messages, we also got a lot of very useful data.
A large amount of customers wanted to know if we accept customers with bad credit. This was worded in many different ways but the gist was the same. The responders always used the same language – bad credit.
On the application page, we didn’t mention bad credit – we mentioned poor credit. A totally different word, even though meaning a similar thing, one which wasn’t resonating with our target audience.
I set up an A/B test using Optimizely which pitted poor credit against bad credit. The uplift was massive – an incredible 111% uplift in customers completing the application form & proceeding to the next page. We got 95% statistical significance in the test, so we were sure this wasn’t a fluke or luck.
One of the downsides was not being able to properly track (due to systems) the revenue uplift which happened due to this successful A/B test. An increase in customers going from one page to another doesn’t automatically signify more revenue which is something important to understand.
Using Hotjar Poll
I used a Hotjar poll to capture this data from the customer. The polls are pretty easy to set up and totally customisable to how you want them to behave and look.
As most of our customers used mobile, we set it up on mobile only. Unlike desktop, it’s difficult to know when a user has the intention of leaving the site, so we put the poll to appear under these conditions:
- after the user had been on the page for more than 2 minutes (this gave them time to complete the form before being distracted – the average time users took to complete the form was less than 2 minutes)
- after the user had scrolled at least 50%
I extracted the data and thought of the hypothesis that if we use the same language as the customer, that will resonate better with them. Also if they are scanning the page (as most users do!) they are more likely to see the words bad credit seeing as though that’s what they are looking for. Giving this reassurance and answering the customers’ question clearly made a positive impact.