A/B Testing – When to Say When
A/B testing; also known as multivariate testing, is the testing of two versions of a web page – an A version (the control) and a B version (the variation) — with live traffic and measuring the effect each version has on your conversion rate.
We’ve been performing A/B tests since the launch of our new design, but when do we know when the testing is complete?
How to Calculate Conversion Rates in A/B Testing
The first thing we should talk about is the definition of a conversion rate. What is it and how do we measure it?
A conversion rate can be calculated by the number of people who did whatever it is defined as converting (email newsletter, made a purchase, and so on).
To calculate a Conversion Rate, you divide the above total number of conversions by the number of visitors to your site.
What Should you Test?
When performing A/B testing the sky really is the limit with the number of elements on your website that you can change and compare, but there are a few categories to start with that will give you immediate insight.
- Calls to Action
- Visual Content (images and video)
The First Thing We Tested (and are still testing)
Most businesses use their front page as their main source of telling their story, showing the benefits of their products or services and attracting visitors to take some sort of action, and that’s exactly how we use our main page.
We’ve been testing four elements…
- Our “hero” or background image
- Our Heading
- Our Subheading
- Our Call to Action Button
You can see that in the image above, our background image gives the feeling of a pathway and that our Heading and Subheadings tell our visitors what we do and who we are.
And our CTA button asks visitors to tell us how we can help them.
How We Created Variations and What We Changed
There are several ways to create content variations, but the service we’ve been testing is Optimizely. You should have a serious look at what they have to offer.
Variation One – Professional Valets
The reasoning behind this image and text were to promote the comparison of our service with that of professional valets and concierge services.
Variation Two – Happy Business Environment
In this one, we took the approach of promoting a positive and fun work environment, while still in a professional setting.
Variation Three – The Serious Entrepreneur and Organizer
Our intention was for this to speak to the solo Entrepreneur in all of us. The one who plans meticulously but may get overwhelmed with all there is to do in running a successful business.
Variation Four – Organic Business Growth
We purposefully took a non-linear approach with this variation in order to convey a message of natural growth.
Our A/B Testing Results
The results of our testing were interesting with no clear winner quite yet, but also surprising in that the variations we thought were just more just for fun, ended up being at the top of the results.
What do you think surprised us most?
You may have guessed correctly, it was the variation we code-named “blueberry”.
This variation is consistently at the top of the results (although more data is needed to call it a clear winner).
Interesting but not quite actionable yet.
When to End your A/B Testing?
After creating even more variations and learning their results, it became quite clear that not only was this really fun and a great way to learn more about our potential customers but also that testing can go on indefinitely if you let it.
That’s not necessarily a bad thing, but at some point we have to declare a winner and move on…or do we?
We think the answer is to never stop testing or in other words, never stop improving our message.
How to Continually A/B Test Without Wasting Time
In short, he advises that we increase the scale of our testing by making sweeping and more dramatic changes and testing these more often, even advising up to five tests per month.
The thinking is that if you are only making incremental changes like a simple CTA button color change or changing only one like of text, that it will take longer to get the results you need to make a decision that will move you forward.
You can read Sean’s article here.