Everyone agrees that A/B testing can be a great way to make significant improvements to your online business. When it’s done right, testing can take your business to the next level. Increasing leads, signups, and improvements to your bottom line.
When done wrong it can drive it into the ground. This video is part 1 in a series of videos on testing mistakes that can kill your business.
Mistake #1. Ending a Test Too Early
This is a common mistake. Many marketers will fire up an A/B test, get impatient, declare a winner, and end the test. They’ll see that one of the variants beat the original, and pop the cork and celebrate their new “successful” test.
But there’s a massive problem when ending a test too early.
Ending tests too early will lead to false positives. What you see as a winner early in your tests may eventually turn to be a loser over the long-term. So if you push the variant because it increases conversions 10%, after a few months it may reverse and settle as a lower conversion rate than what you previously had.
Fortunately, there’s an easy way to avoid this mistake.
How to Avoid This Mistake
When running an A/B test, you need a lot of conversions to get enough data in to ensure that your conclusions are statistically significant. (This essentially means that you have enough data to avoid a false positive).
So, how many conversions do you need?
Some say you need at least 1000 conversions before you can declare a winner.
Others like to stick to a system for declaring a winner – tests need to run for at least 2 weeks, reach 95% significance, and have at least 1000 conversions.
Let the statistical significance be your guide. Most A/B testing tools already have a calculator built into their tool, but you can also use another significance tool. I have one that you can use for free:
So, when you’re running a test, don’t get too excited if one variant significantly increases conversions in the first few days. That will happen. Be patient and wait a couple weeks to see where you stand. Maybe even have a rule of not looking at your test results the first couple weeks. Whatever system you use, just be sure to never end a test before all the data is in and your significance calculator is telling you that you have enough data to declare a winner.
Mistake #2. Optimizing for Conversions at the Expense of Your Brand
A lot of marketers will go so overboard with testing that they’ll lose sight of the big picture – they’re running a business that has a brand that needs to be maintained.
When you test everything on a webpage, you’ll create a robotic website that has no flow. It may be ugly, unfunctional, and inconsistent. A button color on one page may be a different color than on a different page, all because an a/b test you ran gave you different results for different pages.
Or you’ll have the same, or multiple, messages on a page. You’ll wind up with something like this:
These two buttons sit above the fold and lead to the same URL. But they have two different message – Try Kissmetrics and Why Kissmetrics. This is ugly and confusing. As a visitor, I don’t know which button I should click, even though they both point to the same URL.
This is the result of A/B testing gone wrong – conclusions point to different messages converting the best. Then visitors see this and get confused.
Here’s how to avoid making the mistake.
How to Avoid This Mistake
Hopefully your company has a branding book. If you don’t already, create one.
The branding book provides the guidelines of all the things related to your company’s brand. It answers questions like:
- What’s our messaging?
- What do we want people to think when they think of our business?
- What colors do we use?
- What font do we use?
- Who are we speaking to?
- What’s our brand voice?
- What are our core values?
When you stick the brand, you’ll avoid the mistake of optimizing conversions at the expense of your brand. You’ll have guidelines to stick to, so you won’t wind up with multiple messages on the same page. You won’t have 5 different button colors on one page. And you’ll website will function and convert at it’s best, while sticking to your brand.
Mistake #3. Adding Unnecessary Steps
As a marketer, you want your leads and prospects to reach the aha moment as soon as possible. The quicker they can go from your homepage to discovering the value in your product, the better. The longer you make that time, the higher your odds of them losing interest, which means cold leads, for you.
Yet many times websites will unnecessary steps to their signup process. They’ll add a landing page, gimmicks to get you to signup, and email signup hacks. They’ll get caught up in the muck of having a million different campaigns going on and lose sight of what matters in the end – getting users to use the product and understand its value.
How to Avoid This Mistake
You know why Amazon’s checkout process is like 2 steps? Because they want to make it easy for you to purchase. They save all your information (credit cards, addresses, phone number, ets) so all you have to do is add the products to your cart and purchase. If you can get customers to reach the aha moment quicker, you’ll have helped out your business way more than if you add steps that increase conversions. The goal is to get that most amount of people from your homepage to your product, as quickly as possible.
Optimizing conversions is great, but keep in mind that at the end of the day what we all want is to get people to use our product. Figure out how to shorten your sales funnel to get people to use it. Put your focus on the right things that matter.
Mistake #4. Letting the HiPPO Run The Tests
The HiPPO (Highest Paid Person’s Opinion) is in every company. They often think they have the best ideas, but rarely do. They’ll come into a meeting and start declaring the tests that need to be run. And most companies and organizations will have to follow it because the HiPPO is usually the boss and calls the shots (and tests).
You’ve probably experienced this if you have ever worked in a large organization. The CEO, or one of the senior executives, will email you or your team and say that they don’t like something on the website, and want it changed. You try to explain that you’re actually getting more revenue with the way the site is now, but the HiPPO doesn’t care (or doesn’t believe you) and wants it changed. You run the test to change it, and it tanks conversions. But you have to go with it anyway because the HiPPO demanded it. Fast forward a couple months and you’ll meet with that same executive and they’ll ask you why you aren’t making progress on conversion goals.
And your frustration begins.
How to Avoid This Mistake
Make it clear up front with the HiPPO and everyone in the company what your role is. You own the conversion rate of the website. If the conversion rate drops, it’s your problem that’s 100% on you. If the conversion rate increases, that’s because you’re doing your job. The HiPPO may make the most money, but they’re paying you to increase the conversion rates. You are held responsible for that role.
Make it clear what your testing framework is. HiPPOs and coworkers will have ideas on what to test. Create a place where they can submit their ideas. All HiPPOs and coworkers should know how your testing works. You have ideas, hypothesis, a vetting process for those ideas, and, at the end of the day, you call the shots for what gets tested. You own the wins and failures.
As the person who owns conversions in your business, you have all the data of what’s worked and what hasn’t. Use the data and knowledge to your advantage.
View yourself as a goalie in soccer. The net is your website. A lot of HiPPOs are going to kick their ideas towards you. You need to defend the net and knock away any ideas that you think won’t work. But if you think an idea may work, and the hypothesis makes sense, then test it. But you own the final result.
Mistake #5. Micro-Testing the Insignificant Elements
Remember in mistake #2, we discussed how you can over-optimize your website to where it loses all brand recognition? This is one of the best ways to get there.
Marketers will often test the little things like button colors. These are small, insignificant changes that will incrementally increase conversions, but likely won’t lead to big changes overtime. You’ll make your site pretty boring and it won’t have any flow to it because you’ve optimized the insignificant things that most visitors don’t care about.
How to Avoid This Mistake
Think big, but don’t overthink the little things. Instead of asking questions like:
“Will changing this button from red to green increase my conversions?”
Ask questions like:
“Will changing the value proposition on my website to focus on the value we provide increase conversions?”
“Will putting a quiz on the homepage improve lead generation and quality?”
“Will changing the hero image from a stock image to one of our customers improve lead quality?”
Take a broad, macro overview of your optimization strategy. Ask questions like:
- What tests will give me results and ideas to improve my business? (Changing button colors won’t answer this question)
- How quickly can you get someone to the aha moment?
- How can you change the user flow and reduce friction to improve the conversion?
Once you’ve identified these broader tests, start to drill down to smaller optimizations around landing pages and their elements. Focus on improving your pricing page and landing page optimization. Try reducing options or adding product videos.
Think big picture, then decipher those big ideas into tests that will actually move the needle. Don’t get caught up in the minutia of button colors. That’s not focusing on the details, that’s focusing on the insignificant elements.
Mistake #6. Not Focusing on the Right Conversion Metric
We run tests to improve metrics. But too often, marketers mismatch the metrics with the test. They’ll run a homepage test by changing their headline and seeing if it will lead more people to the pricing page.
You need to know your website and what pages or elements lead to a specific metric. Your email signup form is tied to your email list. Your homepage headline is (likely) tied to the signups metric.
Here’s my homepage right now:
You can see the URL field right below. Those are signups. If I change the headline, I shouldn’t be tracking the email signups. The new headline wouldn’t move the email signups, but it would move signups. So if I was going to run a test on this headline, I’d be using signups as the metrics I’m trying to improve.
How to Avoid This Mistake
You need to understand your goals. If you’re running a test on your email signup form, the metric you’re trying to move is email signups. Don’t track your product signups with this test.
It may help to have a website map that you can use internally. All the sections or pages of your website are linked to a particular metric, and any metric you want to improve through testing should move that metric. Then you can track back to see what pages or sections you can test to move that metric.
Conclusion
A/B testing can do wonders for your business. A great marketer can double conversions and increase the bottom line with one great test.
And so can you. But only if you know what you’re doing.
Great testing requires patience, knowledge of the business and market, thinking big picture, and having a great testing framework. And most of all, it involves avoiding common mistakes that are easy to avoid. I hope the six mistakes I’ve outlined in this post will help you in your future testing endeavors.
What mistakes have you made in your a/b testing program?
Comments (7)