Review of Conversion Optimization Minidegree Program (Pt. 11)

Let me ask you a million dollar question when it comes to conversion optimization…

When you should do A/B testing (also known as split testing)?

Chances are you’ve heard about it, but…

do you really know when and why you should use it?

Moreover, do you know what are the alternatives to A/B testing and whether they make sense to use them?

That’s what I think it’s a great question to begin this review.

You see, as a copywriter we’re always told that we should test test test…

Test different headlines or a lead.

That’s going to help us increase conversions. And it’s true, but I always felt the information was kinda incomplete.

That’s until I got into this week’s training from CXL Institute, which was the reason I wanted to become a CRO as well.

Since split testing is so popular in the online marketing space, I thought it was understanding all the reasons why and when to use it that was going to help me.

And it really ended up opening my eyes to see A/B testing in a different way… in a much more clear light.

So if I were to ask you why we do this testing, I’d say… optimization.

Now, testing is for validation and learning. You validate business impact through measurement… and split testing is one way to measure this.

A/B testing make changes safer for consumers and businesses alike.

But you don’t create tests because “you feel” that doing so will increase the response.

That’s not how it works.

Believing that someone can “know” what works and what doesn’t isn’t very accurate to say… If that would be the case, then why would you be doing A/B testing instead of just changing to what would make the uplift?

That’s why we test to begin with.

So the way you want to see split testing is as a way to solve problems. You do this by:

• First seeing where is the problem — you do conversion research and dive into your analytics.
• Next, you want to know what are the problems — use qualitative data to get the answer.
• Laslty, you want to know what’s the root cause of that problem — once you know what’s the issue, you can come up with ideas for a solution.

This ideas are what we call hypothesis.

So you might come up with different hypotheses and you might be wondering “How can you know what to test first?”

That leads us into what’s known as prioritization.

We need to prioritize the hypothesis for the one that’s going to help us get the highest ROI and change in behavior.

There are different frameworks and models for this like the PIE (Potential, Importance, Ease) or ICE (Impact, Cost, Effect) frameworks, but…

these are very subjective in the analysis.

That’s why you want to use a more objetive model that let’s you ask Yes/No questions to determine which hypothesis to prioritize.

Great, now you have an idea of what to test first, now what?

You need to answer 2 questions before you decide to run any test:

1. Do we have enough sample size to run this test?
   Ideally, you’re going to want to use a sample calculator that let’s you know the minimum relative improvement you’ll need/get.
2. When is the test done?

I’d like to spend some time in this second question.

From experience, I’ve run tests for clients for 2 weeks or by just setting a certain number of traffic I considered to be enough (about 500).

What a mistake… a very rookie mistake, if you ask me.

You don’t stop a test:

• when you reach 100 conversions per variation
• you hit 95% statistical significance, or…
• whenever the testing tool tells you (don’t ever trust them, and I will explain why later on).

For you to stop a test, you have to consider 3 key aspects:

A) Sample size → you need representative sample, not convenient sample

B) Multiple business cucles → you want to start and finish on Monday, and let it run ideally for 2–4 full weeks.

C) Statistical significance → in here, using sample size calculators will help you.

Either way, whenever you’re running an A/B test, you should have a P-value of 0.05 (or less), have a statistical power of 80%+, and enough sample size.

You check all of these to make sure you’re not declaring winners that are a false positive (or false negative, for that matter).

Now, it’s also great to mention that when doing split testing, you should always measure things that have an impact on your business (most commonly is conversions, but it can also be sign ups…).

Another thing that’s worth adding is whether you should create just 2 variants (control+ variant1) or add multiple variant tests (MVT).

The answer is simple.

As a rule of thumb, if your traffic is under 100k/month, don’t even consider doing MVT. This type of testing is done mainly to see how different components interact with each other.

But I get it.

In the end, you just want to see an increase in the number of conversions and grow your business… but what do you do when a test falls flat (or you don’t see huge difference)?

Well, you take it as a learning experience and update your customer theory.

Customer theory is the understanding you have about your Avatar, which you are constantly updating after each test.

So if you were testing a hypothesis that ‘using more scarcity in your website will increase sign ups to an event’ but you don’t see any difference, then you know that audience didn’t respond that well to scarcity messages at that time, to that offer.

In that way, plus with some validity and post-test analysis, you’re good to go on actually optimizing and growing your business.

This sums up pretty much the 4 courses about A/B Testing I went through this week.

If you were to ask me a week ago about how to do split testing, I wouldn’t feel confident knowing when to do it, for how long, or even why.

This courses inside the CXL — CRO Minidegree Program really changed that.

I don’t see why any business owner, copywriter and optimizer wouldn’t want to dive deep and miss the chance to become better at what they do.

Knowing this information will help me in so many ways for my future with my clients and my own offers, and you’re going to want to feel confidnet as well with this information.

Next week, I’ll be summing up the A/B testing module (by going a little bit deep) and focus on a completely different area of optimization.

Keep growing.

Ivan I.