An Introduction to A/B Testing (Part 2) - Designing an email A/B test

An Introduction to A/B Testing (Part 2) - Designing an email A/B Test

This is a continuation of our article An Introduction to A/B Testing (Part 1).

Designing an email A/B test

Email sent to prospects who have chosen to give your organization information is a unique medium in modern marketing in the following ways:

• Your prospects have already expressed interest in your brand and have voluntarily shared information with you.
  • name
  • email address
  • phone number
  • product interest
  • ⋯
• Your systems also know something about the prospects
  • origin source/medium
  • keywords
  • previous website & media consumption
  • ⋯
• When you send email, your email automation system knows
  • if the email was received (potentially considered)
  • if (and when) the email was opened and by whom
  • if the CTA was clicked
  • the user on multiple devices (e.g. home computer, work computer, mobile phone, tablet, TV, …)
• You do not have to pay to put the media in front of a prospect
• The action rates (opens and clicks on CTAs) are much higher than other mediums
• Prospects have trackable histories and futures
  • allowing you to do lead scoring
• You can do follow-up experiments on prospects

Picking which email to test

Let’s start at the very beginning
A very good place to start
– “Do-Re-Mi” from “The Sound of Music”,
Rodgers & Hammerstein

When you are starting out, start at the beginning with the first email (the “thank you” email, which should be sent immediately after sign-up) and with the first scheduled campaign email (which is probably being sent when the prospect is not actively engaging with your content).

Early emails are where you win or lose the eyeballs of a fresh prospect with respect to future marketing. If the prospect does not find the emails useful, (s)he is less likely to open future emails. The improvements you make early in an email relationship can lift the results of successive emails and increase your sales/wins. (You can and should measure how each test group does on subsequent identical emails.)

Assigning prospects to groups

All of the email systems we have encountered assign identifiers to prospects.

An easy, recommended scheme for a 50/50$50/50$ A/B$A/B$ test on new prospects is to assign the even-number IDs to one group and the odd-numbered IDs to the other group.

When you begin testing later emails, we recommend what are called “paired tests” in which you build several customer “segments” based on certain characteristics and then divide each segment into two groups (although the even/odd approach often works here, we tend to export the IDs in each segment, randomly assign groups and then import the group assignments). Paired tests reduce variance and often allow you to use smaller sample sizes to obtain statistically significant results.

Deciding what is a conversion

You have many choices: open, click, form completion, in-person visit, RPQ, price quote, sale, referral, …

Your email automation system is collecting opens and clicks on every email. If your system is website-integrated and things are properly set up, then you are probably able to measure customer

What you pick to use as a conversion depends on

• business goals
• the ability of your systems to automatically tie the conversion back to the email
• conversion time

Our advice is to

1. Measure as a conversion the logical next step after encountering the media (an open or a click)
2. AVOID using anything that is too long-term as a conversion (you should always measure the performance; it is important to move at “internet speed”, however — is there a milestone before the long-term event that can be used as an indicator?)
3. Once you have decided what to test, measure (and report) the logical second step the prospect should take sua sponte — if you drive up open rates, but clicks on your CTA don’t increase, did you really improve?
4. Measure and report the performance of the next email for each group — improving the open rate on a bad/low-utility email can do more harm than good — it is important to catch this early.

In the next article in this series, we will discuss "Measuring Results".