Ask the author about a free A/B Test Read Template

Components of an A/B Test Read

It’s important to not only test, but also to report out your learnings. The learnings are arguably the best part of experimentation. So it is imperative to prioritize what you will learn and where your library of information will live. 

Reporting doesn’t require a crazy sophisticated method, so if you’re just starting out and you’re working with tools like Atlassian, Google Drive, Smartsheets, or OneNote, one thing remains the same, there must be consistency and uniformity. The simplest way to do this is by creating the sections listed below and organizing your reads as a post-reporting checklist. Eventually, these post-test reads should serve as a basis for your pre-analysis for potential tests or ideas. 

Lastly, and arguably the most important element of your test read, is setting expectations on when you will provide updates or readouts. Allow time for your data to marinate and choose a cadence that best suits the behavior of your customers. So for any large sample sizes like one million visits, where you are getting 20% of those visits per day, you can comfortably provide a daily update. On the opposite end, for a sample size of one million visits, where you are getting 5% of those visits per day you may want to spread out your updates to a weekly cadence. Just remember, all of this is about consistency! 

A/B Test Brief Section

This section can be filled out with your stakeholders or a central testing team. The purpose is to align on the test goal. Note, this section of the read should not change. 

1. Hypothesis (If, then, because…..)
   1. Example: If the hero banner is changed to reflect the user’s previously browsed categories, then the user will be more likely to click the hero banner. 
      1. Primary KPI: Banner CTR
2. Customer Solve 
   1. How is this test going to help the customer?
   2. How is this test going to help the business?
   3. What impact on the business process can you expect if this test is successful?
3. Primary KPIs
   1. Click-through rate (CTR)
   2. Site conversion (Orders/Visits)
   3. Average order value (AOV)
4. Secondary KPIs (Do not harm)
   1. Bounce rate
   2. Revenue per visit
   3. Orders/checkout rate
   4. Conversion rate (orders/visits)
5. Behavioral KPIs (Tells the story)
   1. Page Depth + Bounce rate + Exit Page
   2. Add to cart rate + Cart/checkout rate + Cart abandonment rate
6. Goal Lift
   1. Index View 
      1. For a 10% index lift your index is 110 
         1. For a conversion rate of 21% at a goal lift of 110 = 21*1.10 =  23.1%
      2. For a 5% lift your index is 105. 
         1. For a conversion rate of 21% at a goal lift of 105 = 21*1.05 =  22.05%
   2. Percent View 
      1. For 10%  percent lift your 
         1. For a conversion rate of 21% at a goal lift of 10% = (21*.1)+21= 23.1%
      2. For a 5% percent lift your
         1. For a conversion rate of 21% at a goal lift of 105 = (21*.05)+21= 22.05%
7. Control and recipe images
   1. Retrieve these from your designers 
8. Deployment dates
   1. Include exact deployment date and time so that the data analysis can be focused to the appropriate window
9. Sample data needed
   1. Based on the sample size per recipe, and number of offers, you can use your daily visit count to get an idea of how many days the test will need to run. 
10. Pre-Analysis Link (optional addition)

A/B Test Setup and Validations Section

The test technical setup will impact the integrity of your test, so let’s not make this the reason the test fails. Include everything important to getting your test launched and what issues you may foresee in the future. These types of forecasts can be labeled in your read as a section to address disclaimers. This section should be validated by the person who is launching and performing QA for the test. 

1. How does a customer qualify for the test?
   1. Is it at the site level?
   2. Does a user have to click through via a banner first?
   3. Does the user have to belong to a certain shopping tier? 
2. What other tests are in flight at the same time?
   1. Are the primary KPIs the same?
   2. Can the logic of the two tests allow them to run concurrently?
3. Did page analytics complete QA requirements?
   1. How is the event tracked? 
   2. What is the tracking plan for the test from beginning to end?
4. Who will deploy the test?
   1. Will it be your testing team?
   2. Will it be the developers?
   3. Does it require a release?
5. Links to any development/UX work completed for the test (optional but encouraged)

Results for Primary and Secondary KPI Section Analysis

Apply the success/failure parameters set by your stakeholders for your primary KPIs. Report out the hard numbers plain and simple with an indication of statistical significance next to them. This section can be filled out by any of the analysts as it’s just a report-out of the hard numbers. 

1. Control/Recipe traffic and conversion funnel
2. Primary KPI confidence met 95%, indicate yes/no
3. Results: flat/positive/negative/trending
4. Secondary KPIs confidence met , inconclusive, trending , positive/negative statistically significant

Insights Section

The insights section is reserved for the big picture. Was this test in its entirety successful? What segments observed performed the best and worst? This too can be filled out by your analysts or it can be reserved for individuals on the team with statistical/data science backgrounds. This section is where your leaders will focus most, so the lens of “what should the business do” must be at the forefront of this analysis. 

1. What did this test tell you about the customer behavior?
2. Did the KPI chosen accurately represent the story of the data?
3. Next step suggestions
4. Behavioral KPIs (Tells the story)
5. What is the probability of the behaviors observed?

Deep Analysis Section

Not all tests require a deeper analysis outside of the insights the business requires, but sometimes, especially for net new products, there are multiple questions tied to one test. In this case, the reads should include the deep analysis along the way. 

1. Forecasting/annualization of revenue
2. Correlation analysis 
3. Test strategy decision tree changes 
4. Pre/post product analysis
5. Site performance analysis

With this general A/B Test Read template, you’ll have a streamlined way to focus on what’s most important rather than chasing down unpredictable stakeholder whims or re-aligning on expectations. 

Get your FREE A/B Test Read template!

Just fill out this short form and we’ll email your template within 48 hours.