Another Approach to Supplementing Universal Holdout Experiments

What is a Shared Control Group and Why Do I Need One?

In a standard A/B/n test, a percentage of traffic is allocated to a “Control Group,” which receives the default experience (in a medical experiment: the placebo; in website testing: the current experience).

A “Shared Control Group” involves conducting various tests using the same control group for comparison. So, in the medical example, this would mean testing different medications against the same group ONLY receiving placebos (not as many qualified participants needed, overall!).

However, the difficulty of involving a Shared Control Group comes in deciding which other experiences to exempt your audience from (i.e., In a medical experiment, is this audience eligible to take allergy medicine? When testing a website, is this audience eligible for exposure to other experiments in unrelated site areas?).

This Sounds a Lot Like a Universal Holdout…

…Because it can be. 

Both the Shared Control Group and Universal Holdout are processes for withholding a percentage of eligible traffic from ALL designated exposures (marketing campaigns, site experiences, etc.) for a period. For example, this means a percentage of your audience is predetermined NOT to receive email campaigns, site testing opportunities, or app pushes during a specified timeframe.  

A Universal Holdout and Shared Control Group work best when set up to complement each other. The Universal Holdout might include withholding all traffic, whereas the Shared Control Group could include a secondary protective layer that withholds traffic groups separately for different platforms, or with a more frequent refresh.

Why Do I Need a Shared Control Group?

Velocity

• A Shared Control Group requires less “control” traffic than multiple separate control groups, resulting in tests reaching statistical confidence quicker, and providing opportunities to run more tests.

QA

• A more frequently refreshed Universal Holdout provides clients more fluidity when evaluating performance. The refresh timing can be customized, and the Shared Control Groups can be specific to a particular platform, site area, or business priority (i.e., customer retention). Whereas, a Universal Holdout is typically refreshed bi-annually or annually, and across all relevant marketing channels.

Additional Considerations

Refresh Frequency

Your organization will need to agree upon a standard cadence for the refreshing of this shared control group to determine an optimal deployment setup.  Refreshing in accordance with sprint cycles could be the best approach for many organizations. In other situations, or organizations, refresh intervals could depend on support around specific marketing insights.

Audience Cleanliness

The nature of a Universal Holdout depends on audience member identification. If a visitor is not recognized on their second visit to the site, they could become eligible for campaigns that previously deemed them ineligible (i.e., a member of the Universal Holdout may be unrecognized, and incorrectly granted exposure to active site tests).

In the case of Customer Data Platforms (CDPs), such as Tealium’s AudienceStream, piece together Customer IDs to identify otherwise unrecognizable customers by clearing cookies, cross-device visitation, or lack of authentication.

We still ALSO Recommend Universal Holdout!

The primary goal of a Universal Holdout is to prove the effectiveness of an established marketing/testing program. By evaluating the success of the exposed (or eligible for exposure) audience against the withheld audience, a client can determine if the program—as a whole—contributes to the success of primary KPIs.

Use the option to include a Shared Control group to your advantage by implementing it in a way that works best for your testing program, audience sizes, and platform capabilities. Your experimentation initiatives might perform even better if you include a separate Universal Holdout!

Want to learn more?

Hannah Alexander and Luke Youtsey will go even deeper on this topic in their September 28, 2022 presentation at Experimentation Nation: The Conference.

For even more information about Evolytics’ experimentation capabilities, get in touch!