Splits

Overview

Say you're a marketer and have just defined a new audience of users. You might have questions like, "Which channel works best here?" or "What copy or offer should I use?"

Hightouch Splits empowers you to use A/B and multivariate testing to answer these questions. With Splits, you can create randomized split groups or splits of users within your audience. You can specify:

• The number of splits you want to use
• The percentage of the total audience membership that each split should have; for example, split A gets 40% and split B gets 60%
• The destination to send each split to; for example, split A goes to Facebook, Split B goes to Google, and split C is a holdout that doesn't get sent to any destination

Using Splits, you can run experiments such as:

• Holdout Tests: Send one split to a marketing destination (email, Google, Facebook, etc.) and have the other split act as a control and receive no messaging. These tests use the holdout group feature and can prove if there was an actual lift from marketing efforts.
• Channel Tests: Send splits to different channels to determine the most effective one.
• Offer/Creative Tests: Deploy different messaging on each split to see what resonates the best.

Configure splits

To configure splits, click into an existing audience on the Audiences overview page, and select the Splits tab. From there, you can toggle Enable splits for the selected audience.

Once you've enabled splits for an audience, you need to configure the following:

1. Split Group Names and Percentages: This is a list of the split group names you want to assign to your audience and the percentage of members to assign to each group.
2. Syncs: This is the destination you want to send each split group to. Click Add sync to add more syncs, and use the dropdown under each split group to assign syncs to split groups.

3. Group Column Name (optional): This is the name of a column that Hightouch automatically creates to denote which split group a row is part of. The default column name is split_column.
4. Stratification Variables (optional): This advanced beta feature lets you select columns to use to evenly distribute your splits. For example, you may want an even distribution of "Age" across your split groups. For more information, see the stratified sampling section.

Enable holdout groups

The Holdout group acts as a control, and isn't syncable to destinations. Members of the holdout group (and all other split groups) are recorded in your data source so long as audience snapshots are enabled. To enable audience snapshots, navigate to the Sync log tab on your source's details page.

If you don't want to use the holdout group, you can turn it off by clicking the toggle or setting the distribution to 0%.

Stratified sampling

Stratified sampling lets you define subgroups based on shared characteristics such as demographic traits. Hightouch uses these characteristics to first divide the audience into subgroups or strata before assigning split groups.

Be aware of the following caveats when using stratified sampling:

• Degrading split percentage accuracy: The more stratification variables you define, the less accurate the split percentages may become. This happens because it becomes difficult to represent the appropriate percentage of users in each split group as stratification becomes more granular.
• Inconsistent assignment: Not all users are guaranteed assignment to the same split group per sync. This removes the ability to perform incremental syncs and forces users to a perform a full resync every time they sync their audience data.

FAQ

How does audience size affect split percentage accuracy?

A general rule of thumb is the larger the audience size, the more accurate the split percentages will be. This is because there are more people to achieve the desired split percentage for each group.

Other factors that can affect the accuracy of your split percentages include the number of split groups you define and their associated percentages. Generally, the lower the number of groups, the more accurate split percentages can be. The overall audience size and percentages you select for split groups also affect achieved percentage accuracy. For example, it's easier to achieve a perfect 20/40/40 split for an audience whose overall size is a multiple of 20.

How are split groups assigned?

Under the hood, Hightouch uses a hashing algorithm that uses the parent model's primary key to consistently assign split groups.

What happens if a user exits and rejoins the same audience?

Because of the consistent hashing described above, if a user exits an audience, then rejoins that same audience, they'll automatically get reassigned to the split group they were in before exiting. This assumes that the split definitions haven't changed (for example, changing the percentage sizes of the split groups).

Example:

1. User is part of Split Group A
2. User exits the audience
3. User rejoins the audience
4. User automatically gets reassigned to Split Group A