Experiments

Run controlled experiments directly in Customer Studio by creating randomized groups to test and measure the true impact of your marketing campaigns.

Learning Objectives

After reading this article, you will be able to:

• Understand when and why to use audience experiments
• Create A/B or multivariate experiment groups within a saved audience
• Sync each group to separate treatments, campaigns, or destinations
• Measure campaign effectiveness using Experiments in Campaign Intelligence
• Use stratified sampling to ensure balanced test and control groups
• Access holdout group logs for detailed post-campaign reporting in your warehouse

Overview

Audience Experiments enable you to create randomized audience groups for accurate, controlled testing of campaign strategies. By assigning audience members to distinct test groups, you can compare outcomes between treatments, channels, or creatives and understand what’s truly driving performance.

What you can measure

• Incrementality: Did your campaign actually cause the desired outcome?
• A/B/n testing: Compare creatives, messages, or channels.
• Personalization impact: Determine whether tailored experiences outperform generic ones.
• Channel effectiveness: Identify which channels drive more downstream conversions.

When to use experiments

Use audience experiments when you want to move beyond assumptions like:

• Correlation: “Purchases increased—but was it because of the campaign?”
• Attribution: “A user converted after seeing my ad—but would they have converted anyway?”

Running a controlled experiment isolates the impact of your campaign by comparing outcomes between randomized groups.

How audience experiments work

Example use cases

Email campaign holdout

• 50/50 experiment
• Group A receives promotional email
• Group B is held out
• Compare conversions or revenue per user

Ad creative A/B test

• Groups A and B synced to separate ad sets
• Compare CTR, CPC, or downstream conversion

Personalization test

• Group A receives personalized product recommendations
• Group B receives a generic message
• Compare engagement and purchases

Onboarding lifecycle experiment

• Test different onboarding flows
• Measure feature adoption or retention after 14 days

Channel comparison

• Group A receives email
• Group B receives paid ads
• Compare downstream LTV or sign-up quality

Setup steps

1. Create experiment groups

1. Open your audience in Customer Studio.
2. Navigate to the Experiments tab.
3. Toggle Enable experiment groups.
4. Under Experiment groups, configure:
   • Number of groups (e.g., 2 for A/B tests, 3+ for multivariate)
   • Percentage distribution (e.g., 50/50)
   • Group names (e.g., Holdout, Treatment A)
5. Optionally add more groups for A/B/n testing.
6. For each group, configure syncs to destinations.
7. Click View results to begin analysis.

2. Measure results

Once your audience has at least one sync involving experiment groups, Hightouch automatically generates a corresponding Experiment.

Clicking View results opens the Experiment overview.

You can also access Experiments anytime under:

Intelligence → Experiments`

Use Experiments to:

• Compare lift across groups
• View statistical significance
• Analyze results using primary or secondary metrics
• Normalize results per member or against a baseline

Learn how to configure and interpret Experiments →

Stratified Sampling

Simple randomization may not provide balanced groups when your audience contains diverse characteristics (e.g., geography, loyalty tier). Stratified sampling ensures proportional representation across experiment groups based on selected variables.

How to enable stratified sampling

1. Open your audience and go to the Experiments tab.
2. Under Advanced configuration, toggle Stratified sampling.
3. Select one or more columns to stratify by (e.g., loyalty_tier).
4. Save your changes.

Holdout group logs

Holdout group logs track which rows were excluded from a sync as part of a holdout (e.g., a 20% control group). This enables advanced SQL or BI analysis.

How to enable holdout group logs

1. Contact Hightouch to enable the Holdout group logs feature flag. Requirements:
   • Lightning sync engine
   • A destination with write access to your warehouse
2. In your workspace:
   • Go to Integrations → Sources
   • Select a source and open the Sync Logs tab
   • Enable Audience holdout group logs

When enabled, Hightouch logs excluded rows in the hightouch_audit.audience_holdout table.

Table definition