Search documentation...

K

Storage

Hightouch stores your query results in an encrypted Amazon S3 bucket. If you want more control over your data, you can host it in your own bucket or Blob Storage.

What data does Hightouch store?

Hightouch stores data at-rest for two purposes:

  1. Change data capture
  2. In-app observability and debugging

Depending on your privacy and compliance needs, you can configure Hightouch to store all data-at-rest within your Virtual Private Cloud (VPC), or in a secure, encrypted bucket.

Change data capture

To prevent making excessive API requests and send only necessary updates to your destinations, Hightouch uses a process called change data capture (CDC) or diffing.

In this process, Hightouch stores query results and execution plans after each sync run. When the next sync run occurs, Hightouch uses these diff files to determine incremental changes that should be sent downstream.

By default, Hightouch stores the diff files in a secure encrypted bucket hosted by Hightouch. Business tier accounts can use the Lightning sync engine to compute and stores diffs directly within their data warehouse.

To learn more about CDC, check out the core concepts docs.

Observability and debugging

In addition to storing previous query results, Hightouch also stores row-level log metadata including successes and failures, operations performed, and API request and response payloads. This data powers the in-app debugger and can be stored either in your VPC or Hightouch's encrypted bucket.

Hightouch-managed storage

If you're on self-service plans Hightouch stores data-at-rest in a secure, encrypted, Higtouch-managed bucket. If you require data-at-rest to live entirely in your VPC, see self-hosted storage.

Data retention

Data automatically expires from Hightouch-managed buckets after 30 days. If change data capture is done in Hightouch-managed buckets, syncs that have not run in over 30 days will require a Full Resync since Hightouch depends on diffing files to detect changes in the data model.

Self-hosted storage

Business tier) customers can configure Hightouch to store all customer data-at-rest within their own external storage bucket or blob. Hightouch integrates with these cloud storage providers:

  • Amazon S3
  • Google Cloud Storage (GCS)
  • Microsoft Azure Blob Storage

If you choose to self-host your storage, Hightouch only processes data-in-transit.

When using your own storage, Hightouch places full control over object lifecycle, security, and expiration into your hands. We will not expire objects automatically, or change your object encryption settings. Ensure that you've configured object expiration, encryption, and access control settings according to your needs.

Once you've run a sync after setting up a custom storage bucket, you can't make further changes to your storage configuration, including disabling it. This is because changing your storage configuration is disruptive to Hightouch syncs. If you need to make such a change, please .

Amazon S3

Before getting started, connect Hightouch with your Amazon Web Services account.

Create your S3 bucket

In Amazon S3, create your bucket. We recommend the name <company>-hightouch.

Make sure to:

  • Block all public access to the bucket.
  • Enable Amazon S3 key encryption (SSE-S3).
  • Disable bucket versioning.
  • Configure your bucket object lifecycle, to enhance security and cut down on costs.

Authenticate Hightouch with AWS

Hightouch supports authenticating with AWS using Cross-account roles (via STS AssumeRole), or with an Access Key ID / Secret Access Key that you provide. We strongly encourage you to use Cross-account roles, as it doesn't require Hightouch to hold any of your secrets.

To set up your Hightouch AWS credential, follow our connection instructions.

Hightouch needs the following IAM actions to store and retrieve items from your bucket:

ActionDetails
s3:GetObjectGrants permission to retrieve objects from Amazon S3
s3:PutObjectGrants permission to add an object to a bucket
s3:ListBucketGrants permission to list some or all the objects in an Amazon S3 bucket (up to 1000)

You can use the following JSON sample to create your IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Sample",
            "Effect": "Allow",
            "Action": [
              "s3:PutObject",
              "s3:GetObject",
              "s3:ListBucket"
            ],
            "Resource": [
              "arn:aws:s3:::${bucketName}/*",
              "arn:aws:s3:::${bucketName}"
            ]
        }
    ]
}

Configure your bucket in Hightouch

  1. In Hightough, on the Storage tab of the Settings page, select Amazon S3 as the Cloud provider.
  2. Select your AWS region, enter your Bucket name, and select the AWS credentials you previously set up.
  3. Click Save.

Hightouch app screenshot

Once you save your settings, your new syncs automatically start using your bucket. Run a few syncs and visit your S3 bucket to check files are saving there. Don't hesitate to if you have any questions.

Google Cloud Storage

Before getting started, connect Hightouch with your Google Cloud account.

Create a bucket

In the Google Cloud console, create a new bucket. We recommend the name <company>-hightouch-bucket. Copy the bucket name and save it for later. Configure your bucket object lifecycle, to enhance security and cut down on costs.

Authenticate with Google Cloud

Hightouch supports authenticating with GCP using Hightouch-managed service accounts, or by using a service account that you control.

To set up your Hightouch GCP credential, follow our connection instructions.

Hightouch needs the following IAM permissions to store and retrieve items from your bucket:

PermissionDetails
storage.objects.listGrants access to view objects and their metadata, excluding ACLs. Can also list the objects in a bucket.
storage.objects.createGrants permission to create, replace, and delete objects; list objects in a bucket; read object metadata when listing (excluding IAM policies); and read bucket metadata, excluding IAM policies.
storage.objects.get* Grants access to view objects and their metadata, excluding ACLs. Can also list the objects in a bucket.

Configure your bucket in Hightouch

  1. In Hightough, on the Storage tab of the Settings page, select Google Cloud Storage as the Cloud provider.
  2. Enter the Project ID and Bucket name and select the Google Cloud credentials you previously set up.
  3. Click Save.

Hightouch app screenshot

Once you save your settings, your new syncs automatically start using your bucket. Run a few syncs and visit your Google Cloud bucket to check files are saving there. Don't hesitate to if you have any questions.

Microsoft Azure Blob Storage

Before getting started, connect Hightouch with your Azure account.

Create a container

In your Azure portal, create a container. We recommend the name <company>-hightouch-container. Copy the container name and save it for later. Configure your storage lifecycle, to enhance security and cut down on costs.

Authenticate with Azure

The easiest way to grant Hightouch access is to grant the app the Storage Blob Contributor role for the container.

If you want to create a custom role with more granular permissions, Hightouch needs the following IAM permissions to store and retrieve items from your Blob Storage container:

  • "actions"."Microsoft.Storage/storageAccounts/blobServices/containers/read"
  • "dataActions"."Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action"
  • "dataActions"."Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write"

Configure your container in Hightouch

  1. In Hightough, on the Storage tab of the Settings page, select Microsoft Azure as the Cloud provider.
  2. Enter the Account name and Container name from the Azure portal (see connecting Hightouch with your Azure account), and select the Azure credentials you previously set up.
  3. Click Save.

Hightouch app screenshot

Once you save your settings, your new syncs automatically start using your container. Run a few syncs and visit your Blob Storage container to check files are saving there. Don't hesitate to if you have any questions.

Common errors

Your Azure storage configuration may fail to save for a few reasons:

  • invalid_client: Check that you've provided your client secret value, not client secret ID. See the configuration docs for more information.

Azure Portal screenshot

  • This request is not authorized to perform this operation: Check that Hightouch has adequate access to your storage account. If you've configured your storage account's Network access to be from selected virtual networks and IP addresses, be sure to add Hightouch's IP addresses to your network rules.

Azure Portal screenshot

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

On this page

What data does Hightouch store?Change data captureObservability and debuggingHightouch-managed storageData retentionSelf-hosted storageAmazon S3Google Cloud StorageMicrosoft Azure Blob Storage

Was this page helpful?