Search documentation...

K

Azure Blob Storage

Transfer data at scale from your warehouse to Azure Blob Storage

Overview

With Hightouch, you can export all the data in your warehouse to JSON, CSV, or Parquet files in any of your Azure Blob Storage containers. Whether it's for internal processes, analytics, or product data, our Azure Blob Storage integration allows you to securely and easily transfer data at web-scale.

Supported syncing

Sync TypeDescriptionSupported Sync Modes
Any data setSync data from a source to Azure Blob Storage as a JSON, CSV, or Parquet fileInsert, All, Diff
  • All - All mode creates a CSV file with all the rows in the query results, every time the sync is run.
  • Insert - Insert mode creates a CSV file with the rows that were added since the last sync.
  • Diff - Creates three CSV files, one for rows added, one for rows changed, and another for rows removed since the last sync

For more information about sync modes, refer to the sync modes docs.

Connect to Azure Blob Storage

Go to the Destinations overview page and click the Add destination button. Select Azure Blob Storage and click Continue. You can then authenticate Hightouch to Azure Blob Storage using Azure credentials. You can select credentials you previously created or click New credentials to set them up.

See the guide for configuring Azure credentials to learn how you can set up a service account and use its credentials. The IAM user whose credentials you use must have programmatic access enabled and permission to write to the Azure Blob Storage container you want to use.

Next, enter your Blob Storage Account name and Container name to complete setup.

Sync configuration

Once you've set up your Azure Blob Storage destination and have a model to pull data from, you can set up your sync configuration to begin syncing data. Go to the Syncs overview page and click the Add sync button to begin. Then, select the relevant model and the Azure Blob Storage destination you want to sync to.

Sync modes

You can select how you want the records to be updated in Azure Blob Storage:

  • All: All mode creates a file with the rows that were added since the last sync.
  • Insert: Insert mode creates a file with all the rows in the query results every time the sync is run.
  • Diff: Diff mode creates three files— one for added rows, one for changed rows, and one for rows removed since the last sync.

File format

Hightouch supports syncing JSON, CSV, and Parquet files to Azure Blob Storage.

File name

The file name field lets you specify the prefix and the name of the file that you want to use for your results.

Specifying the file to be called results.json, with a custom prefix

You can also include timestamp variables in the file name, surrounding each with {}. We support these timestamp variables: YYYY, MM, DD, HH, mm.

For example, you could enter the file name upload/{YYYY}-{MM}-{DD}-{HH}-{mm}-result.json to dynamically include the year, month, date, hour, and minute in each uploaded file.

All dates and times are UTC.

Columns to sync

You can choose to export all columns as they are represented in your model or map specific fields.

Exporting all columns in your model

You can manually map fields if you only want to export certain fields. You may also be interested in this option if you don't want to alter your model but need to rename or transform some of your fields. If you choose this option, Hightough only exports the fields you map.

The following screenshot shows how to selectively export the first_name, email, and last_name fields. Hightouch exports these fields to new fields in the file and ignores all other columns from your results.

Exporting and remapping select columns in your model

Other considerations

If a file already exists at that path you specific, Hightouch will overwrite it. To view different versions of the same file in Azure Blob Storage, you can right-click the file and select View versions.

Tips and troubleshooting

Common errors

To date, our customers haven't experienced any errors while using this destination. If you run into any issues, please don't hesitate to . We're here to help.

Live debugger

Hightouch provides complete visibility into the API calls made during each of your sync runs. We recommend reading our article on debugging tips and tricks to learn more.

Sync alerts

Hightouch can alert you of sync issues via Slack, PagerDuty, SMS, or email. For details, please visit our article on alerting.

Ready to get started?

Jump right in or a book a demo. Your first destination is always free.

Book a demoSign upBook a demo

Need help?

Our team is relentlessly focused on your success. Don't hesitate to reach out!

Feature requests?

We'd love to hear your suggestions for integrations and other features.

Last updated: Mar 16, 2023

On this page

OverviewSupported syncingConnect to Azure Blob StorageSync configurationSync modesFile formatFile nameColumns to syncOther considerationsTips and troubleshootingCommon errorsLive debuggerSync alerts

Was this page helpful?