S3

Transfer data at scale from your warehouse to S3

Setup

To get started, you will need an S3 bucket and AWS credentials. The "Bucket Name" should just be the name of the bucket, not a URL. The IAM user needs to have programmatic access enabled and permission to write to the S3 path you want to use.

Supported authentication methods

See the guide for configuring AWS credentials.

Syncing

Sync modes

• All - All Mode creates a CSV file with all the rows in the query results, every time the sync is run.
• Insert - Insert mode creates a CSV file with the rows that were added since the last sync.

File name

The object key field will allow you to specify the prefix and the name of the file that you want to use for your results. In the screenshot below, we've specified that we want the file to be called results.csv, with a custom prefix.

You can also timestamp variables in the filename, surrounding each with {}. We support these timestamp variables: YYYY, MM, DD, HH, mm. All dates and times will be UTC.

For example: upload/{YYYY}-{MM}-{DD}-{HH}-{mm}-result.csv.

Columns to sync

For this destination, we give you the ability to export all columns as they are represented in your model.

If you need remap the fields that you're exporting, maybe because you don't want to alter your model, you can manually map fields. Only the fields that you map will be exported in this instance. In this example, we're exporting id, email, name, and eventtype. These fields are mapped to new fields in the CSV as user_id, email, and username, and event, respectively. All other columns from your results are ignored.

CSV header

We provide you the option to include a CSV header with your exported results.

Other considerations

If a file at that path already exists at the time of a sync, Hightouch will overwrite it. To keep different versions of the same results file, you can enable versioning in your bucket, or your app can copy the data to another location.

This destination will not respect any sorting that you have in your model. It will export results file sorted by ID.

Tips and troubleshooting

Common errors

• ACCESS_DENIED: This is a permissions-related error. Ensure your Hightouch AWS credentials have read/write access to the S3 bucket specified in your destination. The specific minimal permissions required are s3:GetObject, s3:PutObject, and s3:ListBucket.