Hightouch lets you pull data stored in your Google BigQuery warehouse and push it to downstream destinations.
Connecting Hightouch to BigQuery requires some setup in both platforms. It's recommended to set up a service account with the correct permissions in BigQuery before configuring the connection in Hightouch.
You need to allowlist Hightouch's IP addresses to let our systems contact your warehouse. Reference our networking docs to determine which IPs you need to allowlist.
Find the location of your BigQuery dataset or sets. You can find this by querying the INFORMATION_SCHEMA.SCHEMATA view or by visiting the Google Cloud web console and clicking on a BigQuery dataset in the Explorer panel. You need both Project ID and Data location when connecting Hightouch to BigQuery.
Make sure billing is enabled on your project, otherwise Hightouch can't write into the cluster.
By default, your GCP service account doesn't have permission to read data from BigQuery. You can grant this access by assigning the bigquery.user and bigquery.dataViewer roles to your service account.
You can do this in the Google Cloud web console or by running these snippets in the Cloud Shell.
For optimal performance, Hightouch tracks incremental changes in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.
The standard engine requires read-only access to BigQuery. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to BigQuery.
The Lightning engine requires read and write access to BigQuery. The engine stores previously synced data in a separate schema in BigQuery managed by Hightouch. In other words, the engine uses BigQuery to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.
Hightouch recommends using the Lightning sync engine when syncing more than one million rows of data.
If you select the standard engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the standard engine without recreating BigQuery as a source.
To enable the Lightning engine, you need to provide your service account additional permissions to create schemas read/write data.
Run the following snippet to provision the hightouch_planner and hightouch_audit schemas, which are used for storing logs of previously synced data.
CREATE SCHEMA IF NOTEXISTS `hightouch_audit`;
CREATE SCHEMA IF NOTEXISTS `hightouch_planner`;
GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_planner` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>";
GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_audit` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>";
When setting up a source for the first time, Hightouch validates the following:
Network connectivity
BigQuery credentials
Permission to list schemas and tables
Permission to write to hightouch_planner schema
Permission to write to hightouch_audit schema
All configurations must pass the first three, while those with the Lightning engine must pass all of them.
Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking Test again.
If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to .
Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from BigQuery.
The BigQuery source supports these modeling methods:
When syncing large amounts of data, it can take a long time for your model to reflect changes made in BigQuery. To speed up BigQuery model updates in the Hightouch UI, you can preview a model and save it. New or updated columns should then be reflected in the Hightouch UI.
You may also want to consider storing sync logs in BigQuery. Like using the Lightning sync engine versus the standard one, this feature lets you use BigQuery instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the warehouse sync logs docs to learn more.
To date, our customers haven't experienced any errors while using this source. If you run into any issues, please don't hesitate to . We're here to help.
Ready to get started?
Jump right in or a book a demo. Your first destination is always free.