Hightouch lets you pull data stored in your Google BigQuery warehouse and push it to downstream destinations.
Connecting Hightouch to BigQuery requires some setup in both platforms. It's recommended to set up a service account with the correct permissions in BigQuery before configuring the connection in Hightouch.
You need to allowlist Hightouch's IP addresses to let our systems contact your warehouse. Reference our networking docs to determine which IPs you need to allowlist.
Setup in BigQuery has three main steps:
- Enable BigQuery for your Google Cloud project
- Create a service account
- Grant the Hightouch service account access to your project
- Login to the Google Developers Console.
- Configure the Cloud Platform:
- Copy your Project ID for later use.
- Find the location of your BigQuery dataset or sets. You can find this by querying the
INFORMATION_SCHEMA.SCHEMATAview or by visiting the Google Cloud web console and clicking on a BigQuery dataset in the Explorer panel. You need both Project ID and Data location when connecting Hightouch to BigQuery.
Make sure billing is enabled on your project, otherwise Hightouch can't write into the cluster.
To create a service account, follow the setup instructions in our Google Cloud Provider (GCP) documentation.
By default, your GCP service account doesn't have permission to read data from BigQuery. You can set up your service account to have full access to your project using a predefined role. Otherwise, you can create a custom role and provide limited access according to a user-specified list of permissions.
You can grant full access by assigning the
bigquery.dataViewer roles to your service account.
You can do this in the Google Cloud web console or by running these snippets in the Cloud Shell.
Grant permission to read metadata and list tables:
gcloud projects add-iam-policy-binding <YOUR_PROJECT_NAME> \ --member serviceAccount:<YOUR_SERVICE_ACCOUNT> \ --role roles/bigquery.user
Grant permission to read data from tables and views:
gcloud projects add-iam-policy-binding <YOUR_PROJECT_NAME> \ --member serviceAccount:<YOUR_SERVICE_ACCOUNT> \ --role roles/bigquery.dataViewer
If you don't want to grant full access to your BigQuery service account, you can opt to grant limited access instead.
You can do this by assigning the
bigquery.dataViewer role only to the specific datasets, tables, or views you want to use in Hightouch.
Since you are assigning the
bigquery.dataViewer role only to specific resources, you need to assign the
bigquery.user role and grant the
bigquery.tables.get permission at the project level.
For this, you can create a custom role in the Google Cloud web console based on an existing predefined role (
bigquery.user), which you can name
When setting up the custom role, click Add permissions to add the
bigquery.tables.get permission to this custom role.
gcloud projects add-iam-policy-binding <YOUR_PROJECT_NAME> \ --member serviceAccount:<YOUR_SERVICE_ACCOUNT> \ --role roles/custom.bigquery.user
You can then decide which datasets, tables, or views your GCP service account has access to by granting access to a resource in the Google Cloud web console.
For every resource you would like to use in Hightouch, select your BigQuery service account as the Principal and the
bigquery.dataViewer role as the Role.
Hightouch lists all tables in your BigQuery project when creating a model using the table selector.
However, Hightouch can only query data from tables assigned the
Tables that weren't assigned this role return an error if you attempt to query them.
To get started, go to the Sources overview page and click the Add source button. Select BigQuery and follow the steps below.
Enter the Project ID for the project you enabled the BigQuery API for and the Dataset location.
For optimal performance, Hightouch tracks incremental changes in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.
The standard engine requires read-only access to BigQuery. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to BigQuery.
The Lightning engine requires read and write access to BigQuery. The engine stores previously synced data in a separate schema in BigQuery managed by Hightouch. In other words, the engine uses BigQuery to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.
If you select the standard engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the standard engine without recreating BigQuery as a source.
To learn more, including migration steps and tips, check out the Lightning sync engine docs.
The Lightning sync engine requires granting write access to your data warehouse, which makes its setup more involved than the standard sync engine. However, it is more performant and reliable than the standard engine. This makes it the ideal choice to guarantee faster syncs, especially with large data models. It also supports more features, such as Warehouse Sync Logs, Match Booster, and Identity Resolution.
|Criteria||Standard sync engine||Lightning sync engine|
|Ideal for large data models (over 100 thousand rows)||No||Yes|
|Resilience to sync interruptions||Normal||High|
|Extra features||None||Warehouse Sync Logs, Match Booster, Identity Resolution|
|Ease of setup||Simpler||More involved|
|Location of change data capture||Hightouch infrastructure||BigQuery schemas managed by Hightouch|
|Required permissions in BigQuery||Read-only||Read and write|
|Ability to switch||You can move to the Lightning engine at any time||You can't move to the standard engine once Lightning is configured|
To enable the Lightning engine, you need to provide your service account additional permissions to create schemas read/write data.
Run the following snippet to provision the
hightouch_audit schemas, which are used for storing logs of previously synced data.
CREATE SCHEMA IF NOT EXISTS `hightouch_audit`; CREATE SCHEMA IF NOT EXISTS `hightouch_planner`; GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_planner` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>"; GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_audit` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>";
When setting up a source for the first time, Hightouch validates the following:
- Network connectivity
- BigQuery credentials
- Permission to list schemas and tables
- Permission to write to
- Permission to write to
All configurations must pass the first three, while those with the Lightning engine must pass all of them.
Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking Test again.
If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to .
Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from BigQuery.
The BigQuery source supports these modeling methods:
- writing a query in the SQL editor
- using the visual table selector
- leveraging existing dbt models
- leveraging existing Looker Looks
- leveraging existing Sigma workbooks
When syncing large amounts of data, it can take a long time for your model to reflect changes made in BigQuery. To speed up BigQuery model updates in the Hightouch UI, you can preview a model and save it. New or updated columns should then be reflected in the Hightouch UI.
You may also want to consider storing sync logs in BigQuery. Like using the Lightning sync engine versus the standard one, this feature lets you use BigQuery instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the warehouse sync logs docs to learn more.
You must enable the Lightning sync engine to store sync logs in your warehouse.
If you encounter an error or question not listed below and need assistance, don't hesitate to . We're here to help.
An example of the error message is
No matching signature for operator != for argument types: DATE, TIMESTAMP.
This error means that your model definition is using the named operator, for example,
!= to compare two incompatible types, for example, a
date to a
To resolve the issue, ensure your model columns are properly typed or revise your model query.
An example of the error message is
Unable to process number due to [big.js] Imprecise conversion: [big.js] Imprecise conversion
This error occurs when one or more values have a very high precision, for example
123.12345678910. Hightouch attempts to convert the value to a
To resolve the issue, ensure the values in your model columns have a lower precision. Additionally, if your value doesn't need to be a
number datatype, casting the value to a
string datatype can also resolve this issue.