Google BigQuery

Overview

Hightouch lets you pull data stored in your Google BigQuery warehouse and push it to downstream destinations.

Connecting Hightouch to BigQuery requires some setup in both platforms. It's recommended to set up a service account with the correct permissions in BigQuery before configuring the connection in Hightouch.

BigQuery credential setup

Setup in BigQuery has three main steps:

1. Enable BigQuery for your Google Cloud project
2. Create a service account
3. Grant the Hightouch service account access to your project

Create a project and enable BigQuery API

1. Login to the Google Developers Console.
2. Configure the Cloud Platform:
   • If you don't have a project already, create one.
   • Once you have a project, enable the BigQuery API for it.
3. Copy your Project ID for later use.
4. Find the location of your BigQuery dataset or sets. You can find this by querying the INFORMATION_SCHEMA.SCHEMATA view or by visiting the Google Cloud web console and clicking on a BigQuery dataset in the Explorer panel. You need both Project ID and Data location when connecting Hightouch to BigQuery.

Create a service account

To create a service account, follow the setup instructions in our Google Cloud Provider (GCP) documentation.

Grant access

By default, your GCP service account doesn't have permission to read data from BigQuery. You can grant this access by assigning the bigquery.user and bigquery.dataViewer roles to your service account.

You can do this in the Google Cloud web console or by running these snippets in the Cloud Shell.

Grant permission to read metadata and list tables

gcloud projects add-iam-policy-binding <YOUR_PROJECT_NAME> \
--member serviceAccount:<YOUR_SERVICE_ACCOUNT> \
--role roles/bigquery.user

Grant permission to read data from tables and views

gcloud projects add-iam-policy-binding <YOUR_PROJECT_NAME> \
--member serviceAccount:<YOUR_SERVICE_ACCOUNT> \
--role roles/bigquery.dataViewer

Connection configuration

To get started, go to the Sources overview page and click the Add source button. Select BigQuery and follow the steps below.

Configure your service account

Select the GCP credentials you previously created or click Create new. To learn more about these credentials, see the Google Cloud Provider (GCP) documentation.

Configure your source

Enter the Project ID for the project you enabled the BigQuery API for and the Dataset location.

Choose your sync engine

For optimal performance, Hightouch tracks incremental changes in your data model—such as added, changed, or removed rows—and only syncs those records. You can choose between two different sync engines for this work.

The standard engine requires read-only access to BigQuery. Hightouch executes a query in your database, reads all query results, and then determines incremental changes using Hightouch's infrastructure. This engine is easier to set up since it requires read—not write—access to BigQuery.

The Lightning engine requires read and write access to BigQuery. The engine stores previously synced data in a separate schema in BigQuery managed by Hightouch. In other words, the engine uses BigQuery to track incremental changes to your data rather than performing these calculations in Hightouch. Therefore, these computations are completed more quickly.

Hightouch recommends using the Lightning sync engine when syncing more than one million rows of data.

If you select the standard engine, you can switch to the Lightning engine later. Once you've configured the Lightning engine, you can't move back to the standard engine without recreating BigQuery as a source.

To learn more, including migration steps and tips, check out the Lightning sync engine docs.

Standard versus Lightning engine comparison

Lightning engine permissions

To enable the Lightning engine, you need to provide your service account additional permissions to create schemas read/write data.

Run the following snippet to provision the hightouch_planner and hightouch_audit schemas, which are used for storing logs of previously synced data.

CREATE SCHEMA IF NOT EXISTS `hightouch_audit`;
CREATE SCHEMA IF NOT EXISTS `hightouch_planner`;
GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_planner` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>";
GRANT `roles/bigquery.dataViewer`, `roles/bigquery.dataEditor` ON SCHEMA `hightouch_audit` TO "serviceAccount:<YOUR_SERVICE_ACCOUNT>";

Test your connection

Next steps

Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from BigQuery.

The BigQuery source supports these modeling methods:

• writing a query in the SQL editor
• using the visual table selector
• leveraging existing dbt models
• leveraging existing Looker Looks
• leveraging existing Sigma workbooks

Tips and troubleshooting

To date, our customers haven't experienced any errors while using this source. If you run into any issues, please don't hesitate to reach out. We're here to help.