Search documentation...

K

Git Sync

Hightouch can communicate with two distinct Git repository types. The Git Sync feature interacts with a Git repository to version control your Hightouch settings. This is separate from Git repositories used to leverage dbt models.

Git Sync is a bi-directional integration between your Hightouch workspace and a Git repository. This brings all the great features of git to your Data Activation workflows: commit logs of incremental changes, the ability to roll back to a previous state, and the ability to use code to create and edit Hightouch syncs and models

When enabled, any changes made to Models or Syncs within the Hightouch app will collapse to .yaml files and commit changes to your designated repository (in GitHub, Bitbucket, or GitLab).

This also integrates the opposite direction, where any changes made directly to the .yaml files (or the creation of new .yaml files altogether) will update in the app as if they were completed in the UI.

This is an automated and continuous integration, so within about 60 seconds any changes made in Hightouch will push to Git automatically (marked as OUTBOUND), and any changes made in git will push directly back into Hightouch (marked as INBOUND).

Setup

To get started, you need a few prerequisites:

  1. Create a Git repository, if you don't already have one to store your schema files
  2. Authenticate to Git
  3. Enable Git Sync for your workspace

Authenticate to Git

If you don't already have a Git repository created, the first step is to create one. If you create an empty repository, make sure you follow the instructions to also create your first example commit, for example, a README file, so that a branch exists.

Once you have a repository, fill in your Git credentials in the extension configuration.

For GitHub, GitHub OAuth is the easiest way to get started. You'll also need to configure permissions to the repository, selecting the particular repository Hightouch should have access to.

Enable Git Sync

Next, you'll need to enable Git Sync for your workspace. Head to the Git Sync configuration on the extensions page, pick the repository and branch where you'd like changes saved to and read from.

Optionally, select a Path where the sync and models folders should be created, for example, here we're saving our settings to the Hightouch subfolder.

Hightouch will create folders for syncs and models within this subfolder, or in the root of your Git repository if no path is selected.

After clicking save, Hightouch will create the subfolder if needed and start writing your models and syncs to your repository.

From here, you're all set. Future changes in the UI will be written in the repository, and if you create any new models or syncs in Git, they will be also be created in Hightouch.

As a note, deletions from Git are not supported. If you delete a file in Git, the corresponding model or sync will still remain in Hightouch. The file won't be rewritten to Git until the next change in Hightouch, unless a full resync is triggered. We're working on supporting soft deletes to make deletions in Git more safe.

Schemas

Models have a consistent schema.

name: > the model name/slug. unique per model
source: > the name/slug of the source. the slug is visible on the source page
type: > one of: raw_sql, table, or dbt_model
dbtModel: > null or the name of the dbt model to select, written as `model.<package_name>.<model_name>`
rawSql: > null or the sql to use, for example, select * from albums
tableName: > null or the table name to use
isSchema: > used internally by Hightouch, this should always be false
primaryKey: > primary key column for this model, for example, album_id

The schema for syncs is destination dependent. For this reason, we recommend creating syncs within the UI initially, and then using the written schema to create new syncs or modify existing ones. The name of the YAML is taken as the name of the sync.

model: > model name/slug for this sync.
destination: > destination name/slug for this sync
config:
    > destination dependent configuration for this sync.
schedule: > type of schedule to use. the format here varies based on the
            schedule selected, we recommend using the UI as a starting point.
schedulePaused: > boolean, where this sync is paused, or enabled.

Validating changes

GitHub CI checks is in open beta.

You can use Hightouch's CI checks to validate changes in your Git Sync linked repository. This feature checks your pull requests to ensure they don't break any existing syncs in Hightouch. To enable this, toggle on GitHub Pull Request checks in your Git Sync configuration page.

Once enabled, Hightouch runs a CI check on any pull request made to the connected Git repository. If a breaking change is detected—for example, deleting a model used in a sync—the CI check fails. Clicking into the details of a failed CI check shows which sync the breaking change effects.

The CI check passes if the pull request doesn't affect any Hightouch syncs.

Passed CI check

Changelog

Your Git repository can be used as a changelog to track which user has created/modified which resource. It will back-populate any create/edit events from backend to your Git repository, if you configure the correct email. For examples if your user's Hightouch email matches your GitHub email, GitHub will show the correct user who created or edited the resources.

For example:

Tunneling

If you want to use a Git repository that is hosted on your on-premise server without public internet access, you can use tunneling to connect to your Git server, and Hightouch will use the tunnel to connect to your Git repository.

  1. Go to the Tunnels tab on the Settings page, and create a tunnel or reverse tunnel that connects to your Git server. Refer to the tunneling documentation for further instructions.

  2. Go to the Git sync configuration page and click Manage next to Git credentials under Configuration. Add your credential and select the correct tunnel to use.

It's recommended to choose SSH protocol and ssh_privatekey to connect to a Git repository, as basic auth may have problems forwarding the authentication header through an ssh tunnel and HTTP redirects. You can find more detail in the GitHub documentation on how to use ssh protocol to authenticate. Hightouch doesn't support ssh_privatekey with passphrases.

Next steps

From here, you have the full power of Git in your hands. Here's some potential ideas to try next:

  • Create a new model by copying an existing model and changing the name and query
  • Create a new model and sync at the same time

    Need help?

    Our team is relentlessly focused on your success. We're ready to jump on a call to help unblock you.

    • Connection issues with your data warehouse?
    • Confusing API responses from destination systems?
    • Unsupported destination objects or modes?
    • Help with complex SQL queries?

    Feature Requests?

    If you see something that's missing from our app, let us know and we'll work with you to build it!

    We want to hear your suggestions for new sources, destinations, and other features that would help you activate your data.

On this page

SetupAuthenticate to GitEnable Git SyncSchemasValidating changesChangelogTunnelingNext steps

Was this page helpful?