Get end-to-end visibility for how your data is flowing between dbt and Hightouch.
By Hugo Darwood on August 2nd, 2022Data
When building your team’s data pipelines, it can be difficult to have visibility into how your tables and data are used downstream. This lack of visibility can lead to unintentional breaking changes, wasted time as you try to maintain unused tables and frustration as you try to piece together which syncs are associated with certain tables. dbt recently launched a new feature called Exposures to give you a way to express the downstream utilization of your sources and models in data docs and lineage graphs.
We’re excited to announce that Hightouch now supports dbt Exposures natively in our app. With a simple toggle switch, you can automatically update your dbt repos with Exposures to see where models are being used in your team’s Hightouch syncs. This means better end-to-end visibility of your data lineages, giving you clearer context for how data moves through your pipelines so you can avoid breaking changes.
🎥 Check out this video for a quick demo of how it works.
Pull existing dbt models via git and sync modeled data directly from the warehouse to 100+ business tools
Trigger Hightouch syncs to execute after dbt Cloud jobs run
Get complete context for how Hightouch syncs are related to dbt models
If you’re new to dbt (we’ve been there!), we recently wrote a primer on dbt with some simple tips on how you can jumpstart your knowledge.
Setting up dbt Exposures in Hightouch
To activate dbt Exposures in Hightouch, head to the Extensions tab and select the dbt-compatible source you want to set up. If you have not yet activated dbt model sync for your source, you should do so now. Enable the toggle for dbt Exposures Sync, which will create commits in your dbt repo to create the Hightouch exposures file (just make sure Hightouch has write access to your dbt branch).
Once you have enabled dbt Exposures, you should see the sync status indicator turn green and a commit created in your dbt repo. dbt Exposure syncs will run whenever there are changes to a Hightouch sync that use a dbt model.
dbt Exposures Sync
If Hightouch encounters an error updating your dbt Exposures, you can see the error details under the “activation toggle,” helping you debug faster.
Now, when you rebuild your lineage graphs in dbt on your latest commit, you’ll be able to see exactly which Hightouch syncs are using your dbt models.
Anytime a Hightouch sync is created or updated that uses one of your dbt models, Hightouch will update the Exposures file in your dbt repo, showing you your updated Exposures when you next rebuild your dbt docs. You can also quickly visualize where you need to run your DAG up to to ensure your Hightouch syncs are using the freshest data.
In the next few months, we will continue to invest in our dbt integration to further surface the connection between your dbt models and your downstream data flows. We want to help you ship changes to dbt models with confidence by providing better validation and simulations of the impact radius on all dependent syncs.
We’re also working closely with our partners in data observability to better flag when changes upstream can impact Hightouch syncs. Check out our recent integration with Datafold to learn more.