Materialize is a streaming database powered by Timely and Differential Dataflow, purpose-built for low-latency applications. It lets you ask complex questions about your data using SQL, and incrementally maintains the results of these SQL queries as the underlying data changes.
View Materialize's documentation.
Hightouch lets you pull data stored in your Materialize database and push it to downstream destinations. Most of the setup occurs in the Hightouch UI, but you need access to your Materialize instance for information like your host, port, database name, cluster, and credentials.
You need to allowlist Hightouch's IP addresses to let our systems contact your warehouse. Reference our networking docs to determine which IPs you need to allowlist.
To get started, go to the Sources overview page and click the Add source button. Select Materialize and follow the steps below.
Enter the following required fields into Hightouch:
- Host: The hostname or IP address of your Materialize server.
- Port: The port number of your Materialize server. The default port number is 6875, but yours may be different.
- Database: This specifies the database to use when Hightouch executes queries in Materialize.
Enter the following fields into Hightouch:
- User: This can be your personal Materialize login or a dedicated user for Hightouch. At minimum, this user must have read access to the data you wish to sync.
- Password: The password for the user specified above.
When setting up a source for the first time, Hightouch validates the following:
- Network connectivity
- Materialize credentials
- Permission to list schemas and tables
- Permission to write to
- Permission to write to
All configurations must pass the first three, while those with the Lightning engine must pass all of them.
Some sources may initially fail connection tests due to timeouts. Once a connection is established, subsequent API requests should happen more quickly, so it's best to retry tests if they first fail. You can do this by clicking Test again.
If you've retried the tests and verified your credentials are correct but the tests are still failing, don't hesitate to .
Once your source configuration has passed the necessary validation, your source setup is complete. Next, you can set up models to define which data you want to pull from Materialize.
The Materialize source supports these modeling methods:
- writing a query in the SQL editor
- using the visual table selector
- leveraging existing dbt models
- leveraging existing Looker Looks
- leveraging existing Sigma workbooks
You may also want to consider storing sync logs in Materialize. Like using the Lightning sync engine versus the standard one, this feature lets you use Materialize instead of Hightouch infrastructure. Rather than performance gains, it makes your sync log data available for more complex analysis. Refer to the warehouse sync logs docs to learn more.
You must enable the Lightning sync engine to store sync logs in your warehouse.
If you encounter an error or question not listed below and need assistance, don't hesitate to . We're here to help.
This error occurs when the required execution time of your Materialize query exceeds the timeout limit for the database. To address the error, increase the timeout by executing the following query:
set statement_timeout = '300 s'; -- 300 seconds, 5 minutes
Be sure to adjust the time to be as long as it takes for your query to execute.