Sync data from
Databricks to Elasticsearch
Connect your data from Databricks to Elasticsearch with Hightouch. No APIs, no months-long implementations, and no CSV files. Just your data synced forever.
Trusted by data teams at
Trusted by data teams at
Integrate your data in 3 easy steps
Add your source and destination
Connect to 15+ data sources, like Databricks, and 150+ destinations, like Elasticsearch.
Define your model
Use SQL or select an existing dbt or Looker model.
Sync your data
Define how fields from your model map to Elasticsearch, and start syncing.
Model your Databricks data using any of these methods
dbt Model Selector
Sync directly with your dbt models saved in a git.
Create and Edit SQL from your browser. Hightouch supports SQL native to Databricks.
Select available tables and sheets from Databricks and sync using existing views without having to write SQL.
For less technical users, pass traits and audiences from Databricks using our visual segmentation builder.
Does this integration support in-warehouse planning?
Yes, if you integerate Databricks and Elasticsearch using Hightouch, in-warehouse planning is supported.
Great, but what is in-warehouse planning?
Between every sync, Hightouch notices any and all changes in your data model. This allows you to only send updated results to your destination (in this case Elasticsearch). With the baseline setup, Hightouch picks out only the rows that need to be synced by querying every row in your data model before diffing using Hightouch’s infrastructure.
The issue here is this can be slow for large models.
Warehouse Planning allows Hightouch to do this diff directly in your warehouse. Read more on how this works here.
Why is it valuable to sync Databricks data to Elasticsearch?
Platforms like Databricks have become the standard for modeling and transforming large quantities of data so you can answer complex analytics questions as quickly and efficiently as possible.
On the other hand, production databases like Elasticsearch aren't designed to tackle large complex queries or transform your data. They're built to power your product experiences, and handle large volumes of small transactions in real-time, whether it's looking up or editing information about a single user, processing orders, accepting payments, or even granting access to specific product features.
Unless you have an unlimited amount of time and money, it's impossible to calculate core metrics about your customers and build behavioral prediction models in your Elasticsearch instance. Transactional databases just aren't made for analytics use cases. Most likely you're already ingesting all of your production level data into your data warehouse via an ETL pipeline, so the logical step is simply to sync that data to Elasticsearch.
Providing user-level recommendations to improve your on-site personalization requires you to categorize your users into groups based on their behavior. You might want to offer a coupon to customers with certain products in their cart, or maybe you want to group users into specific categories (e.g., power users, garden lovers, high-value customers, etc.) Either way, it's only possible and performant to build these data models in your warehouse.
Embedded analytics is also another extremely relevant use case for Elasticsearch. If your app offers built-in reporting and visualization features it's much easier to do aggregations and transformations in your warehouse and sync those results to your application database to power your user-facing visualizations.
If you've ever built an internal application like an in-house CRM or a marketing platform, there's a good chance it's running off of your application database. In many cases, your Elasticsearch database doesn't have all of the modeled data you need to power this end tool and that's why it's so important to hydrate your Elasticsearch instance with modeled customer data directly from your warehouse.
Why should you use reverse ETL to connect Databricks and Elasticsearch data?
In the past syncing data from your data warehouse to Elasticsearch required you to integrate with various APIs and build and maintain in-house pipelines. Even if your engineering team successfully builds a custom pipeline to your production database, a single API change or rate limit can quickly break everything.
Integrating with third-party APIs is complex, expensive, and time-consuming, so the path of least resistance is often downloading and uploading manual CSV files. CSVs are not a long-term solution because they go stale quickly, and you can never fully trust the accuracy of your data.
Workflow automation tools have arisen to solve this problem, but managing various if/then statements creates an intricate web of dependencies prone to failure. On the other hand, customer data platforms (CDPs) force you to create a second source of truth and pay for another storage layer in addition to your warehouse.
Hightouch eliminates these problems with Reverse ETL. You can query directly against your data warehouse using standard SQL, your current tables, and even use your existing data models. All your data is automatically diffed between sync runs to ensure you're only syncing the freshest data.
Any failed rows are automatically retried in the next sync. You can easily view live API responses/requests and use a live debugger to identify failed runs and problematic data. Hightouch will write the results of your sync back to your warehouse so you can easily analyze your logs.
Hightouch even integrates with git so you can manage and update your syncs bi-directionally in your git repo. You can even send alerts to your favorite messaging tools like Slack or email.
With Hightouch, all you have to do is connect to your data warehouse and map the proper columns in your data warehouse to the appropriate fields in your destination.
Insert and update documents in Elasticsearch indices
Databricks is a data science and analytics platform built on top of Apache Spark. Databricks implement the Data Lakehouse concept in a single unified, cloud based platform.Learn more about Databricks
Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.Learn more about Elasticsearch
Other Databricks Integrations
Other Elasticsearch integrations
Hightouch Playbooks: Best practices to leverage reverse ETL
Read more about Hightouch
What is Operational Analytics & Why You Should Use It
Operational Analytics shifts the focus from simply understanding data to taking action on it in the tools that run business processes. Instead of using dashboards to make decisions, Operational Analytics is focused on turning insights into action – automatically.Read
Activate data to any of your marketing and advertising tools
This might be one of the greatest inventions for technical marketers since the advent of legacy CDPs back in 2015.
Head of Marketing Technology
Your data is always secure
SOC 2 Type 2 compliant
Your data stays secure, available, and confidential. To see our report, .
If you’re in the EU, your data is only processed on EU data centers.
Healthcare companies like ThirtyMadison, Chapter Health, and Headway trust Hightouch.
To see our DPA (Data Processing Addendum), .
increase in return on ad spend
improvement in email engagement
lift in customer acquisition