Sync data from
Elasticsearch to AlloyDB

Why is it valuable to sync Elasticsearch data to AlloyDB?

Platforms like Elasticsearch have become the standard for modeling and transforming large quantities of data so you can answer complex analytics questions as quickly and efficiently as possible.

On the other hand, production databases like AlloyDB aren't designed to tackle large complex queries or transform your data. They're built to power your product experiences, and handle large volumes of small transactions in real-time, whether it's looking up or editing information about a single user, processing orders, accepting payments, or even granting access to specific product features.

Unless you have an unlimited amount of time and money, it's impossible to calculate core metrics about your customers and build behavioral prediction models in your AlloyDB instance. Transactional databases just aren't made for analytics use cases. Most likely you're already ingesting all of your production level data into your data warehouse via an ETL pipeline, so the logical step is simply to sync that data to AlloyDB.

Providing user-level recommendations to improve your on-site personalization requires you to categorize your users into groups based on their behavior. You might want to offer a coupon to customers with certain products in their cart, or maybe you want to group users into specific categories (e.g., power users, garden lovers, high-value customers, etc.) Either way, it's only possible and performant to build these data models in your warehouse.

Embedded analytics is also another extremely relevant use case for AlloyDB. If your app offers built-in reporting and visualization features it's much easier to do aggregations and transformations in your warehouse and sync those results to your application database to power your user-facing visualizations.

If you've ever built an internal application like an in-house CRM or a marketing platform, there's a good chance it's running off of your application database. In many cases, your AlloyDB database doesn't have all of the modeled data you need to power this end tool and that's why it's so important to hydrate your AlloyDB instance with modeled customer data directly from your warehouse.

Why should you use reverse ETL to connect Elasticsearch and AlloyDB data?

In the past syncing data from your data warehouse to AlloyDB required you to integrate with various APIs and build and maintain in-house pipelines. Even if your engineering team successfully builds a custom pipeline to your production database, a single API change or rate limit can quickly break everything.

Integrating with third-party APIs is complex, expensive, and time-consuming, so the path of least resistance is often downloading and uploading manual CSV files. CSVs are not a long-term solution because they go stale quickly, and you can never fully trust the accuracy of your data.

Workflow automation tools have arisen to solve this problem, but managing various if/then statements creates an intricate web of dependencies prone to failure. On the other hand, customer data platforms (CDPs) force you to create a second source of truth and pay for another storage layer in addition to your warehouse.

Hightouch eliminates these problems with Reverse ETL. You can query directly against your data warehouse using standard SQL, your current tables, and even use your existing data models. All your data is automatically diffed between sync runs to ensure you're only syncing the freshest data.

Any failed rows are automatically retried in the next sync. You can easily view live API responses/requests and use a live debugger to identify failed runs and problematic data. Hightouch will write the results of your sync back to your warehouse so you can easily analyze your logs.

Hightouch even integrates with git so you can manage and update your syncs bi-directionally in your git repo. You can even send alerts to your favorite messaging tools like Slack or email.

With Hightouch, all you have to do is connect to your data warehouse and map the proper columns in your data warehouse to the appropriate fields in your destination.