Databricks Zerobus Ingest

Overview

Databricks Zerobus Ingest sends events directly into your Databricks tables for analysis without waiting for a batch load. Pair it with a Databricks batch sync in Schema only mode to create tables and add columns as your events change.

Supported syncing

This destination supports event streaming for identify, track, page, screen, group, and alias events. It appends rows to existing tables. For model-based syncs, use the Databricks destination.

Connect to Databricks Zerobus Ingest

Use a Databricks workspace in a region that supports Zerobus Ingest. Target tables must be Unity Catalog managed Delta tables. External tables and views aren't supported.

Grant the identity used by Hightouch workspace access and these permissions:

• USE CATALOG on the target catalog.
• USE SCHEMA on the target schema.
• SELECT and MODIFY on the target tables.

See Databricks' authentication and permission instructions. The separate batch connection also needs permission to create tables and add columns, plus access to SQL compute.

Authenticate with OAuth

Create a Databricks service principal and OAuth secret with the permissions above. In the Hightouch destination, select OAuth (recommended) and enter its Client ID and Client Secret.

Authenticate with an access token

To use an existing Databricks identity, select Access token and enter a personal access token. The token's identity must have the same permissions.

Select the workspace and catalog

Enter these values in the Databricks Zerobus Ingest destination:

• Server hostname: Copy the hostname from your Databricks workspace URL, without https:// or a path.
• Workspace ID: Enter the numeric ID from the URL's o= parameter or the workspace details in the Databricks account console.
• Region: Enter the workspace's cloud region from the account console, such as us-west-2.
• Catalog: Enter the Unity Catalog catalog containing your event tables.

The Zerobus connection doesn't require a SQL warehouse. The batch sync uses SQL compute to maintain the tables.

Sync configuration

Stream events

Create two event syncs using the same event sources:

1. A Databricks event storage sync with Schema only enabled. This sync maintains tables and columns without writing event rows.
2. A Databricks Zerobus Ingest event forwarding sync. This sync writes the rows as events arrive.

If you're reusing tables from preexisting batch syncs, check for generated columns and migrate them before starting Zerobus delivery.

Match the event table layout

Set the same explicit Schema name on both destinations so both syncs target the same tables. Without an override, the schema name comes from the event source name, or the sync name when using multiple sources.

Match the catalog, sources, and any event function applied to the events. Also match these settings between the batch destination's Schema settings and the Zerobus destination's Event table layout:

• Per-event tables: Enabled by default. Each track event writes to tracks and a table named after the event. Disable it to use only tracks.
• Always write properties JSON to the Tracks table: Enable this to retain the full properties JSON in tracks when using per-event tables.
• Convert properties and context fields to columns: Enabled by default. Disable this to store those fields as JSON instead of individual columns.
• Write Page and Screen events to the Tracks table: Enable this to include those interactions in tracks. By default, they use separate tables.

See the warehouse schema for event table names and column conventions. These layout settings apply to every sync using the destination.

Schedule schema updates

Set up the Databricks event storage destination and create its batch sync. In the sync's Schedule settings, select Schema only. We recommend scheduling it every 15 minutes to pick up new tables and columns.

Send representative events and run the schema-only sync to create the initial tables. Keep the schema-only sync scheduled alongside Zerobus so new event fields can become columns. Queued or failed batch runs can delay schema updates.

When switching an existing batch sync, finish its active run and deliver any backlog you need before enabling Schema only. Avoid having the batch sync and Zerobus both write rows for the same events, which can create duplicates.

Schema changes and retries

When an event requires a missing table or column, Zerobus delivery retries for a limited time. Run the schema-only sync to create the missing table or column. Delivery can then succeed on a subsequent retry. Completing a batch run doesn't trigger an immediate streaming retry.

Retries aren't indefinite. Events that exhaust their retry limits are archived and need replay after the schema is fixed. Incompatible values, such as text sent to a numeric column, require correction before replay. Check the streaming sync's errors and delivery status separately from the batch run history.

Migrate existing batch tables

Older Databricks batch syncs create _date_partition as a generated column, computed from timestamp. Zerobus can't write to tables containing generated columns, even when the other columns match.

When Zerobus is enabled for your Hightouch workspace, newly created batch tables use a regular _date_partition DATE column. Preexisting tables aren't converted automatically, and require a manual migration.

For each existing event table, including per-event tables, inspect its definition:

SHOW CREATE TABLE <catalog>.<schema>.<table>;

If _date_partition includes GENERATED ALWAYS AS, migrate the table before streaming into it. The following example preserves existing rows and the standard _date_partition and _sync_partition partitioning. Replace the placeholders, and use unused names for the replacement and backup tables.

1. Pause batch runs

Complete this migration before starting the Zerobus sync. Keep the existing batch sync Enabled, change its Schedule to Manual, and wait for any active run to finish. Events can continue arriving and queue for the batch sync while you migrate the tables. Stop any other applications from writing to these tables during the migration.

2. Copy into a replacement table

If you use row filters or column masks, have your Databricks administrator ensure the copy includes the full, unmasked data. Run this SQL to copy the data with a regular date column:

CREATE TABLE <catalog>.<schema>.<table>_new
USING DELTA
PARTITIONED BY (_date_partition, _sync_partition)
AS
SELECT
  * EXCEPT (_date_partition, _sync_partition),
  CAST(_date_partition AS DATE) AS _date_partition,
  _sync_partition
FROM <catalog>.<schema>.<table>;

The replacement has a new table identity and history. Restore required ownership, grants, comments, constraints, table properties, and security policies before resuming access.

3. Verify the copy and restore access

Compare row counts, inspect sample records, and confirm the replacement has no generated columns:

SELECT COUNT(*) FROM <catalog>.<schema>.<table>;
SELECT COUNT(*) FROM <catalog>.<schema>.<table>_new;
SHOW CREATE TABLE <catalog>.<schema>.<table>_new;

Grant the Zerobus identity SELECT and MODIFY on the replacement table. Retain its USE CATALOG and USE SCHEMA permissions, and restore access for the batch connection and other readers or writers.

4. Swap names and resume

With writers still stopped, rename the original table to a backup and give the replacement the original name:

ALTER TABLE <catalog>.<schema>.<table>
RENAME TO <catalog>.<schema>.<table>_backup;

ALTER TABLE <catalog>.<schema>.<table>_new
RENAME TO <catalog>.<schema>.<table>;

The two renames aren't atomic. After the swap, run the batch sync with Schema only off to deliver queued events. You can restore its schedule to continue batch loading. Keep the backup until you've verified new deliveries.

When you're ready to start Zerobus, briefly pause incoming events and finish delivering the remaining batch backlog. Then select Schema only, restore the batch schedule, enable the Zerobus sync, and resume incoming events. Events consumed in schema-only mode won't be backfilled by the new streaming sync.

Other applications writing to this table must now supply _date_partition themselves. For queries that filter by timestamp, add an explicit _date_partition filter when needed for partition pruning.

Tips and troubleshooting

Behavior and limitations

Delivery is at least once, so retries and replays can create duplicate rows. For example, a track event can succeed in tracks before its per-event table fails. Retrying the event can write another row to tracks. Use the event's id column to deduplicate when needed.

Validate your setup

Send a test event with a recognizable messageId. In Databricks, query its event table for the matching id and verify the field values. For a track event with per-event tables enabled, check both tracks and the event-specific table.

Then send an event with a new property. Confirm the schema-only batch run adds the column without adding rows, and the streaming event arrives after a successful retry. Databricks acknowledgment confirms durable receipt, so a successful delivery can precede query visibility.

Monitor delivery

Use the source Debugger to inspect received events and individual delivery attempts. Use the streaming sync's error monitoring to check retries and archived events, and configure alerts for delivery failures. Monitor the schema-only batch sync too: failed schema updates can prevent new event fields from reaching Databricks.