Syncs

Syncs declare how you want query results from a model to appear in your destination.

To learn more about syncs on a conceptual level, see the core concepts page. Refer to destination docs for specific configuration details. To learn more about the information the Hightouch UI displays on your syncs and what actions you can take from the UI, continue to the following sections.

Syncs overview page

Hightouch displays all syncs, both active and inactive on the Syncs overview page.

For each sync, the overview shows:

• The sync's status
• The sync's associated model
• The sync's associated destination
• The date and time, in your timezone, of the sync's last run
• The date the sync was created at, and who created the sync

You can view more in-depth information for any sync by clicking on it. This takes you to that individual sync's overview page.

Sync status

Your sync's status is based on the latest information available on a run:

• Healthy: a recent run completed
• Pending: no runs have been scheduled yet
• Disabled: the sync has been turned off
• Querying: the sync is querying its associated model
• Preparing: the sync is preparing data to send to its associated destination
• Queued: the sync is scheduled to begin writing to its destination
• In Progress: the sync is writing to its destination
• Incomplete: the sync had a temporary interruption and will resume in a new run
• Aborted/Failed: there was an error in the sync progress—check the errors on the sync to see why the sync failed

Individual sync overview page

You can view an individual sync's information by clicking on it on the on the general Syncs overview page.

A particular sync's overview page shows general information, including:

• The sync's associated model and destination
• Whether the sync is currently enabled
• The option to manually Run the sync

You can also choose the resync the full query results of a model or delete a sync from here.

The overview page also shows more specific information including:

• Sync ID: a numerical unique identifier for the sync that is necessary when setting up an Airflow operator, or using the Dagster or Prefect integrations
• Sync schedule type: manual, interval, custom, cron, dbt Cloud-, or Fivetran-triggered
• Sync slug: a hyphenated string unique identifier for the sync
• Update mode: update, upsert, insert, archive, etc.
• Any associated labels: see label based access-controls for more information

Finally, you can click the tabs on the overview page to view and edit settings:

• Runs: You can see the status, time, and more details from the most recent runs. Selecting a particular run brings you to the Live Debugger.
• Configuration: You can view and edit the sync's configuration.
• Schedule: You view and edit the sync's run schedule.
• Alerts: You view and edit the sync's alerts.
• Sync logs: You can view and edit where your sync's historical logs are stored.

Resync full query

By default, Hightouch keeps track of changes to your data between syncs and only sends necessary updates. Therefore, an initial sync runs often shows that many records were 'Added'. Subsequent runs often show fewer records 'Added', and instead show many records have 'Changed' or 'Removed'.

If you wish to reset the sync's tracking to its initial state—as if it were syncing for the first time again—you can select Resync full query from the menu on your sync's main page.

Run details

When viewing a specific sync, you can inspect detailed run-level information. A run is a particular invocation of a sync, either triggered manually or scheduled.

Each run provides:

• The particular run's status
• The time the run started and its duration in seconds
• The number of rows queried
• The total number of operations, including rows added, changed, removed
• The results—the total number of successful operations in green and the number of rejected operations in red

If there were any errors, you can click the View Error button to view details related to the run.

Configuration

The configuration section exposes destination-specific configuration options. While sync configurations vary depending on the destination, they generally include:

• Type: What data type to sync to in your destination; for example, objects, events, subscription lists, etc. To learn more, see sync types and modes
• Mode: What update mode to use; for example, upsert, insert, update, etc. To learn more, see sync types and modes
• Record matching If updating or upserting, how rows in the source should be matched to records in the destination
• Field mapping: What columns from your source should be synced to the destination, and what destination fields they should appear as
• Batching: How data should be batched in API requests
• Delete behavior: What to do when a record leaves your source—in other words, when your model no longer returns a record that was returned in the previous sync run

Delete behavior

Depending on the destination and update mode, these are common options for delete behavior:

Check out the particular destination's documentation for more details.

Configuration actions

When working with a sync configuration, you can also choose to test a row or edit the configuration as a JSON object.

Test a row

You can test a sync by sending just one row from your model to your destination. Sending just one row lets you verify that the sync is working as intended before initiating a run of your entire sync.

From your sync's configuration page, click Test and select a row to sync. You can choose to sync the row as an added, changed row, or removed row and observe the results.

Edit as JSON

You can edit your sync's configurations as JSON code to quickly make multiple changes or copy and paste one sync's configurations to another.

For example:

{
    "mode": "upsert",
    "type": "object",
    "fromId": "track_visitor_id",
    "object": "user",
    "mappings": [
        {
            "to": "uuid-western",
            "from": "uuid"
        },
        {
            "to": "Trialer Status",
            "from": "trialer_status"
        }
    ],
    "customMappings": [
        {
            "to": "unique_notification_status",
            "from": "notify_status"
        }
    ]
}

Schedule

Part of sync configuration is deciding how frequently the sync should run. Besides triggering syncs manually, you can schedule syncs with these options:

• Interval: You can schedule your sync to run on a set interval, for example once every day.
• Custom recurrence: You can schedule your sync to run a specific date and time, for example every Monday at 9 AM.
• Cron expression: You can schedule your sync using a cron expression.
• dbt Cloud: You can trigger syncs automatically via dbt Cloud.
• Fivetran: You can trigger syncs automatically via Fivetran.

Though not listed as schedule options, you can also trigger syncs using Airflow, Dagster, Prefect, or the REST API. To learn more about your scheduling options, refer to the sync scheduling page.

If you have dependencies between syncs, for example you require one sync to complete before another begins, sync Sequences can help you manage scheduling.

Alerts

You can configure alerting on a per-sync basis from this section.

Sync logs

You can configure Warehouse Sync Logs from this section. This feature writes information on the results of your syncs back into your warehouse so that you can perform more complex analysis.