Google PubSub

Overview

Hightouch supports streaming events from Google PubSub in realtime.

Simply configure your Google PubSub source in the Hightouch UI, and Hightouch will automatically extract, transform, and load your data into Hightouch Events. Your data can then be synced to your warehouse or streamed to our catalog of realtime destinations.

Setup

To get started, create an event source, select Google PubSub and follow the steps below.

Configure your source

Configure the following fields in Hightouch:

• Credentials: GCP credentials for connecting to PubSub.
• Subscription: The full name of the PubSub subscription to consume events from.
• Project ID: The Google Cloud Project ID.
• Regional API Endpoint: List of endpoints

Grant permissions

Hightouch believes in the principle of least privilege. We ask for no more permissions than necessary. For the Google PubSub event source, Hightouch requires roles/pubsub.subscriber and roles/pubsub.viewer. Ensure you run the command in your Google Cloud Console to grant Hightouch's service account the correct permissions.

Message format

Hightouch Events supports PubSub messages encoded using Avro and Protobuf as long as a PubSub Schema is configured and associated with the topic. Otherwise, messages need to be encoded as JSON.

Event Structure

All events are ingested into Hightouch as track events. The PubSub message id, data, and attributes are exposed as custom properties on the track event. Metadata such as subscription are also exposed in the context.pubsub object.

For example, the following PubSub message:

{
  "id": "123",
  "data": {
    "user_id": "user_123",
    "amount": 123.45
  },
  "attributes": {
    "a1": "abc"
  }
}

Will be ingested into Hightouch as:

{
  "type": "track",
  "event": "Google PubSub Event",
  "properties": {
    "id": "123",
    "data": {
      "user_id": "user_123",
      "amount": 123.45
    },
    "attributes": {
      "a1": "abc"
    }
  },
  "context": {
    "pubsub": {
      "subscription": "projects/my-project/subscriptions/my-subscription"
    }
  },
  "timestamp": "2024-07-29T14:52:37.648Z"
}

Field mappings

You can configure how Hightouch transforms messages from your PubSub topic to Hightouch track events. Mappings can be configured for the following fields:

• event
• userId
• anonymousId
• messageId
• timestamp

You can also transform your events using Functions in order to standardize the schema or enrich its contents.

Schema Enforcement

Event Contracts can be attached to Google PubSub sources just like any other Event Source.

Tips and troubleshooting

If you encounter an error or question not listed below and need assistance, don't hesitate to reach out. We're here to help.