Amazon Kinesis

Overview

Hightouch can read events from an Amazon Kinesis data stream in real time. After you connect a stream, Hightouch polls every open shard and converts each record into a Hightouch track event. From there, you can load events into your warehouse or stream them to real-time destinations.

Setup

To get started, create an event source and select Amazon Kinesis under Streaming / Message queues.

Configure your credentials

Select an existing AWS credential or create a new one. Hightouch supports two credential types:

• Cross-account role (recommended): Create an IAM role in your AWS account that trusts Hightouch, using the Account ID and External ID shown in Hightouch, then paste the role's ARN into Hightouch.
• Access key: Provide the Access Key ID and Secret Access Key of an IAM user.

For step-by-step instructions on creating either credential type, see the Amazon Kinesis destination setup. The process is the same; only the required permissions differ.

Grant permissions

Hightouch follows the principle of least privilege. The IAM role or user needs the following IAM actions on each stream:

You can use the following JSON sample to create your IAM policy. Leave out the second statement if you don't configure a dead-letter stream.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:DescribeStreamSummary",
        "kinesis:ListShards",
        "kinesis:GetShardIterator",
        "kinesis:GetRecords"
      ],
      "Resource": "arn:aws:kinesis:YOUR_REGION:YOUR_ACCOUNT_ID:stream/YOUR_STREAM_NAME"
    },
    {
      "Effect": "Allow",
      "Action": ["kinesis:PutRecords"],
      "Resource": "arn:aws:kinesis:YOUR_REGION:YOUR_ACCOUNT_ID:stream/YOUR_DLQ_STREAM_NAME"
    }
  ]
}

Configure your source

Enter the following fields:

• Stream name: The name of the Kinesis data stream to read events from.
• Region: The AWS region where the stream lives.
• DLQ stream name (optional): A Kinesis stream in the same region where Hightouch writes records it can't process. See Dead-letter stream.

The stream must have a status of ACTIVE or UPDATING. Hightouch doesn't support SSH tunnels for Kinesis sources.

Message format

Hightouch Events requires each Kinesis record's data to be JSON. Records that aren't valid JSON, or that exceed the 96 KB event size limit, can't be processed.

Dead-letter stream

If Hightouch encounters a fatal error while processing a record, such as data that isn't valid JSON, it writes the record to the dead-letter stream you configured. If you don't configure a dead-letter stream, Hightouch drops those records.

Each dead-letter record uses the original record's partition key. Its data is a JSON object that contains the original record, the error, and the ID of the event source that wrote it:

{
  "message": {
    "partitionKey": "user_123",
    "sequenceNumber": "49590338271490256608559692538361571095921575989136588898",
    "shardId": "shardId-000000000000",
    "streamName": "my-stream",
    "approximateArrivalTimestamp": "2026-09-28T14:52:37.648Z",
    "data": "not valid json",
    "truncated": false
  },
  "error": "Unexpected token o in JSON at position 1",
  "hightouchEventSourceId": "YOUR_EVENT_SOURCE_ID"
}

Hightouch truncates the original data to 2,048 characters and sets truncated to true when it's longer.

Starting position

When you first connect a stream, Hightouch starts reading from the latest position in each shard. Records written to the stream before the source was created aren't ingested. After that, Hightouch checkpoints its position in each shard after every batch, so it resumes where it left off after a restart.

Hightouch also follows resharding. When a shard closes after a split or merge, Hightouch continues reading from its child shards.

Event structure

Hightouch ingests every Kinesis record as a track event:

• The record's partition key and parsed JSON data go into properties.partitionKey and properties.value.
• Stream metadata (streamName, shardId, and sequenceNumber) goes into context.kinesis.
• The record's approximate arrival time becomes the event timestamp.

For example, a record written to the my-stream stream with partition key user_123 and this data:

{
  "user_id": "user_123",
  "amount": 123.45
}

Is ingested as:

{
  "type": "track",
  "event": "My Kinesis Source Event",
  "properties": {
    "partitionKey": "user_123",
    "value": {
      "user_id": "user_123",
      "amount": 123.45
    }
  },
  "context": {
    "kinesis": {
      "streamName": "my-stream",
      "shardId": "shardId-000000000000",
      "sequenceNumber": "49590338271490256608559692538361571095921575989136588898"
    }
  },
  "timestamp": "2026-09-28T14:52:37.648Z"
}

By default, the event name is your event source's name followed by Event, and the message ID is built from the source ID, shard ID, and sequence number.

Field mappings

You can configure how Hightouch maps fields in each record to Hightouch track events. Mappings can be configured for the following fields:

• Event Name: a constant value or a field from the record
• User ID
• Anonymous ID
• Message ID
• Timestamp

Field paths are relative to the record's JSON data. For example, to use user_id from the record above as the user ID, enter user_id, not properties.value.user_id. If an event ends up with neither a user ID nor an anonymous ID, Hightouch generates a random anonymous ID for it.

You can also use Functions to standardize schemas or enrich event data.

Schema enforcement

Event contracts can be attached to Amazon Kinesis sources just like any other event source.

Tips and troubleshooting

If you encounter an error or question not listed below and need assistance, don't hesitate to reach out. We're here to help.