Search documentation...

K

HTTP Request

Connect to any internal system, third-party API, or other web service.

Overview

This destination makes it possible to integrate Hightouch with almost any API in just a few minutes, all without writing a single line of code. Using our visual interface, you can exercise granular control over HTTP endpoints, request payloads, rate limits, concurrency limits, error handling, and more.

We designed the HTTP Request destination to be as flexible as possible. We want our customers to build their own integrations with third-party APIs, internal tools, and other services not yet supported in our growing catalog of 100+ native destinations. Under the hood, the HTTP Request destination still leverages Hightouch's powerful sync engine, so you'll continue to benefit from the security and observability features available in all of our native destinations.

In this article, we'll provide a technical overview of how the HTTP Request destination works. Towards the end, we'll demonstrate how easy it's to create a custom integration with Stripe's REST API in under five minutes.

How it works

Hightouch queries your data source and monitors the query results for added rows, changed rows, and/or removed rows. When configuring the HTTP Request destination, you specify which of these events should trigger an HTTP request. For each trigger, you then specify an endpoint, payload, and several other parameters.

To help Hightouch play nicely with other services, you can also specify constraints such as rate limits, concurrency limits, and special error handling logic. All of this is configured using our visual interface—no code required.

Let's walk through the whole process to see how it works.

Create a new destination

When setting up the HTTP Request destination for the first time, you can optionally provide headers to be included in each outbound HTTP request. These headers often include access tokens used for authentication.

That's it for destination setup. Endpoints, payloads, and other parameters are defined later during sync setup.

The HTTP Request destination doesn't automatically refresh tokens used for authentication, as may be required by some third-party APIs. This could prevent you from using this destination for certain use cases. If you need to fetch a new token during each sync, consider using one of our serverless function destinations (for example, AWS Lambda) or reach out to us for other solutions.

Choose request triggers

After setting up your HTTP Request destination, you can create a sync.

In this first step, you tell Hightouch when to trigger HTTP requests. Each trigger is configured separately. This allows your integration to have different behavior depending on whether a row is added, changed, and/or removed. (Most APIs use different endpoints and HTTP methods for creating, updating, and deleting resources.)

For example, suppose you're syncing customer data to an email marketing tool. You might want to create a contact when a row is added, update an existing contact when a row changes, and delete an existing contact when a row is removed. Having separate configuration for each trigger offers the flexibility required to support this use case.

A valid sync configuration requires at least one trigger.

Specify webhook endpoint

In this step, you tell Hightouch how to send HTTP requests to your internal system, third-party API, or other web service. Hightouch supports POST, PUT, PATCH, and DELETE requests to any endpoint.

When specifying the endpoint URL, you may either provide a static URL or a dynamic, templated URL.

If you provide a static URL, that same URL will be used for all HTTP requests triggered by the sync. For example, Stripe's REST API uses a single, static URL for creating new Customer objects: POST https://api.stripe.com/v1/customers.

If you provide a dynamic URL, Hightouch will generate a different URL for each HTTP request by incorporating data from your model. For example, Stripe's REST API requires you to reference an ID when deleting specific Customer objects: DELETE https://api.stripe.com/v1/customers/{{customer_id}}

How to use Liquid templates

Hightouch uses the Liquid templating language to generate dynamic content. Liquid supports variable injection, control flow, iteration, filters, and more. Details are available in the Liquid docs.

To insert a value from your model, use {{row.column_name}}, where column_name references any column in your model. For example, {{row.first_name}} or {{row.email_address}}.

To change the output of a Liquid object, append a filter like this: {{ row.column_name | filter }}. For example, {{ row.first_name | upcase }}{{ "Alice" | upcase }}ALICE. You can also chain multiple filters together like this: {{ row.product_id | remove: "0" | prepend: "SKU_"}}{{ "000123" | remove: "0" | prepend: "SKU_"}}SKU_123.

Customize request payload

Payload options in the Hightouch UI

In this step, you tell Hightouch how to build the request payload using data from your model. Hightouch supports JSON and XML payloads, as well as URL encoded forms. You can also send an empty body, if needed.

Most modern web APIs use JSON payloads. This destination offers three methods of composing a JSON request body:

Use JSON editor

Selecting the JSON editor method in the Hightouch UI

With the JSON editor, you can compose any JSON object using the Liquid template language. This is particularly useful for complex payloads containing nested objects and arrays, which can sometimes be difficult to model entirely in SQL.

Suppose your data model looks like this:

full_nameageemail_addressphone_number
John Doe30john@example.com+14158675309

And you want your HTTP request to have a payload like this:

{
  "name": "John Doe",
  "age": 30,
  "contact_info": [
    {
      "type": "email",
      "value": "john@example.com"
    },
    {
      "type": "phone",
      "value": "+14158675309"
    }
  ]
}

Your Liquid template should look like this:

{
  "name": "{{row.full_name}}",
  "age": {{row.age}},
  "contact_info": [
    {
      "type": "email",
      "value": "{{row.email_address}}"
    },
    {
      "type": "phone",
      "value": "{{row.phone_number}}"
    }
  ]
}

This makes it so you can reference any column using the syntax {{row.column_name}}. You can also use advanced Liquid features to incorporate control flow and loops into your dynamic payloads.

When injecting strings into your JSON request body, be sure to surround the Liquid tag in double quotes.

Use one column from model

Selecting using one column from the model as the paylod construction method in the Hightouch UI

If you're already storing JSON data in your source, or if you have the ability to construct a JSON object using SQL, you can select one column in your model that already contains the full request payload.

This setting is commonly used when syncing web events that have already been collected and stored as JSON objects in your database.

Use multiple columns from model

Selecting using multiple columns from the model as the paylod construction method in the Hightouch UI

For the simplest use cases, Hightouch can construct a JSON object with key/value pairs based on multiple columns in your model.

Suppose your model looks like this:

emailfirst_namelast_name
alice.doe@example.comAliceDoe
bob.doe@example.comBobDoe
carol.doe@example.comCarolDoe

The field mapping in the preceding screenshot would generate the following payload for the first row:

{
  "customer_first_name": "Alice",
  "customer_last_name": "Doe",
  "customer_email": "alice.doe@example.com"
}

You can use the field mapper to rename fields. For example, first_name can be mapped to customer_first_name.

Configure rate limiting and concurrency

In this step, you tell Hightouch how to avoid overwhelming your HTTP endpoint with requests. Most modern web APIs enforce rate limits, which set a maximum allowed number of requests per second, minute, or hour. Occasionally, APIs may also have concurrency limits, which set a maximum allowed number of requests that can be processed simultaneously.

Hightouch defaults to 1000 requests per second and 100 concurrent requests. You can override these defaults to meet the requirements of your web service. It goes without saying that rate limits and concurrency limits will both affect sync speed.

To ensure that Hightouch never exceeds your chosen limits, a small offset will be applied to the values provided in the sync configuration form.

Concurrency and rate limit controls aren't shared between syncs and do not persist across sync runs.

Configure error handling

In this step, you tell Hightouch how to handle errors, such as HTTP request timeouts and error responses.

These errors will be retried indefinitely until they succeed. You have a choice between retrying immediately or waiting until the next sync run. If you elect to retry immediately, you can specify how many retries should be attempted during the sync run. If all of these retries fail, the request will be retried during subsequent sync runs until it succeeds.

Any HTTP response with a 400- or 500-level status code is considered an error.

Configure initial sync behavior

In this step, you tell Hightouch how to handle rows present in your model results during the first sync run.

Certain workflows, such as hydrating a CRM for the first time, may require performing a backfill of all rows during the initial sync. For other use cases, such as sending product notifications, you might only want to make HTTP requests for future data changes.

Limitations

This destination makes one HTTP request per row added, changed, and/or removed. Therefore, it can't perform complex operations that involve lookups or multiple requests chained together.

If you need to make multiple requests per row, consider using one of our serverless function destinations: AWS Lambda, Google Cloud Functions, or Azure Functions. You could also build your own integration using our Embedded Destination framework.

Tutorial

In this guided tutorial, we'll use the HTTP Request destination to integrate Hightouch with Stripe's REST API. Specifically, we'll create a Customer object for each row in our data model.

Authentication via HTTP header

According to Stripe's docs, API requests can be authenticated using the Authorization header. We want every HTTP request to include a header like this:

Authorization: Bearer sk_test_4eC39HqLyjWDarjtT1zdp7dc

Obviously, the example key should be replaced with a real key that belongs to your Stripe account.

In Hightouch, you would provide this HTTP header during destination setup. The key would be Authorization, and the value would be similar to Bearer sk_test_4eC39HqLyjWDarjtT1zdp7dc.

Creating a customer object

Next, we want to understand Stripe's API for creating a Customer object. According to Stripe's docs, this can be done by making a POST request to the /v1/customers endpoint.

Here's how we'll configure the "rows added" trigger in Hightouch.

To construct the request payload, we'll need to reference our Hightouch model, which might look like this:

full_nameemail_addressphone_number
John Doejohn@example.com+14158675309

Stripe's docs tell us that request bodies should be "form-encoded," so we should use the URL encoded form payload type in Hightouch. To keep this tutorial short and sweet, we'll create our Customer objects with only the name, email, and phone parameters. Note that our field mapping makes sure to rename our model columns so that the final payload conforms to Stripe's API specification.

Next, we'll reference Stripe's docs regarding rate limits.

Stripe allows no more than 25 operations per second (while in test mode), so we'll set a rate limit of 25 requests per second and a concurrency limit of 25 requests at a time.

In the unlikely scenario that Stripe's API is experiencing downtime, we would want to wait a while before retrying failed rows. Therefore, we'll configure our sync to retry errors during the next sync run, and we'll give Stripe 30 seconds to respond to each of our HTTP requests.

Lastly, we want to make sure that our first sync run backfills our Stripe account with Customer objects for all rows present in the initial model results.

That's it. In just a few minutes, we built a custom integration with Stripe to create new Customer objects whenever new rows appear in our Hightouch model. The best way to verify our work is to test a row.

Updating or deleting a Customer object

In the future, we might want to update and delete existing Customer objects so that Stripe always stays in sync with the latest data in our source. To do this, we would enable the "rows changed" and "rows removed" triggers for the HTTP Request destination. The configuration for each of these triggers would be different from the tutorial above, but the same concepts apply. Just reference the Stripe API docs to determine the appropriate HTTP method, endpoint, and payload for each operation.

    Need help?

    Our team is relentlessly focused on your success. We're ready to jump on a call to help unblock you.

    • Connection issues with your data warehouse?
    • Confusing API responses from destination systems?
    • Unsupported destination objects or modes?
    • Help with complex SQL queries?

    Feature Requests?

    If you see something that's missing from our app, let us know and we'll work with you to build it!

    We want to hear your suggestions for new sources, destinations, and other features that would help you activate your data.

On this page

OverviewHow it worksCreate a new destinationChoose request triggersSpecify webhook endpointCustomize request payloadConfigure rate limiting and concurrencyConfigure error handlingConfigure initial sync behaviorLimitationsTutorialAuthentication via HTTP headerCreating a customer objectUpdating or deleting a Customer object

Was this page helpful?