This is the first part of our series on push technologies. In this part, we will provide a primer on Webhooks and look at real-world APIs that support this style.
Webhooks are used in the style of API where the server pushes, or streams, data to the client. The client does not have to make repeated requests to the server. This push/streaming architectural style of API is well suited to use cases where the underlying data is refreshing itself constantly, such as a stock ticker or a social activity stream.
In a nutshell, a Webhook is a different approach from your typical Web API. As opposed to the usual RESTful API deployment where a server hosts an HTTP-based API endpoint that clients (the "API consumers") pull data from one request at a time, Webhooks reverse the direction of the conversation. It is the client that hosts an HTTP-based API endpoint to which the server pushes data as it becomes available. That endpoint is known as a webhook.
Webhooks are a push notification style that, in comparison to the routing capabilities of other push/streaming-styled APIs, sits at the coarse-grained end of the spectrum in terms of sophistication. Compared to more narrowly focused push mechanisms, this style of push provides limited capabilities for routing to individual application users. It's doable, but Webhooks are better for pushing notifications to one or a small number of endpoints. If the notification is intended for an individual application user, the owner of the endpoint typically takes responsibility for marshaling the notifications received by the Webhook to the correct recipient.
Webhooks employ an HTTP endpoint that supports the POST HTTP method to provide a means to allow an API provider to "call back" an API consumer with the result of a long-running or out-of-band process. The clients in these client/server interactions are almost always servers themselves; therefore, these callbacks are almost always server-to-server integrations. Using Webhooks to push directly to client applications, such as mobile apps, would be impractical and difficult to implement given the need for each client to host an HTTP endpoint and to be in possession of a publicly addressable domain name. Moreover, securing this network using traditional means, such as basic authentication or mutual SSL, would involve an almost unmaintainable administration overhead.
Webhooks have no formal standards at the time of this article's writing, and implementations tend to vary among the API providers that support them. However, a Webhooks implementation would typically include three steps, during which the API consumer calls the API with a request to receive notifications, and the server calls back with its stream. Those steps are:
- An API provider implements an API that invokes long-running processes
that are impossible to wait on over a synchronous connection or that generate out-of-band events. What is next required is the notification of the API consumer. An example could be a help desk API that creates tickets that need human interaction to complete over a number of days.This API would also trigger status updates that the API consumer needs to know throughout the ticket's lifespan. - An API consumer registers to use the API and configures its settings (via the provider's developer portal) with the URL to his or her publicly available endpoint (with some security features in place). The API provider can "stream" back to this endpoint when the long running process completes or as this process triggers events that should be reported back to the consumer.
- A client-side process might then continue some workflow on the basis of the content of the data that was streamed to its Webhook. For example, in the spirit of programmed-trading, a Webhook might belong to a stock brokerage firm and the stream of data being pushed to that Webhook could include stock prices that might trigger the sale or purchase of a publicly-traded stock.
The scenario described above involves pre-registered URLs, but it is technically feasible to supply a Webhook URL on the fly when an API call is made by the consumer. Both methods have pros and cons:
- Pre-registered Webhooks are less flexible for the API consumers that host them, because configuration changes are required whenever the consumer wishes to change the Webhook address.
- On-the-fly Webhooks could be subject to a security threat if the inbound request is intercepted and changed by a man-in-the-middle style attack. Additional security, such as message signing or certificate pinning, is required to ensure non-repudiation of both parties.
Examples
While Webhooks are a coarse-grained mechanism for enabling push notifications, they are also simple and powerful. Many API providers design their Webhooks with features that make sense in the context of the provider's market or product offering. For example, an API provider might offer separate dedicated endpoints for specific event types. Here are some examples of API providers that support a Webhook-based push/streaming API architectural style:
Stripe
Stripe is a popular payments API provider that employs Webhooks for out-of-band events that are generated as a result of using the Stripe API. They notify the API consumer of disputed charges and recurring billing events. When an event fires, Stripe creates an object that is pushed to the registered URL. Moreover, Stripe also allows API consumers to register multiple URLs and filter which events go to which URLs. The event types are configurable in their developer portal and include account updates, balance changes, etc.
This design mimics the kind the flexibility offered by a true publish/subscribe-based system that employs topics as a means of tailoring the events that are pushed to the API consumer.
It has many benefits for API consumers because it can support a granular, loosely coupled architecture where API consumers could create multiple microservices, each processing a different callback event from Stripe.
GitHub
Like Stripe, GitHub, which is arguably the world's leading software-as-a-service (SaaS) code repository, provides Webhooks as a means of sending out-of-band events to API consumers. Examples of such events include:
- Branch or tag deletion
- A repository being forked
- Push and pull requests
GitHub allows developers to create Webhooks for an entire organization or for individual repositories, giving API consumers a reasonable amount of flexibility over how to configure their services. Again like Stripe, events can also be filtered at source, so API consumers tailor exactly which events should trigger notifications, with filters available to developers in their repository settings:
The GitHub example is also interesting because the events being pushed to the Webhook may not originate from activity on the GitHub API. Any organizational or repository activity can generate an event that can fire a callback to a registered Webhook, with the event being used to initiate activities such as code review or even triggering a message in a Slack channel. This shows how broad the application of Webhooks can be.
Twilio
Twilio is a telecommunications API provider that delivers extensive capabilities via Webhooks for API consumers to receive events based on messaging and telephone call activity, categorized as "pre" and "post" events. The pre-event callback provides an API consumer the ability to apply rules that restrict certain events from taking place. For example, a developer can use a pre-event callback sent via a Webhook to prevent an SMS message from being sent or to halt a call taking place. Developers can use the Twilio Portal to configure both their Webhook URL and the type of events they wish to see:
Like the GitHub example, Twilio demonstrates the fact that Webhooks are not simply coupled to returning the results of long-running tasks that cannot be returned by a synchronous API call, they also allow developers to add interesting and useful functionality to their applications. Through these additions, developers can add value to the core features of the service they are consuming.
Slack
Slack, the extremely popular team collaboration tool implements both incoming and outgoing Webhooks to allow users to integrate bots and services into their Slack channels in a very simple manner.
Incoming Webhooks in Slack's context are designed to make a developer's life easy by giving them an easy-to-use interface for posting unstructured data to Slack channels, which of course is how a bot initiates a conversation when prompted from an external stimulus; incoming Webhooks are, however, really a simple API call.
Slack's outgoing Webhooks are also extremely easy to configure and allow calls to be made to bots or external services to elicit a response in the channel. A developer is required to set a trigger word and URL and Slack will automatically POST an event to the URL; the developer's service needs only to respond with a simple JSON fragment with the property "text," the contents of which will be displayed in the channel.
Therefore, Slack outgoing Webhooks allow developers to make a conversational response very easily with minimal restrictions from Slack itself, enabling all manner of bot related integrations.
This is part one of our series on push technologies. In part two we will provide a primer on PubSubHubbub.