How and why to provide event-driven streaming APIs

The grand majority of APIs that make up ProgrammableWeb’s API directory are REST-fashioned APIs. Here at ProgrammableWeb, we don’t get too religious about what is and what isn’t a REST API. Doing so would be a means to the wrong end. Developers just want to be able to find APIs that are generally patterned after REST (that depend on HTTP verbs) versus something like RPC-styled APIs that depend on SOAP and XML. We of course want to enable those searches.

But, as we take a closer look at the various APIs that are either already registered in the directory, or that are being added to our directory, we are starting to see some critical mass gather around the various push/streaming API technologies such as Webhooks and WebSockets. It is no longer uncommon for an API vendor to launch two or more endpoints for developers to consume. Github, for example, now offers three separate APIs for programmatic interaction with its service; one each for REST , Webhooks , and GraphQL .

In contrast to their REST-fashioned cousins which are usually implemented in polling scenarios (where a service has nothing to offer until a client explicitly asks for it), push/streaming APIs are event-driven. When some event happens, it triggers a data flow to a client that’s waiting for the inbound data in order to process it in near real-time. Twitter’s infamous “firehose” API streams tweets back to waiting clients as those tweets happen.

But, unlike the way most REST-fashioned Web APIs follow the same basic patterns, there are multiple approaches to push/streaming that feature some serious differences from one to the next. With Web APIs for example, you can package a query directly into the URL string that addresses an API’s endpoint, or if the API provider allows it, you can package that same query into a JSON payload (that goes like an attachment to the Web request) instead. These differences are subtle.

But just looking at two of the many push/streaming approaches -- Webhooks and Websockets -- the differences are not so subtle. For example, Websockets allow for bidirectional streams. Webhooks (which are technically REST-fashioned given the way servers “call” clients after an event) are unidirectional. Two other push/streaming technologies -- XMPP and SMPP -- differ substantially in that one (XMPP) is primarily based on XML payloads while the other relies on binary data.

In other words, once you’ve decided it’s time to offer some sort of real-time, push/streaming-styled APIs to your developers (and more than likely, you will), there are some big decisions to make. And that’s why we’ve put this series on understanding push/streaming APIs together for you; so that you will not only understand the pros and cons of the various approaches to event-driven push/streaming APIs, but that you can also identify some of the turnkey solutions that will help get you to the finish line.

Like all of ProgrammableWeb’s API University series, we will continually update this series as we come across new information that might be relevant to the decisions you're making. For example, just as we were “going to press” on this series, we discovered a new API that was based on an older (but apparently still-relevant) innovation from Zapier called REST Hooks. Similarly, we’re studying Google’s gRPC which, unbeknownst to many, supports bi-directional streaming (thanks to its roots in HTTP/2) in addition to the polling that’s typical of most of today’s APIs. As we conclude our findings on those and other event-driven API technologies, we’ll be sure to update this series so as to keep it as timely and relevant to your decisions as possible.

Today the conditions are more favorable than ever for organizations to consider implementing event-driven non-polling based data retrieval patterns in their APIs. This series aims to help you to better understand your options when it comes to push/streaming API architectural styles.

This is the first part of our series on push technologies. This article will look at Webhooks, explaining what they are, how they work and why they are well suited to use cases where the underlying data is constantly refreshing. We will also look at examples of APIs that are supporting this style.

This is the second part of our series on push technologies. This article will look at PubSubHubbub, a closely related cousin to Webhooks. PubSubHubbubb is an API technology used to publish information on the Internet by pushing content instead of forcing clients to poll for it.

Continuing our series on push technologies, part three looks at WebSockets and how they are appropriate for when you need a server-to-client real time interactive exchange. Part three discusses what you can do with WebSockets and how it differs from other push technologies.

Even though Webhooks, PubSubHubbub, and WebSockets are really popular, they aren't the only options at your disposal. This part of the series provides overviews of a number of other push alternatives that you might find helpful for meeting the needs of your particular application.

The previous parts of this series helped you become familiar with various kinds of push technologies. Sometimes you will want to build your own push API from scratch. Other times that isn't the case and this article helps familiarize you with some of the turnkey push API infrastructure...

GraphQL Subscriptions are a game-changer in the way developers interact with an API. They complement GraphQL's default non-subscription behavior in a way that both synchronous HTTP request/response communication and asynchronous event-driven interactions are available from a single API...