Understanding the less popular push/streaming protocols (XMPP, CoAP, MQTT, etc.)

XMPP was originally used to implement a real time chat strategy named In-Band Real Time Text . However, it has moved beyond that phase in recent years. Today you can find XMPP used in a variety of servers , clients , and libraries . In fact, you may be surprised at just how many apps, such as Apple Messages and Mozilla Thunderbird, rely on XMPP. In addition, XMPP powers many emerging technology, including the Internet of Things (IoT), as described in the next section of this part. The things that define XMPP are that it's:

• Extensible
• Real-time Oriented
• Status Aware
• Standards Oriented

The thing that makes XMPP so fast is that it pushes messages from sender to recipient in real time without any form of polling taking place. Of course, this could mean a loss of data if the recipient isn't available to receive the data, which is where the presence part of the abbreviation comes into play. A recipient provides status information: offline, online, or busy so that the sender knows when to send messages. These status messages are broadcast as the recipient status changes so that every node on the network knows the current recipient status at all times.

You use XMPP for real-time data transfer. The messages are all in XML and you can create new message types as needed to fit specific situations. Consequently, it's possible to extend XMPP to support any messaging scenario as long as the messaging occurs in a real-time environment and all the parties involved can handle XML data. (Some recent work with a few public web services, such as AWS, shows that the move to JSON has definitely picked up speed.) XML transfers occur in stanzas, where each stanza is a level 1 element. The server streams the XML stanzas to the recipient. Many people compare the XMPP to the same strategy used by technologies such as Comet (see below for more on Comet).

As with many other current push strategies, XMPP is bidirectional, which means both client and server can push messages. XMPP is non-blocking; it relies on an event-driven environment. In addition, even though XMPP doesn't directly use REST methods, it does have IQ stanzas that support REST-like semantics, making it easier to learn. XMPP relies on TLS to provide secure communication and SASL to provide flexible authentication.

One of the most interesting XMPP extensions is XMPP-IoT . You could use a technology like this to allow devices to send update information to other devices, such as the browser on a smartphone. By making the push strategy viable, it's possible to keep informed about the growing list of IoT devices you depend upon without having to poll each device individually several times a day.

However, just checking device status doesn't really make full use of XMPP. Because messaging happens in real-time using an efficient messaging strategy, it's possible to obtain constant device feedback. In addition, the bidirectional nature of the data transmission means that you could replace custom device interfaces with a smartphone or other equivalent. A simple connect sequence to a device makes it possible to obtain status, perform corrections to device configuration, and receive the results of those corrections all in real-time.

Of course, the use of the term device has also become fuzzy. A device need not be physical in nature. You could use this strategy to interact with soft devices as well; those without any physicality at all. The XMPP-IoT tutorial provides a lot more in the way of insights for this interesting extension technology.

MQTT is a machine-to-machine (M2M) IoT push technology. This protocol uses a publish/subscribe model. The main focus is to provide data transfer across low speed networks in resource constrained environments. As a consequence, you often see MQTT used in remote environments where network bandwidth is at a premium and with small, less capable devices. The healthcare industry commonly uses this technology to provide in-home connectivity between patient devices and a hospital. The in-home devices push data to the hospital so that a doctor or other professionals can constantly monitor patient condition without having to place the patient in a nursing home or hospital. The list of projects on the MQTT site is smallish, but it does provide you with some ideas of how the technology currently sees use.

As with many protocols in this part, MQTT is open source and freely available for everyone to use. It doesn't support bidirectional communication. In order to create two-way communication, you must establish two connections and also provide two different topics for these connections.

A potential problem with MQTT is that it doesn't require security. Given that MQTT is often used with Supervisory Control and Data Acquisition (SCADA) applications, the lack of required security could mean unfortunate breaches. It relies on two separate ports for communication, one of which is an SSL port. Unless you use SSL, any authentication information appears in clear text. MQTT doesn't support data encryption, you must add this feature using a third party product.

Comet doesn't actually refer to a single technology or a single method of creating a push strategy. What Comet does do is define a technique for developing event-driven apps that can send data from the server to the client without the client having to request it. All of the methodologies have one thing in common; they rely on native browser technologies, such as JavaScript, rather than using proprietary technologies to get the job done. Consequently, when using Comet techniques, you don't have to worry as much about the client involved because the client is likely to already support any requirements needed to make the push technology work. The two common forms of Comet techniques are:

• Streaming: The browser opens a single, constant, connection to the server and waits for Comet events to arrive. All event processing occurs in the browser. Examples of this approach are Hidden IFrame and XMLHttpRequest Server Push.
• Long Polling: The browser makes an AJAX request to the server. The server keeps the connection open until such time as it has new data for the browser to process. After sending the data, the browser processes it and then makes another AJAX request to the server.

Of the two, the Long Polling approach is the most often used because it overcomes problems with the streaming approach such as an inability to track request calling process state and issues with error handling. In addition, some streaming approaches are only supported on specific browsers, giving them limited appeal.

Using Long Polling as a strategy overcomes a number of issues. The two most important issues are stale data and network traffic. When applications cache data locally to reduce server load, there is a potential for the data becoming stale and the user seeing less than accurate output as a result. However, constantly polling to verify that the data is current causes an increase in network traffic. By allowing a browser to make a request and then wait until the server has an update, it becomes possible for a browser to continue using cached data without having to make constant update requests.

CoAP is a REST-based protocol that uses all of the methods you expect: GET, POST, and so on. An advantage of CoAP is that it's standards-based and freely available, just like HTTP. The emphasis is on communication with devices that have constrained hardware with few resources. Unlike most other entries in this part, CoAP doesn't require a central server, but instead allows communication between peers. Both CoAP and HTTP can be mapped on REST design principles, so it shouldn't be surprising to find that the coding of CoAP looks surprisingly the same as HTTP, even though the underlying implementation is significantly different.

The differences between CoAP and traditional strategies are many. For example, unlike the ASCII-encoded headers used with HTTP, CoAP relies on binary-encoded headers to reduce message size. CoAP also relies on UDP, rather than TCP, which is used by many other strategies. The use of UDP means that it can't guarantee message delivery. However, given that these are constrained devices, UDP also reduces network resource requirements and makes delivery faster. The CoAP messaging layer does address message duplication and fragmentation concerns.

Creating a publish/subscribe configuration using CoAP usually requires use of a broker . When working with a server, communication goes from the server, through a gateway, through the broker, and then to the individual devices that support CoAP. However, some implementations place a mapper between the HTTP-side of the communication and the CoAP-side of the communication. The point is that you can't perform the task using a straight connection due to all of the differences between HTTP/REST and CoAP.

Unlike many other protocols in use today, CoAP emphasizes strong, required security that actually does work with constrained devices. The default DTLS parameter settings for CoAP provide an equivalent of a 3072-bit RSA key, making hacking less likely. (Of course, anyone who is determined enough will break any security that you provide, but delaying them is always a good idea.)

The push technology you use depends a lot on the effects you want to achieve. In the case of WASTE (which was named after the postal service in a book by Thomas Pynchon entitled, The Crying of Lot 49) you have a decentralized network of peers. This is a secure network where each member must provide an RSA public key to obtain access. Every member can push data to every other member. Consequently, you can create a collaborative environment in which individuals of the group need not know each other, but can still be confident of the security of the configuration. The services currently provided by WASTE include:

• Instant Messaging (with presence)
• Group Chat
• File browsing/searching
• File transfer (upload and download)

The mesh network supported by WASTE depends on having sufficient nodes route traffic to ensure consistent and reliable data transfer. The higher the proportion of routers, the larger the network can become. Because WASTE is open source and free, you see it used in all sorts of places including educational institutions, which are only too happy to provide detailed setup instructions.

The idea behind Socket is to provide the means to automatically process pushed data, where the idea of pushing can come in many forms, including things like email. For example, when working with forms, you incorporate Socket into the web form and then use it as a hosted Webhook URL. The product makes it possible to forward data anywhere you need it.

As another example, imagine being able to automatically handle some types of email. The email demonstration shows how marking entries in a sample email makes it possible to allow Socket to perform processing on it and handle it automatically. In fact, you can find use cases for this product in these areas:

• Mail Parsing
• Hosted Webhook URL
• Forms Webhook Integration
• API Integration
• WordPress Contact Form 7 Processing (this use relies on the Powerup CF7 plugin)
• Mobile Apps Localized Processing
• Google Sheet Webhooks

Some automatic processing scenarios can become complex. When working with Google Sheets the process works at four different levels: new data row entry, data row updates, data sheet forwarding (either whole or partial), and timed events. In all four cases, Socket monitors the sheet and generates events based on user activity or timed events using a Webhook. The ability to incorporate several levels of processing means that Socket makes it possible to perform a lot of work with only a little coding on your part.

This is part four of our series on push technologies. In part five we will look at a number of turnkey push API infrastructure providers.