This ProgrammableWeb series on getting the most ROI out of your API walks you through all the major decisions that you need to make when building a successful API strategy for a business, government agency, not-for-profit, or enterprise. So far, our focus has recognized that APIs are a business concern. APIs enable businesses to optimize internal efficiencies, reach new customers, and continue to grow. A successful API strategy must be in line with the overall business vision, take a customer-centric approach to internal or external developers, have a business model that identifies the goals of the API strategy, and be willing to create all the components for product managing the API strategy.
Now that all of the business components are in place, building the API is now possible. The API strategy moves to becoming more of a technical concern for several steps as you resolve API design, security, and API Ops.
What Does My Reference Architecture Look Like?
You can take a number of different reference architecture approaches with APIs. For the most part, these involve a configuration of:
- On-premise, cloud-based, or a hybrid mix of servers that host databases and back-end server applications
- In some cases, having a messaging server and enterprise service bus (ESB) is necessary
- Authentication and identity management services
- An API gateway
- Frontend clients
However, before starting to decide on what API architectural style to choose, mapping your current overall reference architecture is often useful. Take a minute to map out your current infrastructure stack. Take note of what programming languages and frameworks your development team are most comfortable using when building solutions. Some of this wider infrastructure arrangement may influence your API technical decisions.
What Architecture Style Do I Use?
James Higginbotham, author of A Practical Approach to API Design and founder of API consultancy LaunchAny, says a crucial piece of the training puzzle is missing for many API development teams — the need for a matrix that lays out all of the various API architecture styles and that discusses which style is best for which situation.
"There is a massive disconnect between how we define, how we implement, and how we drive usage of APIs. We need a picture painted of the convergence of where we are aiming. It is only then that you can choose an API specification format, for example," says Higginbotham. An API specification format is a way to describe an API and record the metadata of the API. AN API specification format allows an API to be described in a machine readable way and can become the single source of truth so that as the API is built, it can be tested against the specification file to make sure that it still delivers the functionality and capabilities that were originally intended. (See the next section for a discussion on how to choose an API Specification format.)
"For things like resource-based access, you would normally need to build REST APIs," said Higginbotham. Resource-based access is a software development approach that defines permissions and access constraints per resource. Instead of setting roles for users that are universal across an API, the idea is that instead each resource has its own access control definitions. This allows much finer-grain control setting of business capabilities and data that are being exposed via an API, and allows developers to set clear boundaries around who has access to specific API functionalities. Higginbotham continued, "It's like saying, I will expose these nouns and create a lifecycle around them. For hypermedia, I have additional questions; for example, am I representing a workflow in which case I need [Hypermedia as the Engine of Application State] (HATEOAS)? Or am I using hypermedia so I can capture links and embed related resources by way of linked data, like with HAL or the W3C-ratified JSON-LD standard? Or will I go down the JSON-API or UBER or Collection-JSON or Siren route that offers me more compatibilities as they are driving full workflow state that needs more support for the client?" he says.
Jakub Nešetřil
Jakub Nešetřil, co-founder and CEO at API design tool Apiary (recently acquired by Oracle), says before choosing which architectural style to use, designers must first commit to a contract-based approach. Like a legal contract, an API's contract refers to the specific technical details that both the API provider and the API consumer adhere to. For example, the exact URL and parameters that a developer must provide in order to retrieve certain data or functionality. Both the developer and provider are on the hook to stick to the specifics of the contract.
Nešetřil points to numerous enterprises who are moving to a microservices and API architecture and says that in order to be successful, they need clearly defined contracts. Without that, says Nešetřil, managing a distributed teams organization is impossible. When a contract approach has been made, this may also help identify which architectural style is most suited to your API, in line with Higginbotham's comments about resources, workflow, and hypermedia needs.
"Contract-driven development is important," urges Nešetřil. "You need clear language around boundaried contexts. The definition of the contract is the first thing you do in the workflow and then you drive the development through the contract. This gives all stakeholders a clear language around the API in unambiguous terms. Without this, any collaboration is imprecise," he says.
Nešetřil says a second benefit of starting with the contract is that it can drive automation and enable the use of continuous integration, document auto-generation, and testing tools.
"An essential part of the workflow is testing. Without a testing framework, all the contract work is an architect's pipedream that can live up in the cloud but never exists in reality," says Nešetřil, whose Apiary product offers a code editor to assist with writing contracts. This also leads to automatically generating a mock server that encourages API consumers to co-edit the specification of the document "so the backend team can work on a draft of the API, while in parallel, the frontend team can be working on it as well. This way, you have the tools available that can help you review all the changes that have been made to the specification. It is a very powerful collaboration workflow," he says.
Marc MacLeod, founder and CEO at API lifecycle tool Stoplight.io, agrees. "We recommend that you put in a full contract test. Your documentation is basically saying 'this is what your API is.' You need to make sure that your API takes that input test and that your docs are always in sync with your API's response."
A to Z of API Architecture Types
Wikipedia offers the following definitions for some of the more common and emerging choices when deciding on an API architectural style (some definitions may have been updated according to ProgrammableWeb's view). Whereas some of the choices below are most definitely architectural styles of APIs, others are technologies that conform to one or more architectural styles:
RPC: In distributed computing a remote procedure call (RPC) is when a computer program causes a procedure (subroutine) to execute in another address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. RPC as an architectural style rode a wave of popularity when it turned up as a part of XML-RPC, which is one of the precursor integration technologies to the API economy. XML-RPC eventually evolved into SOAP, and many current APIs are said to be have a XML/SOAP as their architectural style. However, fundamentally, it's still an RPC-based architecture and the same can be said of gRPC, which is a newer RPC-based protocol geared towards performance.
SOAP: Although many people who choose to work with SOAP view it as an architectural choice, in ProgrammableWeb's view, the Simple Object Access Protocol (SOAP) is not an architectural style, but rather just a protocol specification for exchanging structured information in the implementation of web services in computer networks. Its purpose is to induce extensibility, neutrality, and independence and architecturally is most closely associated with the ideas of Web services and Services Oriented Architecture (often referred to as "SOA" ... pronounced "so-ahh") which in turn, predominantly infers an RPC-based architectural style. In fact, the predecessor to SOAP-based Web services was called XML-RPC.
SOAP uses XML Information Set for its message format and relies on application layer protocols, most often Hypertext Transfer Protocol (HTTP) or Simple Mail Transfer Protocol (SMTP), for message negotiation and transmission.
SOAP and the description specification that most commonly went with it — WSDL or Web Services Description Language — were widely regarded as de facto standards for integrating systems until the REST-styled architecture was discovered and paired with technologies like JSON and the OpenAPI specification.
gRPC: gRPC is an open source RPC system initially developed at Google. It uses HTTP/2 for transport, uses Protocol Buffers as the interface description language, and provides features such as authentication, bidirectional streaming and flow control, blocking or nonblocking bindings, and cancellation and timeouts. It generates cross-platform client and server bindings for many languages.
GraphQL: GraphQL is a data query language developed internally by Facebook in 2012 before being publicly released in 2015. It provides an alternative to REST and ad-hoc web service architectures. Although Facebook doesn't offer any public GraphQL endpoints, several other API providers, including GitHub, have either launched public endpoints or intend to.
More information:
HATEOAS: HATEOAS is a constraint of the REST application architecture that distinguishes it from most other network application architectures. The principle is that a client interacts with a network application entirely through hypermedia (e.g. hyperlinks) provided dynamically by application servers. A REST client needs no prior knowledge about how to interact with any particular application or server beyond a generic understanding of hypermedia. For example, when an application calls the root URL (endpoint) of a web-based API (i.e., http://acme.com/api), the response could include a list of top-level resources that are available from that API as well as the URL to call those resources. That list could change based on the underlying data, thereby providing the source (or being the "engine") of the application state. By contrast, in some service-oriented architectures (SOA), clients and servers interact through a fixed interface shared through documentation or an interface description language (IDL), such as WADL (or Web Application Description Language, see the section below on API specification formats).
Hypermedia: James Higginbotham points out that while HATEOAS APIs need hypermedia, not all hypermedia APIs need to be HATEOAS. In other words, the dynamism of hypermedia isn't just about driving application state. He suggests some hypermedia APIs may be more focused on embedding links and references to additional resources within their responses rather than helping manage a workflow or drive an application's state. This is suggestive of a more graph-like use case that can work within and across domains.
REST: Representational State Transfer (REST) or RESTful services are one way of providing interoperability between computer systems on a network, including the internet. Though the web's protocol (HTTP) as a transport mechanism isn't a technical requirement, the grand majority of REST implementations rely on the web. REST-compliant web services allow requesting systems to access and manipulate textual representations of web-based resources using a uniform and predefined set of stateless operations. Other forms of web services exist, which expose their own arbitrary sets of operations, such as web application description language (WSDL) and SOAP.
Push/Streaming: Whereas most API architectural styles discussed so far primarily involve a workflow such that a client (the API consumer) calls a server (the API endpoint) and the server responds accordingly, this request-and-response format is not well suited to scenarios involving data that changes in real-time. For example, a stock ticker. To stay updated, the client side would have to make an extraordinary number of requests over short periods of time which would be inefficient in terms of the required compute and network resources.
The Push-styled API architecture solves this problem by essentially reversing the flow in such a way that the API client keeps a special channel open for the API server to push updates to. So, as opposed to the API client constantly asking if there are any updates, the API server just pushes updated data to the client as that data becomes available. Twitter's infamous streaming API, otherwise known as its "firehose," is a good example. Currently, the two web API technologies most commonly associated with the push/streaming architectural style are known as Webhooks and Websockets.
The push/streaming architecture is often discussed in the context of publish-and-subscribe technologies (where clients subscribe to topics) and more recently, in the context of event-driven systems (the change of a stock ticker's price would be viewed as an event) where the occurence of an event triggers a data push or some other API-related process. Microservices and ephemeral serverless functions (sometimes based on Functions-as-a-Service technologies like Amazon's Lambda), both of which are typically wrapped in APIs, are examples of processes that might be triggered by an event.
As interest in event-driven business processes gains steam, more solutions for enabling them are coming to market. For example, the Kafka distributed streaming platform from the Apache Software Foundation is currently attracting a lot of attention thanks in part to its performance, reliability (both driven by it's clustering capability) and the availability of a wide array of SDKs for subscribing to and consuming event-driven data.
More information:
Which Specification Format Do I Use?
Most API providers will want to standardize on an API specification format — a means for describing an API and its contract in both human and machine-readable form — before API implementation begins. You have several to choose from. Once machines can parse through the definition of an API, several downstream artifacts ranging from documentation to API testing harnesses can be generated automatically.
While Jérôme Louvel, CEO of API and software development tooling company Restlet, also encourages a contract approach, he says that understanding the functionality first will help people decide on which API specification format is most suited to the task. "My recommendation would be to make this type of decision not too early in the project, as you get too caught up with technical decisions too early," suggests Louvel. "So when you have the requirements and know what type of data or service you want to expose, we recommend thinking about what a web API is and what the key elements are, and design at this level first, and then you need to derive your API contract in [a specification format, assisted by] some tooling. It is more important to decide on your functionality because that is when you are ready to look at the pros and cons of each specification format."
Choosing an API specification format can be a matter of personal developer choice and often boils down to the developer team preferences. Some specification formats that are more mature than others have a greater range of associated tooling and libraries available to help developers build APIs.
Daniel Cerecedo
Daniel Cerecedo is founder of the API consultancy, Byteflair. He has worked previously at IBM as an IT architect helping telcos and banks to deliver digital experiences. With his extensive background in API architecture with IBM, startups, and businesses, his teams tend to lean towards Java programming languages. "We always develop APIs using hypermedia because the frameworks we use are based on Spring and Spring HATEOAS out of the box," explains Cerecedo. "We usually go for that. Right now, if I were to choose a standard REST specification, I think I would choose [RESTful API Modeling Language (RAML)] because it is the most mature specification language. It allows for externalizing parts of the configuration; for example, if I have schemas in external files, I can reference that from the core files," says Cerecedo. "That said, there are new trends in the market that are changing and may change our approach in the future. For example, there is a really good presentation by Oliver Gierke on Spring REST and he argues that the next evolution of REST will get close to RPC, and even mix in the concept of event sourcing as future APIs will be more business-operation interested. That is something that we need to consider," he says.
"The second consideration for the future is GraphQL. But I think there are still concerns around security: How are you going to have the current granularity of permissions at the endpoint level? And as a new technology, there is not a mature ecosystem yet supporting GraphQL, although it is growing quickly," said Cerecedo, speaking of the emergent architectural style for which no specification format currently exists. "For GraphQL to become mainstream, there needs to be a lot of adoption in the industry and support in terms of coding libraries and that is something we still don't have. Regarding GraphQL, I believe we we will see more proxies being built on top of existing APIs. I completely see the value of GraphQL but as for implementing a GraphQL API right now, well, that can be tough if your security requirements are very specific," he says.
A to Z of API Specification Formats
The role of API specification formats is to explain to API consumers what the API is for and how you can use it in both a human-readable and machine-readable way. That means API consumer developers can quickly scan the specification and understand what an API can and cannot do, and programs can integrate the API into lifecycle tooling (such as documentation auto-generation, testing processes, CI/CD pipelines, SDK generation, or discovery channels) by using the specification format.
API Blueprint
API Blueprint is an open source specification that takes an API design first approach to creating an API. It acts as a representation of the contract for an API. It's used as the core specification format in Apiary's lifecycle tooling (although Apiary can import other formats, too). Oracle recently acquired Apiary.
Collection+JSON
Collection+JSON is a JSON-based read/write hypermedia-type designed to support management and querying of simple collections.
GraphQL Schema
Unlike REST APIs, GraphQL does not make use of an API specification format file to describe the API. Instead, a schema of the complete data model is used as the key file to explain the use case for the API and how to query it.
HAL
HAL is a lean hypermedia format aimed at providing a consistent and easy way to hyperlink between resources in an API.
Hydra
Hydra is a fairly mature API format with a range of tooling available to help developers design APIs hypermedia APIs, in a way that enables smarter clients.
JSON-API
JSON API is a specification for how a client should request that resources be fetched or modified, and how a server should respond to those requests. The JSON API website says the specification has been built in a way to reduce the number of requests and responses required to be transmitted between client and server "without compromising readability, flexibility, or discoverability."
JSON-LD
JSON-LD is a hypermedia format that uses linked data within JSON responses to create a network of standards-based machine interpretable data so that an application can "follow the links" in returned responses to other pieces of relevant data it might need to complete a query.
Open API Specification (formerly known as Swagger)
In mid 2017, the OpenAPI Initiative (the OAI) published version 3.0 of the OpenAPI Specification (OAS) and the entire API industry rapidly moved to to embrace it as the standard specification for expressing the description of a RESTful API. Using OAS, APIs can be described using JSON or YAML and once an API is described according to the OAS, there are many potential downstream benefits including but not limited to the automatic generation of API documentation, the provisioning of interactive API consoles and mocking services, and the generation of API test harnesses.
RAML
The open source Restful API Modeling Language is a technology that operates at a layer above the OpenAPI Specification and that focuses on the activity of modeling APIs — something that's normally done during an API's design phase. Once modeling and design is finished, API architects can use the open sourced API Modeling Framework (AMF) to turn the RAML-modeled API into an OAS-described API to gain all the downstream benefits of working with OAS.
Siren
Siren is a hypermedia specification for representing entities. As HTML is used for visually representing documents on a website, Siren is a specification for presenting entities via a web API. Siren offers structures to communicate information about entities, actions for executing state transitions, and links for client navigation.
Spring HATEOAS
Spring HATEOAS aims to ease creating REST representations that follow the HATEOAS principle. The core problem it tries to address is link creation and representation assembly.
UBER
Uber stands for Uniform Basis for Exchanging Representations and was designed by Mike Amundsen and Irakli Nadareishvili. It's designed to be a message format and not a source code format. It's optimized for cross-format use (XML and JSON), and it's not optimized to be serialized directly into source code or data objects.
WADL
WADL stands for Web Application Description Language and is a machine-readable XML description of HTTP-based web services. Technically an Interface Description Language, it models the resources provided by a service and the relationships between services. It's considered the REST version of WSDL.
WSDL
WSDL is used with SOAP and is an XML-based definition language that describes services as collections of network endpoints.
In our next chapter, we look at API tooling and how to create your API stack.