7 steps to improve API performance

The next step was to identify the bottlenecks that prevented us from getting to the 100 transactions per second. After analyzing the slower transactions and looking at each activity in the flow, we found a few problems.

• The API payload size for some requests was much larger due to more history of orders with detailed line items of each order and those transactions were taking comparatively longer.
• We were trying to retrieve multiple types of information from different databases, like customer demographics and order history, then consolidating them in one monolith operation synchronously.
• The customer demographic database had a bottleneck at 200 transactions per second based on the server size and was being shared among multiple applications. The database could only be scaled vertically by increasing the size of the server and also needed downtime which was not an easy task within the timeframe.
• The API was deployed on three workers/servers and could handle the number of transactions for regular load but were not able to scale to peak loads.
• The error transactions were taking much longer than successful transactions.

Looking at these problems, we had to make the following changes to the design for performance:

To solve the problem of relatively large response payloads we had to limit order history by adding pagination and provide JSON linking using hypermedia/Hateoas links for getting order line item details by making one more call with order ID as the key. This significantly reduced the payload size for customers with a long history of orders.

We broke the single monolithic API into multiple microservices by dividing them into customer system API to connect and access the customer demographic data. The order system API to connect and access the order database. The process API for orchestrating and transforming the data from customer and order APIs. The experience APIs called the process API and transformed the data needed on the mobile application based on the MuleSoft API-led connectivity approach. This provided capability to reuse the process and system APIs in the future as they were designed with the microservices approach and could be scaled separately as per demand.

We were able to call customer demographic system API and Order System API in parallel with the new approach by breaking them down into two separate APIs. This reduced the time taken as this was being done synchronously, one after the other, earlier.

We used connection pooling to connect to the database as a significant amount of time was being spent in creating and closing a database connection.

We noticed 30% of the requests were the same for customer demographics within a 15 minutes period. The customer demographics don’t change frequently so we decided to add a caching policy using the API Manager and reduced the capacity bottleneck of the Customer database by reducing the number of calls.

We deployed the APIs within auto-scaling groups where it could scale between two and five instances depending on normal and peak times of the day.

Error logging was changed to asynchronous which removed the wait time for logging.

With all these changes, we got to the target performance of 100 transactions per seconds. Had we kept the performance requirements in mind from the start, these changes could have been avoided. Going forward, you should consider performance as a critical requirement while designing an API and keep these best practices in mind.

As a next step, having a good proactive approach to performance testing and monitoring is a better option than working on them after encountering performance issues. It is very important to continuously monitor the APIs performance with MuleSoft’s Anypoint Monitoring which bridges the divide between application performance monitoring and log management, it is the de-facto monitoring tool for enterprise-grade visibility. Anypoint Monitoring gives granular visibility into all insights related to APIs, integrations, and dependencies that are extracted automatically, with very low-performance overhead and no required application modifications. More than 40 metrics are available through built-in dashboards, covering everything from inbound requests, outbound calls, and dependencies to performance, failures, JVM, and infrastructure metrics.