Control and Manage LLM APIs with Anypoint Flex Gateway

  • Overview
  • Resources
  • Transcript
Sue Siao
Technical Product Marketing Manager, Salesforce


Technology is moving fast and the recent introduction of AI innovation is exciting, especially with the promise of increased productivity. However, the innovation comes with overall security risks of accessing data across various systems and ensuring that private data doesn't end up in a public large language model for others to potentially access.

Learn how you can manage your LLM APIs integrated into your applications using Anypoint Flex Gateway and Custom Policies with PII Detection and API Key Management.


Many of us are excited about A I and


its potential to improve our systems


ways of working and business processes.


This means that A I is about to be integrated


with various business critical systems


with sensitive information


and this poses several challenges for it.


Teams such as fragmented


data across growing applications which


leads to missing context,


una actionable A I insights, uncertainty


and what will happen to the data center LLM


and risk to data security.


In this video, we will cover how it teams


can ensure that these innovations with A I


do not end up with confidential data leaks.


Large language models or LL MS



are mainly accessed through API


S. This is what allows them to be integrated


with various systems and can be used by end




and API S are best managed by an


API gateway. Most commercial API


gateways comes with out of box policies


for authentication contracts and rate limits.


However, this does not account for


data present in the prompts


to ensure that only selective data gets sent


to LL MS through the prompts. We recommend


that first users authenticate through


the gateway.


Two limit the usage to reduce


unnecessary costs. Three manage


API keys with the LLM vendor




four check if the data sent


to the LLM are compliant before


sending the request to the large language model.


Any point Flex gateway from Mulesoft is


an ultra-fast lightweight API gateway


to control and secure any API deployed




In addition to out of the box policies, Flex


gateway allows your team to create custom


policies so that you can extend the gateways capability.


Let's see how any point Flex gateway can


control access and usage of LLM


API S with a combination of out of the box


and custom policies


within any point platform. We have API


manager where we can go ahead and control


and manage API S


here. We have open A I API


added to Flex gateway


within API manager. We can set up SL


A tiers contracts, alerts


and also set up policies


all through one user interface.


We already have authentication and rate


limit policies added which are


out of the box, but we also want



to have data protection in API key




So before we start adding more


policies, let's make a test call


to the API.


As you could see, we weren't able to


make a request to open A I API and


it's because the user doesn't have


the API key information for open


A I. So now


let's go back and add the


API key management, custom policy.


So within API manager,


you can choose to add different policies.


And Flex gateway already comes with out


of the box policies for authentication


rate limit


Heather injection and so much more.


But you can also develop your own custom




The API key management custom policy


sends the API key to the LLM API


so that the end user can have access


without actually needing the key.


This means that your organization can


centrally manage API keys and


the user will still be able to interact with the


LLM API without any hiccup.


OK? Now that the policy is enabled,


let's make another test call.


So here we see the response but


personally identifiable information


or PI I was sent to the OM. And


we definitely don't want that. So we need a way


to be able to block these calls with sensitive




And by using our another custom policy


for PI I detection,


this custom policy checks for pi I


imprompt and contact sent to the


LM API using an open source


data protection. And the ISDK


when private information such as credit card


numbers, names or driver's license


information is part of the prompt or the context,


the policy detects them and lets the user


know it will also block the prompt from


reaching the LLM API in the first




If there is no sensitive data, the gateway


will pass the request to the LLM API.


So now let's see how all these policies


work together by making a test call.


The gateway responds that the request body includes


sensitive data and rejects the request


with a combination of out of the box policies


and custom policies. With any point Flex gateway,


you can proactively protect the data sent


to LM API S while allowing


your organization to innovate with A


I any point Flex gateway


is now able to control traffic to the LMM


API through authentication rate limit


API key management and PI I detection


with a combination of out of the box policies


and custom policies with any point Flex gateway.


Your it team can proactively


protect the data center to LM API S


while allowing your organization


to innovate with A I.

Start free trial