Control and Manage LLM APIs with Anypoint Flex Gateway

[00:00:07.368]

Many of us are excited about A I and

[00:00:09.569]

its potential to improve our systems

[00:00:11.608]

ways of working and business processes.

[00:00:14.398]

This means that A I is about to be integrated

[00:00:16.879]

with various business critical systems

[00:00:18.920]

with sensitive information

[00:00:20.969]

and this poses several challenges for it.

[00:00:23.109]

Teams such as fragmented

[00:00:25.149]

data across growing applications which

[00:00:27.440]

leads to missing context,

[00:00:29.219]

una actionable A I insights, uncertainty

[00:00:32.189]

and what will happen to the data center LLM

[00:00:34.679]

and risk to data security.

[00:00:37.719]

In this video, we will cover how it teams

[00:00:40.029]

can ensure that these innovations with A I

[00:00:42.158]

do not end up with confidential data leaks.

[00:00:45.609]

Large language models or LL MS

[00:00:47.668]

are mainly accessed through API

[00:00:49.709]

S. This is what allows them to be integrated

[00:00:52.000]

with various systems and can be used by end

[00:00:54.189]

users

[00:00:55.880]

and API S are best managed by an

[00:00:57.950]

API gateway. Most commercial API

[00:01:00.240]

gateways comes with out of box policies

[00:01:02.450]

for authentication contracts and rate limits.

[00:01:05.209]

However, this does not account for

[00:01:07.430]

data present in the prompts

[00:01:10.239]

to ensure that only selective data gets sent

[00:01:12.359]

to LL MS through the prompts. We recommend

[00:01:14.500]

that first users authenticate through

[00:01:16.829]

the gateway.

[00:01:17.859]

Two limit the usage to reduce

[00:01:20.069]

unnecessary costs. Three manage

[00:01:22.418]

API keys with the LLM vendor

[00:01:24.519]

centrally

[00:01:26.028]

four check if the data sent

[00:01:28.069]

to the LLM are compliant before

[00:01:30.299]

sending the request to the large language model.

[00:01:33.370]

Any point Flex gateway from Mulesoft is

[00:01:35.409]

an ultra-fast lightweight API gateway

[00:01:37.588]

to control and secure any API deployed

[00:01:40.000]

anywhere.

[00:01:41.138]

In addition to out of the box policies, Flex

[00:01:43.489]

gateway allows your team to create custom

[00:01:45.569]

policies so that you can extend the gateways capability.

[00:01:48.888]

Let's see how any point Flex gateway can

[00:01:50.989]

control access and usage of LLM

[00:01:53.219]

API S with a combination of out of the box

[00:01:55.750]

and custom policies

[00:01:58.668]

within any point platform. We have API

[00:02:00.790]

manager where we can go ahead and control

[00:02:03.400]

and manage API S

[00:02:05.129]

here. We have open A I API

[00:02:07.659]

added to Flex gateway

[00:02:09.849]

within API manager. We can set up SL

[00:02:12.099]

A tiers contracts, alerts

[00:02:14.460]

and also set up policies

[00:02:16.500]

all through one user interface.

[00:02:19.300]

We already have authentication and rate

[00:02:21.528]

limit policies added which are

[00:02:23.618]

out of the box, but we also want

[00:02:25.770]

to have data protection in API key

[00:02:27.939]

management.

[00:02:30.058]

So before we start adding more

[00:02:32.419]

policies, let's make a test call

[00:02:34.469]

to the API.

[00:02:37.719]

As you could see, we weren't able to

[00:02:39.770]

make a request to open A I API and

[00:02:41.868]

it's because the user doesn't have

[00:02:44.159]

the API key information for open

[00:02:46.479]

A I. So now

[00:02:48.770]

let's go back and add the

[00:02:50.808]

API key management, custom policy.

[00:02:53.949]

So within API manager,

[00:02:55.960]

you can choose to add different policies.

[00:02:58.800]

And Flex gateway already comes with out

[00:03:00.960]

of the box policies for authentication

[00:03:03.740]

rate limit

[00:03:05.129]

Heather injection and so much more.

[00:03:07.659]

But you can also develop your own custom

[00:03:09.990]

policies.

[00:03:11.629]

The API key management custom policy

[00:03:14.058]

sends the API key to the LLM API

[00:03:16.740]

so that the end user can have access

[00:03:19.240]

without actually needing the key.

[00:03:22.110]

This means that your organization can

[00:03:24.159]

centrally manage API keys and

[00:03:26.278]

the user will still be able to interact with the

[00:03:28.319]

LLM API without any hiccup.

[00:03:31.118]

OK? Now that the policy is enabled,

[00:03:33.149]

let's make another test call.

[00:03:35.288]

So here we see the response but

[00:03:37.319]

personally identifiable information

[00:03:39.580]

or PI I was sent to the OM. And

[00:03:41.868]

we definitely don't want that. So we need a way

[00:03:44.319]

to be able to block these calls with sensitive

[00:03:46.520]

information.

[00:03:47.538]

And by using our another custom policy

[00:03:50.058]

for PI I detection,

[00:03:52.118]

this custom policy checks for pi I

[00:03:54.338]

imprompt and contact sent to the

[00:03:56.538]

LM API using an open source

[00:03:58.679]

data protection. And the ISDK

[00:04:02.330]

when private information such as credit card

[00:04:04.569]

numbers, names or driver's license

[00:04:06.649]

information is part of the prompt or the context,

[00:04:09.649]

the policy detects them and lets the user

[00:04:11.778]

know it will also block the prompt from

[00:04:13.879]

reaching the LLM API in the first

[00:04:15.889]

place.

[00:04:17.350]

If there is no sensitive data, the gateway

[00:04:19.500]

will pass the request to the LLM API.

[00:04:23.309]

So now let's see how all these policies

[00:04:25.709]

work together by making a test call.

[00:04:28.970]

The gateway responds that the request body includes

[00:04:31.290]

sensitive data and rejects the request

[00:04:34.350]

with a combination of out of the box policies

[00:04:36.730]

and custom policies. With any point Flex gateway,

[00:04:39.209]

you can proactively protect the data sent

[00:04:41.379]

to LM API S while allowing

[00:04:43.769]

your organization to innovate with A

[00:04:45.778]

I any point Flex gateway

[00:04:48.149]

is now able to control traffic to the LMM

[00:04:50.369]

API through authentication rate limit

[00:04:52.660]

API key management and PI I detection

[00:04:55.540]

with a combination of out of the box policies

[00:04:57.759]

and custom policies with any point Flex gateway.

[00:05:00.139]

Your it team can proactively

[00:05:02.149]

protect the data center to LM API S

[00:05:04.178]

while allowing your organization

[00:05:06.250]

to innovate with A I.