AI Gateway Mind Map: LiteLLM, Bifrost, Portkey, AgentCore Gateway, Azure API Management and Apigee, with Use Cases
One control point between your apps, agents and every model provider: why you need it, what it does, the open-source gateways you host yourself, the AWS, Azure and Google Cloud services, and the scenarios each one fits.
Why a gateway
One control point
- What it is
- An AI gateway is a proxy layer between your applications, agents and MCP clients on one side and the model providers and tools on the other. Instead of every service holding its own OpenAI, Anthropic, Gemini or Bedrock credentials, everything calls one endpoint that authenticates, routes, limits, caches and logs.
- When to use it
- As soon as more than one app or team calls models, or more than one provider is in play.
Key sprawl
Provider keys everywhere
- What it is
- Without a gateway each microservice, notebook and agent stores provider API keys; rotating one means redeploying many services, and a leaked key has no owner. A gateway keeps the provider keys in one place and hands out its own scoped keys.
- When to use it
- Security reviews that ask who can call which model with which key.
One API surface
OpenAI-compatible
- What it is
- Most gateways expose an OpenAI-compatible endpoint (some also the Anthropic Messages API) and translate to each provider's format, so switching a model is a change of model name, not of SDK.
- When to use it
- Teams that want to try a new model without touching application code.
Cost attribution
Spend per team and app
- What it is
- Provider invoices show one total. The gateway records tokens and cost per key, user, team or tag, so finance can charge back and engineering can see which feature burns the budget.
- When to use it
- Internal chargeback, per-customer billing in a SaaS, budget alerts.
Failover
Outages and 429s
- What it is
- Providers rate-limit (HTTP 429), degrade and retire models. The gateway retries, moves the request to another deployment, region or provider, and trips a circuit breaker on an unhealthy backend so apps keep answering.
- When to use it
- Customer-facing assistants with an availability target.
Governance and audit
Policies in one place
- What it is
- Which models are approved, which data may leave the network, what gets logged and for how long: a gateway enforces these once for every caller instead of trusting each team to implement them.
- When to use it
- Regulated companies and any AI policy that must be provable to an auditor.
Agent and MCP sprawl
Tools need a front door
- What it is
- Agents call MCP servers, internal APIs and other agents as well as models. Gateways now front that traffic too: one authenticated MCP endpoint, per-tool permissions, outbound credentials injected for each tool, and logs of every tool call.
- When to use it
- More than a handful of MCP servers, or agents that act on behalf of users.
- AI angle
- Semantic tool search keeps prompts small when an agent can reach hundreds of tools.
Not a secrets manager
Pair it with a vault
- What it is
- A gateway routes requests and applies access policies; it is not automatically where provider credentials should live. Keep them in HashiCorp Vault, AWS Secrets Manager or Azure Key Vault, or use managed identities, and let the gateway read them at start-up.
- When to use it
- Any production deployment: the gateway database should never be the only copy of a provider key.
Core capabilities
What a gateway does
- What it is
- The common feature set across open-source and cloud gateways: identity and keys, limits and budgets, routing and resilience, caching, safety, observability and tool proxying. Availability differs by product, version and paid tier.
- When to use it
- Use it as the checklist when comparing gateways.
Virtual keys
Scoped, revocable keys
- What it is
- The gateway issues its own keys, each limited to some models, a budget and rate limits, and owned by a user or team. Revoking one never touches the provider key behind it. In LiteLLM they need a master key and a Postgres database.
- When to use it
- One key per app, per environment and per developer.
- In code
curl http://localhost:4000/key/generate \ -H "Authorization: Bearer $LITELLM_MASTER_KEY" \ -d '{"models": ["smart"], "max_budget": 50, "budget_duration": "30d", "tpm_limit": 100000}'
Rate limits
Requests and tokens
- What it is
- Limits on requests per minute (RPM) and tokens per minute (TPM) per key, team or IP, so one app cannot drain the shared provider quota. Token-aware limits count prompt and completion tokens, not just calls.
- When to use it
- Several apps sharing one provisioned deployment or one provider account.
Token budgets
Spend caps per period
- What it is
- A spend or token cap per key, user or team over a period (daily, monthly). When it runs out the gateway rejects new requests; Vercel AI Gateway, for example, answers HTTP 402 and documents its budgets as soft caps.
- When to use it
- Hackathons, trials, coding agents for many developers, per-customer plans.
Routing and load balancing
Pick the backend
- What it is
- Spread traffic over several deployments, keys, regions or providers: round-robin, weighted, priority (use the prepaid capacity first) or latency-based, and in some gateways model routing that picks a model per prompt.
- When to use it
- Filling provisioned throughput before spilling to pay-as-you-go.
Retries and fallbacks
Survive provider errors
- What it is
- Bounded retries on timeouts and 429s, then a fallback chain to another deployment or model, with a circuit breaker that stops sending traffic to a failing backend for a while.
- When to use it
- Any user-facing path; keep the fallback model evaluated, not just configured.
Exact cache
Same request, no call
- What it is
- Identical requests are answered from a cache (often Redis) instead of the provider: zero tokens and millisecond latency. Different from provider prompt caching, which only discounts a repeated prompt prefix.
- When to use it
- Evaluation reruns, deterministic classification, repeated FAQ prompts.
Semantic cache
Similar prompt, reuse
- What it is
- The prompt is embedded and compared with earlier prompts; above a similarity threshold the stored completion is returned. Saves tokens on paraphrased questions but can return a wrong answer if the threshold is loose.
- When to use it
- High-volume support questions; never for user-specific or time-sensitive answers.
- AI angle
- Tune the threshold on real traffic and scope the cache per tenant.
Guardrails and PII
Filter in and out
- What it is
- Checks on prompts and responses at the gateway: prompt-injection and jailbreak detection, denied topics, content moderation, and redaction of personal data before it reaches the provider. Often delegated to Bedrock Guardrails, Azure AI Content Safety, Model Armor or a third-party guard.
- When to use it
- Chatbots open to the public, PII in support tickets, regulated advice.
Logs, metrics, traces
Tokens, latency, cost
- What it is
- Every request logged with model, provider, tokens, latency, status and cost; metrics exported to Prometheus, Azure Monitor or CloudWatch; OpenTelemetry spans so a model call shows up inside the trace of the request that caused it.
- When to use it
- Debugging slow answers, capacity planning, cost dashboards.
Model aliases
"smart", "fast", "cheap"
- What it is
- Apps ask for a logical name and the gateway maps it to a concrete provider model. Upgrading or replacing a model becomes a gateway config change, and a retired model never breaks an app.
- When to use it
- Model upgrades and A/B tests without redeploying apps.
- In code
model_list: - model_name: smart litellm_params: model: anthropic/<model-id> api_key: os.environ/ANTHROPIC_API_KEY - model_name: smart litellm_params: model: bedrock/<model-id>
MCP and A2A proxying
Tools behind the gateway
- What it is
- Gateways now expose MCP servers (and agent-to-agent APIs) through the same front door as models: one endpoint that aggregates tools, authenticates callers, applies per-tool access rules and logs each call. LiteLLM, Bifrost, Portkey, Kong, Azure API Management, Apigee and AgentCore Gateway all document it.
- When to use it
- Agents that need internal APIs as tools without each agent holding credentials.
Open source, self-hosted
Run it in your VPC
- What it is
- Gateways you deploy yourself (Docker, Kubernetes, a VM): prompts and logs stay in your network and you pay for compute, not per request. Most follow an open-core model, with SSO, advanced RBAC or some guardrails in a paid enterprise edition.
- When to use it
- Data that must not pass through a third-party gateway, multi-cloud, or many providers at once.
LiteLLM
Best general start
- What it is
- BerriAI's Python SDK and proxy server: 100+ LLM providers behind the OpenAI format, virtual keys, spend tracking per key, user and team, budgets and rate limits, router with retries and fallbacks, caching, guardrails, an admin UI, MCP and A2A gateways, and logging to Langfuse, MLflow and others. Enterprise features sit under a commercial licence.
- When to use it
- Python-heavy AI teams that want the widest provider coverage and a quick start.
- In code
docker run -p 4000:4000 -v ./config.yaml:/app/config.yaml \ ghcr.io/berriai/litellm:v1.83.0-stable --config /app/config.yaml- Key figures
- Vendor benchmark: 8 ms P95 latency at 1,000 requests per second.Source: LiteLLM README
Bifrost
Go, low overhead
- What it is
- Maxim's gateway written in Go under Apache 2.0: one OpenAI-compatible API over 23+ providers, automatic failover and load balancing across keys and providers, semantic caching, MCP support, virtual keys with budgets and rate limits, Prometheus metrics and tracing, and a web UI. A drop-in replacement for the OpenAI, Anthropic and Google GenAI SDKs by changing the base URL.
- When to use it
- High-throughput production paths where gateway overhead must stay in microseconds.
- In code
npx -y @maximhq/bifrost docker run -p 8080:8080 maximhq/bifrost- Key figures
- Vendor benchmark: 11 µs added per request at a sustained 5,000 requests per second on an AWS t3.xlarge.Source: Bifrost README
Portkey Gateway
TypeScript, edge-ready
- What it is
- Portkey's MIT-licensed gateway in TypeScript: routing to 1,600+ models across 45+ providers, fallbacks, up to five automatic retries, load balancing, timeouts, input and output guardrails, simple caching, and a separate MCP gateway. Runs with npx, Docker, Node.js or Cloudflare Workers; semantic caching and prompt management are in the hosted or enterprise product, which Portkey says is merging into open source with its 2.0 release.
- When to use it
- JavaScript stacks, edge deployments and teams that want guardrails built in.
- In code
npx @portkey-ai/gateway # http://localhost:8787/v1- Key figures
- Vendor claims: under 1 ms latency and a 122 KB footprint.Source: Portkey Gateway README
Agent Router (Envoy AI Gateway)
Kubernetes-native
- What it is
- The control plane for AI and agent traffic built on Envoy and Envoy Gateway, Apache 2.0. Called Envoy AI Gateway until September 2026, when it joined the Agentic AI Foundation as Agent Router with the same code, maintainers and CRD names. One OpenAI-compatible API for hosted providers, self-hosted inference and MCP servers; platform teams manage credentials, routing, quotas, failover and usage attribution.
- When to use it
- Platform teams already running Envoy or the Kubernetes Gateway API.
- In code
aigw run # OpenAI-compatible router on localhost:1975
Apache APISIX
API gateway + AI plugins
- What it is
- A high-performance API gateway whose ai-proxy plugin translates one config into each provider's format (OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, DeepSeek, OpenRouter, any OpenAI-compatible endpoint) and logs tokens and time to first token. Companion plugins: ai-proxy-multi, ai-cache, ai-rate-limiting, ai-prompt-guard, ai-rag and content moderation.
- When to use it
- Companies that already run APISIX for their REST APIs and want one gateway for both.
Kong AI Gateway
Kong + AI policies
- What it is
- Kong's AI layer: AI Model and Provider entities for routing, AI Consumer Groups for per-team token budgets, an AI MCP Server entity that exposes APIs as tools, and policies such as AI Prompt Guard, AI Semantic Cache, AI Sanitizer (PII), AI Rate Limiting Advanced and connectors to AWS Guardrails, Azure Content Safety and Model Armor. Check which plugins your Kong licence includes.
- When to use it
- Kong shops that want AI traffic under the same policies and portal as their APIs.
Self-hosted models behind it
vLLM, Ollama, SGLang
- What it is
- vLLM, SGLang and Ollama serve open-weight models on OpenAI-compatible endpoints, so the gateway treats them as one more provider: route sensitive traffic to the local model and the rest to a frontier API, behind the same keys and logs.
- When to use it
- Hybrid setups: private data on your GPUs, hard reasoning on a frontier API.
AWSBedrock + AgentCore
- What it is
- On AWS the gateway role is split: Amazon Bedrock gives one API, IAM and CloudTrail over many model families; Bedrock Guardrails filters content; AgentCore Gateway fronts tools, agents and model providers for agents; and AWS publishes a LiteLLM-based guidance for a self-managed multi-provider gateway.
- When to use it
- Workloads already on AWS that want IAM, VPC endpoints and CloudTrail for AI traffic.
Amazon BedrockMany models, one API
- What it is
- Managed access to foundation models from several vendors through one AWS API (Converse), with IAM permissions per model, private VPC endpoints, CloudTrail logging and per-account quotas. It is a model service, not a full gateway: budgets per team and non-AWS providers need something in front.
- When to use it
- The default model layer for AWS-native teams.
Cross-Region inference
Geographic or global
- What it is
- Inference profiles let Bedrock pick the Region that serves a request: a geographic profile keeps processing inside the US, EU or APAC, a global profile routes to any commercial Region. Traffic stays on the AWS network and CloudTrail records where each request ran.
- When to use it
- Burst capacity; geographic profiles when data residency applies.
- Key figures
- No extra routing cost; global profiles cost about 10% less than geographic ones.Source: AWS docs, Cross-Region inference
Bedrock Guardrails
Filters in and out
- What it is
- Configurable safeguards on prompts and responses: content filters (hate, insults, sexual, violence, misconduct, prompt attack), denied topics, word filters, PII blocking or masking, contextual grounding checks and Automated Reasoning checks. The ApplyGuardrail API runs them without invoking a Bedrock model.
- When to use it
- One safety policy reused by a gateway in front of any model, not only Bedrock ones.
AgentCore Gateway
Agents to tools, models
- What it is
- A fully managed AI gateway for agentic traffic: it turns APIs, Lambda functions and services into MCP tools, fronts other agents and HTTP services (including A2A), and routes inference across model providers through one model-based routing endpoint. It handles inbound auth (who the agent is) and outbound auth (credentials for each tool) and is serverless with built-in observability.
- When to use it
- AWS agents that need company APIs and SaaS tools without custom integration code.
Gateway targets
OpenAPI, Lambda, MCP
- What it is
- Targets can be Lambda functions, API Gateway REST APIs, OpenAPI specifications, Smithy models, existing MCP servers and built-in connectors (Salesforce, Slack, Jira, Asana, Zendesk). Clients see one consolidated tools/list across all targets.
- When to use it
- Publishing an existing REST API to agents from its OpenAPI spec.
Semantic tool search
Thousands of tools
- What it is
- Enabled when the gateway is created: a built-in search tool (x_amz_bedrock_agentcore_search) backed by a vector index of tool descriptions, so the agent asks for the tools that fit its task instead of loading every tool into the prompt.
- When to use it
- Agents with large tool catalogues where prompt size and latency grow with every tool.
- AI angle
- Fewer tools in context usually means better tool selection.
AgentCore Identity
Agent identity, tokens
- What it is
- Identity and credential management for agents: agent identities as workload identities, an inbound JWT authorizer for your IdP, credential providers that store OAuth tokens and API keys, and a consent flow so an agent can act on behalf of a user with audit trails.
- When to use it
- Agents that call Google, Microsoft or Salesforce APIs as the signed-in user.
Multi-provider gateway guidance
LiteLLM on ECS or EKS
- What it is
- AWS's published reference architecture deploys LiteLLM with Terraform on ECS Fargate or EKS behind an Application Load Balancer with AWS WAF, images in ECR, TLS from ACM, KMS encryption and CloudWatch: one OpenAI-style endpoint for Bedrock and other providers with per-team budgets.
- When to use it
- AWS shops that need budgets and non-Bedrock providers under one endpoint.
AzureAPIM AI gateway
- What it is
- The AI gateway in Azure API Management is a set of policies and features of the existing API gateway, not a separate product. It governs OpenAI-compatible, Anthropic Messages and Vertex AI APIs, models in Microsoft Foundry or other providers such as Amazon Bedrock, self-hosted models, MCP servers and A2A agent APIs. Availability varies by tier.
- When to use it
- Microsoft-centred companies that already publish APIs through API Management.
llm-token-limit
TPM and token quotas
- What it is
- A policy that enforces a tokens-per-minute limit or a token quota per hour, day, week, month or year on any counter key (subscription, IP or a custom expression), and can pre-count prompt tokens so an over-limit request never reaches the backend.
- When to use it
- Sharing one PTU deployment between apps without one starving the others.
- In code
<llm-token-limit counter-key="@(context.Subscription.Id)" tokens-per-minute="500" estimate-prompt-tokens="false" remaining-tokens-variable-name="remainingTokens" />
llm-emit-token-metric
Tokens per consumer
- What it is
- Emits token counts to Azure Monitor with custom dimensions (client IP, API, user ID from a header), plus logging of prompts and completions to Application Insights and a built-in dashboard of token consumption.
- When to use it
- Chargeback per department and auditing who sent what.
Semantic caching
Redis + embeddings
- What it is
- The llm-semantic-cache-store and llm-semantic-cache-lookup policies use an embeddings API and Azure Managed Redis (or another RediSearch-compatible cache) to reuse completions for semantically similar prompts.
- When to use it
- Internal assistants with many near-identical questions.
Load balancer and circuit breaker
PTU first, then PAYG
- What it is
- Backend pools with round-robin, weighted, priority and session-aware balancing, and a circuit breaker whose trip duration follows the backend's Retry-After header. Priorities let you fill provisioned (PTU) capacity before pay-as-you-go.
- When to use it
- Multi-region Azure OpenAI deployments with an availability target.
Content safety and identityNo keys in apps
- What it is
- Managed identities authenticate the gateway to Azure AI services without API keys, the credential manager handles OAuth for apps and agents calling APIs or MCP servers, and a policy moderates prompts with Azure AI Content Safety.
- When to use it
- Removing provider keys from application config entirely.
MCP servers in APIM
REST API to MCP
- What it is
- API Management can expose an existing REST API as an MCP server, pass through and govern an existing MCP server, and import A2A agent APIs; Azure API Center keeps an organisational catalogue of APIs, MCP servers and skills.
- When to use it
- Giving Copilot Studio or Foundry agents governed access to internal APIs.
Unified model API
Preview, one endpoint
- What it is
- A preview feature that exposes several LLM backends from different providers through one OpenAI-compatible endpoint, translating formats automatically and applying governance policies once across all models.
- When to use it
- New multi-provider setups on Azure.
AI gateway in Foundry
Preview, from the portal
- What it is
- An API Management instance attached to a Microsoft Foundry resource governs it from the Foundry portal: token quotas and rate limits for model deployments, registration of agents running anywhere, and governed MCP tools in the Foundry inventory.
- When to use it
- Foundry-first teams that do not want to manage APIM policies by hand.
Google CloudApigee + Model Armor
- What it is
- Apigee, Google's API management platform, carries the AI gateway features: token quotas, model routing, semantic caching, MCP security and Model Armor for prompt and response screening. Vertex AI can also call Model Armor directly.
- When to use it
- Google Cloud and Gemini workloads, or companies already on Apigee.
LLM token policies
LLMTokenQuota and more
- What it is
- LLMTokenQuota and PromptTokenLimit cap tokens per user or app (for example a daily maximum), and Apigee tracks token usage by model and app with latency, error rates and MCP tool performance.
- When to use it
- Token budgets per partner or app published through Apigee.
Model Armor
Screen prompts, responses
- What it is
- Google's screening service against prompt injection and harmful content. In Apigee the SanitizeUserPrompt and SanitizeModelResponse policies send traffic to Model Armor, which can allow, block or redact; Apigee and Model Armor must share a region, and Apigee needs a Comprehensive environment.
- When to use it
- Public-facing Gemini apps and any model behind Apigee.
Model routing and cache
Policy-driven
- What it is
- Dynamic model routing sends a request to a different model by use case through policy rules, and a semantic caching policy serves cached answers to similar queries.
- When to use it
- Sending easy requests to a smaller Gemini model and hard ones to a larger one.
APIs as MCP tools
API hub, OAuth
- What it is
- Apigee exposes APIs as MCP tools, secures MCP proxies with an OAuth v2 policy and JSON-RPC policies, and manages MCP servers, tools and agentic skills in API hub with natural-language search.
- When to use it
- Agent builders on Google Cloud reusing the company API catalogue.
Hosted gateways
No proxy to run
- What it is
- Gateways run by a vendor: one key, one endpoint, logs, budgets and failover without operating a proxy, database or control plane. Traffic passes through the vendor, so data handling terms matter.
- When to use it
- Start-ups and prototypes that want gateway features this week.
Cloudflare AI Gateway
Analytics, cache, limits
- What it is
- Available on all Cloudflare plans: analytics (requests, tokens, cost), logging, caching at Cloudflare's edge, rate limiting, retries and model fallback, and routing with model aliases and dynamic routes. Works with Workers AI, OpenAI, Anthropic, Gemini and others.
- When to use it
- Apps already behind Cloudflare that want visibility in one line of code.
Vercel AI Gateway
No markup, BYOK
- What it is
- A hosted gateway at ai-gateway.vercel.sh, callable from any cloud with the AI SDK, OpenAI or Anthropic clients: request logs, cost per request, budgets per team, project, key or member, prompt caching, provider and model fallbacks, and bring-your-own-key. Vercel states zero markup on provider token prices.
- When to use it
- TypeScript teams and coding agents that need many providers fast.
OpenRouter
Hundreds of models
- What it is
- One OpenAI-compatible API (base URL openrouter.ai/api/v1) to hundreds of models, with automatic provider fallbacks and routing to the most cost-effective option per request.
- When to use it
- Trying many open and closed models without separate accounts.
Hosted or self-hosted
The trade-off
- What it is
- Hosted: nothing to run, fastest start, but prompts pass through another company and features follow its roadmap. Self-hosted: you run Postgres, Redis and upgrades and own security patching, but data and logs stay in your VPC.
- When to use it
- Decide on data classification first, convenience second.
Reference architecture
Clients to providers
- What it is
- The layout recommended for a central AI platform serving many apps and agents: clients, then the gateway, then a security and governance layer, then cloud and self-hosted models, with observability across all of it.
- When to use it
- The starting diagram for an enterprise AI platform proposal.
Clients
Apps, agents, MCP clients
- What it is
- Web apps, backend services, agents, workflows and coding assistants: each gets a gateway key and a base URL, never a provider key.
- When to use it
- Onboarding a new team is issuing a key, not a procurement ticket.
Gateway layer
Auth, route, limit, log
- What it is
- Authentication, rate limits, model routing, fallbacks, cost tracking, caching, request logging and access policies, in one stateless tier that scales horizontally.
- When to use it
- Run at least two replicas behind a load balancer; it is on every request path.
Security and governance
Vault, RBAC, audit
- What it is
- Provider credentials in HashiCorp Vault or the cloud secrets manager, RBAC from your identity provider (SSO groups map to gateway teams), and audit logs shipped to a store the gateway cannot edit.
- When to use it
- Centralised credentials without spreading provider keys across microservices.
Cloud LLMsOpenAI, Anthropic, Gemini
- What it is
- Frontier APIs called directly or through Bedrock, Microsoft Foundry and Vertex AI. Going through a cloud keeps billing, private networking and data-processing terms inside an existing contract.
- When to use it
- Hard reasoning, long-context and agentic coding traffic.
Self-hosted models
vLLM, Ollama on GPUs
- What it is
- Open-weight models on your GPUs, registered in the gateway like any provider; the gateway decides which requests may leave the network and which stay local.
- When to use it
- Sensitive data, predictable high volume, offline sites.
OpenTelemetry
Traces and metrics
- What it is
- The gateway emits OTel spans and metrics (tokens, latency, errors per model) to the same backend as the rest of your services, so an LLM call is one span in the end-to-end trace.
- When to use it
- Finding whether a slow answer is the model, the retriever or the gateway.
Langfuse
LLM traces and evals
- What it is
- An open-source LLM observability tool that LiteLLM and other gateways log to: prompt and completion traces, cost per trace, evaluations and prompt versions. It complements OTel with the content of each call.
- When to use it
- Quality monitoring and regression checks when a model or prompt changes.
State: Postgres and RedisKeys, spend, cache
- What it is
- Self-hosted gateways keep keys, teams and spend in Postgres and use Redis for shared rate-limit counters and the cache, so replicas agree on limits. Back both up; losing them loses budgets and keys.
- When to use it
- Any deployment with more than one gateway replica.
Deploy on KubernetesReplicas, HPA, private
- What it is
- The gateway as a Deployment with a horizontal autoscaler, an internal load balancer, secrets mounted from the vault, and egress only to approved provider endpoints.
- When to use it
- Platform teams that already run their services on EKS, AKS or GKE.
Supply-chain hygiene
Pin, verify, rotate
- What it is
- A gateway holds every provider key, which makes it a prime target. In March 2026 two LiteLLM releases on PyPI were malicious; pin exact versions or image digests, use the vendor's stable images, scan dependencies and follow upstream advisories.
- When to use it
- Every self-hosted gateway, whichever project you choose.
- Key figures
- litellm 1.82.7 and 1.82.8 were live on PyPI for about 40 minutes on 24 March 2026; the official Docker image was not affected and 1.83.0 shipped through a rebuilt pipeline. Affected users were told to rotate every secret.Source: LiteLLM security update, March 2026
Scenarios and use cases
When you need one
- What it is
- The situations that justify a gateway, each with the capabilities it leans on, plus a chooser for the gateway that fits each stack.
- When to use it
- Start from the scenario, then pick the product.
Enterprise AI gateway
Every app, one door
- What it is
- A central platform team offers models to every department: one endpoint, SSO-backed keys per team, approved-model list, budgets and chargeback, central logs and one place to switch providers.
- When to use it
- Companies past the pilot stage with several AI apps in production.
- AI angle
- Leans on virtual keys, budgets, model aliases, logging and RBAC.
AI agent gateway
Agents, MCP, tools
- What it is
- Agents need models, tools and other agents. The gateway aggregates MCP servers behind one authenticated endpoint, injects each tool's credentials, limits which agent may call which tool, and logs tool calls next to model calls.
- When to use it
- Agent platforms with many MCP servers or agents acting on behalf of users.
- AI angle
- AgentCore Gateway, Azure API Management, Apigee, LiteLLM and Kong all cover this.
Multi-provider cost and failover
Cheapest model that works
- What it is
- Route each request to the cheapest model that handles it, cache repeats, spill to a second provider on 429s or outages, and keep a tested fallback for model retirements.
- When to use it
- High-volume products where token spend is a top cost line.
Secure AI gateway
Keys, access, audit
- What it is
- Provider keys only in the vault, least-privilege keys per service, PII redaction and guardrails on the way out, and an immutable audit trail of who sent which prompt to which model.
- When to use it
- Finance, health, legal and any team under an AI policy or NDA.
Data residency
Keep data in region
- What it is
- The gateway pins sensitive traffic to in-region endpoints (Bedrock geographic profiles, regional Azure deployments) or to a self-hosted model, and only lets anonymised requests reach global endpoints.
- When to use it
- EU personal data under the GDPR, public sector, contractual residency clauses.
Coding agents at scaleBudgets per developer
- What it is
- Coding agents such as Claude Code or Codex pointed at the gateway's base URL: one key and budget per developer, usage per team, and the model behind them chosen centrally. LiteLLM and Vercel AI Gateway document this setup.
- When to use it
- Rolling coding agents out to hundreds of engineers with a cost ceiling.
Per-customer AI in a SaaS
Meter and cap tenants
- What it is
- One gateway key per tenant gives usage-based billing, plan limits (tokens per month), per-tenant cache isolation and a way to block one abusive customer without touching the rest.
- When to use it
- B2B products that resell AI features inside their plans.
When you can wait
One app, one provider
- What it is
- A single app calling a single provider can rely on the provider's own limits and dashboards: Microsoft's docs note that a TPM limit on the deployment is enough until the portfolio grows. Add a gateway when the second app, team or provider arrives.
- When to use it
- Prototypes and single-purpose tools.
Which gateway
Match it to the stack
- What it is
- A starting recommendation per situation; prove it with a load test and a security review on your own traffic before committing.
- When to use it
- Shortlisting two candidates for a proof of concept.
General purpose
LiteLLM
- What it is
- The widest provider coverage, virtual keys, budgets and an admin UI, in a Python stack most AI teams already know.
- When to use it
- Python AI teams and multi-cloud platforms.
High throughput
Bifrost
- What it is
- A Go gateway built for low per-request overhead at thousands of requests per second.
- When to use it
- Latency-sensitive, high-volume production paths.
Kubernetes platform
Agent Router (Envoy)
- What it is
- AI traffic managed with the same Envoy data plane and Gateway API resources as the rest of the cluster.
- When to use it
- Platform teams that standardise on Envoy.
AWS-nativeBedrock + AgentCore
- What it is
- Bedrock with Guardrails and cross-Region inference for models, AgentCore Gateway and Identity for agents and tools; add the LiteLLM guidance when you need team budgets or non-AWS providers.
- When to use it
- Companies whose data and identity already live in AWS.
Azure and MicrosoftAPI Management
- What it is
- API Management's AI gateway policies in front of Foundry and other providers, with managed identities and Content Safety.
- When to use it
- Microsoft 365 and Azure-first organisations.
Google CloudApigee + Model Armor
- What it is
- Apigee token policies, routing and caching with Model Armor screening in front of Gemini and other models.
- When to use it
- Gemini-centred stacks and Apigee customers.
Fastest start
A hosted gateway
- What it is
- Cloudflare AI Gateway, Vercel AI Gateway or OpenRouter: logs, budgets and fallbacks with no infrastructure, if the data may pass through the vendor.
- When to use it
- Prototypes and small teams.
Sources, read 2026-11-16: the GitHub READMEs and documentation of LiteLLM (BerriAI/litellm), Bifrost (maximhq/bifrost), Portkey Gateway (Portkey-AI/gateway), Agent Router (envoyproxy/ai-gateway), Apache APISIX and Kong AI Gateway; the LiteLLM security update of March 2026; AWS documentation for Amazon Bedrock cross-Region inference and Guardrails, Amazon Bedrock AgentCore Gateway and AgentCore Identity, and the AWS Guidance for Multi-Provider Generative AI Gateway; Microsoft Learn, "AI gateway capabilities in Azure API Management"; Google Cloud documentation for Apigee AI capabilities and the Model Armor integration; the Cloudflare AI Gateway, Vercel AI Gateway and OpenRouter documentation; the Agentic AI Foundation announcement of September 2026. Latency figures are the vendors' own benchmarks. Features change between versions and tiers: check the docs of the release you deploy.