DavidGadyan

AI Gateway Mind Map: LiteLLM, Bifrost, Portkey, AgentCore Gateway, Azure API Management and Apigee, with Use Cases

One control point between your apps, agents and every model provider: why you need it, what it does, the open-source gateways you host yourself, the AWS, Azure and Google Cloud services, and the scenarios each one fits.

  1. Why a gateway

    One control point

    What it is
    An AI gateway is a proxy layer between your applications, agents and MCP clients on one side and the model providers and tools on the other. Instead of every service holding its own OpenAI, Anthropic, Gemini or Bedrock credentials, everything calls one endpoint that authenticates, routes, limits, caches and logs.
    When to use it
    As soon as more than one app or team calls models, or more than one provider is in play.
    1. Key sprawl

      Provider keys everywhere

      What it is
      Without a gateway each microservice, notebook and agent stores provider API keys; rotating one means redeploying many services, and a leaked key has no owner. A gateway keeps the provider keys in one place and hands out its own scoped keys.
      When to use it
      Security reviews that ask who can call which model with which key.
    2. One API surface

      OpenAI-compatible

      What it is
      Most gateways expose an OpenAI-compatible endpoint (some also the Anthropic Messages API) and translate to each provider's format, so switching a model is a change of model name, not of SDK.
      When to use it
      Teams that want to try a new model without touching application code.
    3. Cost attribution

      Spend per team and app

      What it is
      Provider invoices show one total. The gateway records tokens and cost per key, user, team or tag, so finance can charge back and engineering can see which feature burns the budget.
      When to use it
      Internal chargeback, per-customer billing in a SaaS, budget alerts.
    4. Failover

      Outages and 429s

      What it is
      Providers rate-limit (HTTP 429), degrade and retire models. The gateway retries, moves the request to another deployment, region or provider, and trips a circuit breaker on an unhealthy backend so apps keep answering.
      When to use it
      Customer-facing assistants with an availability target.
    5. Governance and audit

      Policies in one place

      What it is
      Which models are approved, which data may leave the network, what gets logged and for how long: a gateway enforces these once for every caller instead of trusting each team to implement them.
      When to use it
      Regulated companies and any AI policy that must be provable to an auditor.
    6. Agent and MCP sprawl

      Tools need a front door

      What it is
      Agents call MCP servers, internal APIs and other agents as well as models. Gateways now front that traffic too: one authenticated MCP endpoint, per-tool permissions, outbound credentials injected for each tool, and logs of every tool call.
      When to use it
      More than a handful of MCP servers, or agents that act on behalf of users.
      AI angle
      Semantic tool search keeps prompts small when an agent can reach hundreds of tools.
    7. Not a secrets manager

      Pair it with a vault

      What it is
      A gateway routes requests and applies access policies; it is not automatically where provider credentials should live. Keep them in HashiCorp Vault, AWS Secrets Manager or Azure Key Vault, or use managed identities, and let the gateway read them at start-up.
      When to use it
      Any production deployment: the gateway database should never be the only copy of a provider key.
  2. Core capabilities

    What a gateway does

    What it is
    The common feature set across open-source and cloud gateways: identity and keys, limits and budgets, routing and resilience, caching, safety, observability and tool proxying. Availability differs by product, version and paid tier.
    When to use it
    Use it as the checklist when comparing gateways.
    1. Virtual keys

      Scoped, revocable keys

      What it is
      The gateway issues its own keys, each limited to some models, a budget and rate limits, and owned by a user or team. Revoking one never touches the provider key behind it. In LiteLLM they need a master key and a Postgres database.
      When to use it
      One key per app, per environment and per developer.
      In code
      curl http://localhost:4000/key/generate \
        -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
        -d '{"models": ["smart"], "max_budget": 50, "budget_duration": "30d", "tpm_limit": 100000}'
    2. Rate limits

      Requests and tokens

      What it is
      Limits on requests per minute (RPM) and tokens per minute (TPM) per key, team or IP, so one app cannot drain the shared provider quota. Token-aware limits count prompt and completion tokens, not just calls.
      When to use it
      Several apps sharing one provisioned deployment or one provider account.
    3. Token budgets

      Spend caps per period

      What it is
      A spend or token cap per key, user or team over a period (daily, monthly). When it runs out the gateway rejects new requests; Vercel AI Gateway, for example, answers HTTP 402 and documents its budgets as soft caps.
      When to use it
      Hackathons, trials, coding agents for many developers, per-customer plans.
    4. Routing and load balancing

      Pick the backend

      What it is
      Spread traffic over several deployments, keys, regions or providers: round-robin, weighted, priority (use the prepaid capacity first) or latency-based, and in some gateways model routing that picks a model per prompt.
      When to use it
      Filling provisioned throughput before spilling to pay-as-you-go.
    5. Retries and fallbacks

      Survive provider errors

      What it is
      Bounded retries on timeouts and 429s, then a fallback chain to another deployment or model, with a circuit breaker that stops sending traffic to a failing backend for a while.
      When to use it
      Any user-facing path; keep the fallback model evaluated, not just configured.
    6. Exact cache

      Same request, no call

      What it is
      Identical requests are answered from a cache (often Redis) instead of the provider: zero tokens and millisecond latency. Different from provider prompt caching, which only discounts a repeated prompt prefix.
      When to use it
      Evaluation reruns, deterministic classification, repeated FAQ prompts.
    7. Semantic cache

      Similar prompt, reuse

      What it is
      The prompt is embedded and compared with earlier prompts; above a similarity threshold the stored completion is returned. Saves tokens on paraphrased questions but can return a wrong answer if the threshold is loose.
      When to use it
      High-volume support questions; never for user-specific or time-sensitive answers.
      AI angle
      Tune the threshold on real traffic and scope the cache per tenant.
    8. Guardrails and PII

      Filter in and out

      What it is
      Checks on prompts and responses at the gateway: prompt-injection and jailbreak detection, denied topics, content moderation, and redaction of personal data before it reaches the provider. Often delegated to Bedrock Guardrails, Azure AI Content Safety, Model Armor or a third-party guard.
      When to use it
      Chatbots open to the public, PII in support tickets, regulated advice.
    9. Logs, metrics, traces

      Tokens, latency, cost

      What it is
      Every request logged with model, provider, tokens, latency, status and cost; metrics exported to Prometheus, Azure Monitor or CloudWatch; OpenTelemetry spans so a model call shows up inside the trace of the request that caused it.
      When to use it
      Debugging slow answers, capacity planning, cost dashboards.
    10. Model aliases

      "smart", "fast", "cheap"

      What it is
      Apps ask for a logical name and the gateway maps it to a concrete provider model. Upgrading or replacing a model becomes a gateway config change, and a retired model never breaks an app.
      When to use it
      Model upgrades and A/B tests without redeploying apps.
      In code
      model_list:
        - model_name: smart
          litellm_params:
            model: anthropic/<model-id>
            api_key: os.environ/ANTHROPIC_API_KEY
        - model_name: smart
          litellm_params:
            model: bedrock/<model-id>
    11. MCP and A2A proxying

      Tools behind the gateway

      What it is
      Gateways now expose MCP servers (and agent-to-agent APIs) through the same front door as models: one endpoint that aggregates tools, authenticates callers, applies per-tool access rules and logs each call. LiteLLM, Bifrost, Portkey, Kong, Azure API Management, Apigee and AgentCore Gateway all document it.
      When to use it
      Agents that need internal APIs as tools without each agent holding credentials.
  3. Open source, self-hosted

    Run it in your VPC

    What it is
    Gateways you deploy yourself (Docker, Kubernetes, a VM): prompts and logs stay in your network and you pay for compute, not per request. Most follow an open-core model, with SSO, advanced RBAC or some guardrails in a paid enterprise edition.
    When to use it
    Data that must not pass through a third-party gateway, multi-cloud, or many providers at once.
    1. LiteLLM

      Best general start

      What it is
      BerriAI's Python SDK and proxy server: 100+ LLM providers behind the OpenAI format, virtual keys, spend tracking per key, user and team, budgets and rate limits, router with retries and fallbacks, caching, guardrails, an admin UI, MCP and A2A gateways, and logging to Langfuse, MLflow and others. Enterprise features sit under a commercial licence.
      When to use it
      Python-heavy AI teams that want the widest provider coverage and a quick start.
      In code
      docker run -p 4000:4000 -v ./config.yaml:/app/config.yaml \
        ghcr.io/berriai/litellm:v1.83.0-stable --config /app/config.yaml
      Key figures
      • Vendor benchmark: 8 ms P95 latency at 1,000 requests per second.Source: LiteLLM README
    2. Bifrost

      Go, low overhead

      What it is
      Maxim's gateway written in Go under Apache 2.0: one OpenAI-compatible API over 23+ providers, automatic failover and load balancing across keys and providers, semantic caching, MCP support, virtual keys with budgets and rate limits, Prometheus metrics and tracing, and a web UI. A drop-in replacement for the OpenAI, Anthropic and Google GenAI SDKs by changing the base URL.
      When to use it
      High-throughput production paths where gateway overhead must stay in microseconds.
      In code
      npx -y @maximhq/bifrost
      docker run -p 8080:8080 maximhq/bifrost
      Key figures
      • Vendor benchmark: 11 µs added per request at a sustained 5,000 requests per second on an AWS t3.xlarge.Source: Bifrost README
    3. Portkey Gateway

      TypeScript, edge-ready

      What it is
      Portkey's MIT-licensed gateway in TypeScript: routing to 1,600+ models across 45+ providers, fallbacks, up to five automatic retries, load balancing, timeouts, input and output guardrails, simple caching, and a separate MCP gateway. Runs with npx, Docker, Node.js or Cloudflare Workers; semantic caching and prompt management are in the hosted or enterprise product, which Portkey says is merging into open source with its 2.0 release.
      When to use it
      JavaScript stacks, edge deployments and teams that want guardrails built in.
      In code
      npx @portkey-ai/gateway   # http://localhost:8787/v1
      Key figures
      • Vendor claims: under 1 ms latency and a 122 KB footprint.Source: Portkey Gateway README
    4. Agent Router (Envoy AI Gateway)

      Kubernetes-native

      What it is
      The control plane for AI and agent traffic built on Envoy and Envoy Gateway, Apache 2.0. Called Envoy AI Gateway until September 2026, when it joined the Agentic AI Foundation as Agent Router with the same code, maintainers and CRD names. One OpenAI-compatible API for hosted providers, self-hosted inference and MCP servers; platform teams manage credentials, routing, quotas, failover and usage attribution.
      When to use it
      Platform teams already running Envoy or the Kubernetes Gateway API.
      In code
      aigw run   # OpenAI-compatible router on localhost:1975
    5. Apache APISIX

      API gateway + AI plugins

      What it is
      A high-performance API gateway whose ai-proxy plugin translates one config into each provider's format (OpenAI, Anthropic, Gemini, Vertex AI, Bedrock, DeepSeek, OpenRouter, any OpenAI-compatible endpoint) and logs tokens and time to first token. Companion plugins: ai-proxy-multi, ai-cache, ai-rate-limiting, ai-prompt-guard, ai-rag and content moderation.
      When to use it
      Companies that already run APISIX for their REST APIs and want one gateway for both.
    6. Kong AI Gateway

      Kong + AI policies

      What it is
      Kong's AI layer: AI Model and Provider entities for routing, AI Consumer Groups for per-team token budgets, an AI MCP Server entity that exposes APIs as tools, and policies such as AI Prompt Guard, AI Semantic Cache, AI Sanitizer (PII), AI Rate Limiting Advanced and connectors to AWS Guardrails, Azure Content Safety and Model Armor. Check which plugins your Kong licence includes.
      When to use it
      Kong shops that want AI traffic under the same policies and portal as their APIs.
    7. Self-hosted models behind it

      vLLM, Ollama, SGLang

      What it is
      vLLM, SGLang and Ollama serve open-weight models on OpenAI-compatible endpoints, so the gateway treats them as one more provider: route sensitive traffic to the local model and the rest to a frontier API, behind the same keys and logs.
      When to use it
      Hybrid setups: private data on your GPUs, hard reasoning on a frontier API.
  4. AWS

    Bedrock + AgentCore

    What it is
    On AWS the gateway role is split: Amazon Bedrock gives one API, IAM and CloudTrail over many model families; Bedrock Guardrails filters content; AgentCore Gateway fronts tools, agents and model providers for agents; and AWS publishes a LiteLLM-based guidance for a self-managed multi-provider gateway.
    When to use it
    Workloads already on AWS that want IAM, VPC endpoints and CloudTrail for AI traffic.
    1. Amazon Bedrock

      Many models, one API

      What it is
      Managed access to foundation models from several vendors through one AWS API (Converse), with IAM permissions per model, private VPC endpoints, CloudTrail logging and per-account quotas. It is a model service, not a full gateway: budgets per team and non-AWS providers need something in front.
      When to use it
      The default model layer for AWS-native teams.
    2. Cross-Region inference

      Geographic or global

      What it is
      Inference profiles let Bedrock pick the Region that serves a request: a geographic profile keeps processing inside the US, EU or APAC, a global profile routes to any commercial Region. Traffic stays on the AWS network and CloudTrail records where each request ran.
      When to use it
      Burst capacity; geographic profiles when data residency applies.
      Key figures
      • No extra routing cost; global profiles cost about 10% less than geographic ones.Source: AWS docs, Cross-Region inference
    3. Bedrock Guardrails

      Filters in and out

      What it is
      Configurable safeguards on prompts and responses: content filters (hate, insults, sexual, violence, misconduct, prompt attack), denied topics, word filters, PII blocking or masking, contextual grounding checks and Automated Reasoning checks. The ApplyGuardrail API runs them without invoking a Bedrock model.
      When to use it
      One safety policy reused by a gateway in front of any model, not only Bedrock ones.
    4. AgentCore Gateway

      Agents to tools, models

      What it is
      A fully managed AI gateway for agentic traffic: it turns APIs, Lambda functions and services into MCP tools, fronts other agents and HTTP services (including A2A), and routes inference across model providers through one model-based routing endpoint. It handles inbound auth (who the agent is) and outbound auth (credentials for each tool) and is serverless with built-in observability.
      When to use it
      AWS agents that need company APIs and SaaS tools without custom integration code.
    5. Gateway targets

      OpenAPI, Lambda, MCP

      What it is
      Targets can be Lambda functions, API Gateway REST APIs, OpenAPI specifications, Smithy models, existing MCP servers and built-in connectors (Salesforce, Slack, Jira, Asana, Zendesk). Clients see one consolidated tools/list across all targets.
      When to use it
      Publishing an existing REST API to agents from its OpenAPI spec.
    6. AgentCore Identity

      Agent identity, tokens

      What it is
      Identity and credential management for agents: agent identities as workload identities, an inbound JWT authorizer for your IdP, credential providers that store OAuth tokens and API keys, and a consent flow so an agent can act on behalf of a user with audit trails.
      When to use it
      Agents that call Google, Microsoft or Salesforce APIs as the signed-in user.
    7. Multi-provider gateway guidance

      LiteLLM on ECS or EKS

      What it is
      AWS's published reference architecture deploys LiteLLM with Terraform on ECS Fargate or EKS behind an Application Load Balancer with AWS WAF, images in ECR, TLS from ACM, KMS encryption and CloudWatch: one OpenAI-style endpoint for Bedrock and other providers with per-team budgets.
      When to use it
      AWS shops that need budgets and non-Bedrock providers under one endpoint.
  5. Azure

    APIM AI gateway

    What it is
    The AI gateway in Azure API Management is a set of policies and features of the existing API gateway, not a separate product. It governs OpenAI-compatible, Anthropic Messages and Vertex AI APIs, models in Microsoft Foundry or other providers such as Amazon Bedrock, self-hosted models, MCP servers and A2A agent APIs. Availability varies by tier.
    When to use it
    Microsoft-centred companies that already publish APIs through API Management.
    1. llm-token-limit

      TPM and token quotas

      What it is
      A policy that enforces a tokens-per-minute limit or a token quota per hour, day, week, month or year on any counter key (subscription, IP or a custom expression), and can pre-count prompt tokens so an over-limit request never reaches the backend.
      When to use it
      Sharing one PTU deployment between apps without one starving the others.
      In code
      <llm-token-limit counter-key="@(context.Subscription.Id)"
        tokens-per-minute="500" estimate-prompt-tokens="false"
        remaining-tokens-variable-name="remainingTokens" />
    2. llm-emit-token-metric

      Tokens per consumer

      What it is
      Emits token counts to Azure Monitor with custom dimensions (client IP, API, user ID from a header), plus logging of prompts and completions to Application Insights and a built-in dashboard of token consumption.
      When to use it
      Chargeback per department and auditing who sent what.
    3. Semantic caching

      Redis + embeddings

      What it is
      The llm-semantic-cache-store and llm-semantic-cache-lookup policies use an embeddings API and Azure Managed Redis (or another RediSearch-compatible cache) to reuse completions for semantically similar prompts.
      When to use it
      Internal assistants with many near-identical questions.
    4. Load balancer and circuit breaker

      PTU first, then PAYG

      What it is
      Backend pools with round-robin, weighted, priority and session-aware balancing, and a circuit breaker whose trip duration follows the backend's Retry-After header. Priorities let you fill provisioned (PTU) capacity before pay-as-you-go.
      When to use it
      Multi-region Azure OpenAI deployments with an availability target.
    5. Content safety and identity

      No keys in apps

      What it is
      Managed identities authenticate the gateway to Azure AI services without API keys, the credential manager handles OAuth for apps and agents calling APIs or MCP servers, and a policy moderates prompts with Azure AI Content Safety.
      When to use it
      Removing provider keys from application config entirely.
    6. MCP servers in APIM

      REST API to MCP

      What it is
      API Management can expose an existing REST API as an MCP server, pass through and govern an existing MCP server, and import A2A agent APIs; Azure API Center keeps an organisational catalogue of APIs, MCP servers and skills.
      When to use it
      Giving Copilot Studio or Foundry agents governed access to internal APIs.
    7. Unified model API

      Preview, one endpoint

      What it is
      A preview feature that exposes several LLM backends from different providers through one OpenAI-compatible endpoint, translating formats automatically and applying governance policies once across all models.
      When to use it
      New multi-provider setups on Azure.
    8. AI gateway in Foundry

      Preview, from the portal

      What it is
      An API Management instance attached to a Microsoft Foundry resource governs it from the Foundry portal: token quotas and rate limits for model deployments, registration of agents running anywhere, and governed MCP tools in the Foundry inventory.
      When to use it
      Foundry-first teams that do not want to manage APIM policies by hand.
  6. Google Cloud

    Apigee + Model Armor

    What it is
    Apigee, Google's API management platform, carries the AI gateway features: token quotas, model routing, semantic caching, MCP security and Model Armor for prompt and response screening. Vertex AI can also call Model Armor directly.
    When to use it
    Google Cloud and Gemini workloads, or companies already on Apigee.
    1. LLM token policies

      LLMTokenQuota and more

      What it is
      LLMTokenQuota and PromptTokenLimit cap tokens per user or app (for example a daily maximum), and Apigee tracks token usage by model and app with latency, error rates and MCP tool performance.
      When to use it
      Token budgets per partner or app published through Apigee.
    2. Model Armor

      Screen prompts, responses

      What it is
      Google's screening service against prompt injection and harmful content. In Apigee the SanitizeUserPrompt and SanitizeModelResponse policies send traffic to Model Armor, which can allow, block or redact; Apigee and Model Armor must share a region, and Apigee needs a Comprehensive environment.
      When to use it
      Public-facing Gemini apps and any model behind Apigee.
    3. Model routing and cache

      Policy-driven

      What it is
      Dynamic model routing sends a request to a different model by use case through policy rules, and a semantic caching policy serves cached answers to similar queries.
      When to use it
      Sending easy requests to a smaller Gemini model and hard ones to a larger one.
    4. APIs as MCP tools

      API hub, OAuth

      What it is
      Apigee exposes APIs as MCP tools, secures MCP proxies with an OAuth v2 policy and JSON-RPC policies, and manages MCP servers, tools and agentic skills in API hub with natural-language search.
      When to use it
      Agent builders on Google Cloud reusing the company API catalogue.
  7. Hosted gateways

    No proxy to run

    What it is
    Gateways run by a vendor: one key, one endpoint, logs, budgets and failover without operating a proxy, database or control plane. Traffic passes through the vendor, so data handling terms matter.
    When to use it
    Start-ups and prototypes that want gateway features this week.
    1. Cloudflare AI Gateway

      Analytics, cache, limits

      What it is
      Available on all Cloudflare plans: analytics (requests, tokens, cost), logging, caching at Cloudflare's edge, rate limiting, retries and model fallback, and routing with model aliases and dynamic routes. Works with Workers AI, OpenAI, Anthropic, Gemini and others.
      When to use it
      Apps already behind Cloudflare that want visibility in one line of code.
    2. Vercel AI Gateway

      No markup, BYOK

      What it is
      A hosted gateway at ai-gateway.vercel.sh, callable from any cloud with the AI SDK, OpenAI or Anthropic clients: request logs, cost per request, budgets per team, project, key or member, prompt caching, provider and model fallbacks, and bring-your-own-key. Vercel states zero markup on provider token prices.
      When to use it
      TypeScript teams and coding agents that need many providers fast.
    3. OpenRouter

      Hundreds of models

      What it is
      One OpenAI-compatible API (base URL openrouter.ai/api/v1) to hundreds of models, with automatic provider fallbacks and routing to the most cost-effective option per request.
      When to use it
      Trying many open and closed models without separate accounts.
    4. Hosted or self-hosted

      The trade-off

      What it is
      Hosted: nothing to run, fastest start, but prompts pass through another company and features follow its roadmap. Self-hosted: you run Postgres, Redis and upgrades and own security patching, but data and logs stay in your VPC.
      When to use it
      Decide on data classification first, convenience second.
  8. Reference architecture

    Clients to providers

    What it is
    The layout recommended for a central AI platform serving many apps and agents: clients, then the gateway, then a security and governance layer, then cloud and self-hosted models, with observability across all of it.
    When to use it
    The starting diagram for an enterprise AI platform proposal.
    1. Clients

      Apps, agents, MCP clients

      What it is
      Web apps, backend services, agents, workflows and coding assistants: each gets a gateway key and a base URL, never a provider key.
      When to use it
      Onboarding a new team is issuing a key, not a procurement ticket.
    2. Gateway layer

      Auth, route, limit, log

      What it is
      Authentication, rate limits, model routing, fallbacks, cost tracking, caching, request logging and access policies, in one stateless tier that scales horizontally.
      When to use it
      Run at least two replicas behind a load balancer; it is on every request path.
    3. Security and governance

      Vault, RBAC, audit

      What it is
      Provider credentials in HashiCorp Vault or the cloud secrets manager, RBAC from your identity provider (SSO groups map to gateway teams), and audit logs shipped to a store the gateway cannot edit.
      When to use it
      Centralised credentials without spreading provider keys across microservices.
    4. Cloud LLMs

      OpenAI, Anthropic, Gemini

      What it is
      Frontier APIs called directly or through Bedrock, Microsoft Foundry and Vertex AI. Going through a cloud keeps billing, private networking and data-processing terms inside an existing contract.
      When to use it
      Hard reasoning, long-context and agentic coding traffic.
    5. Self-hosted models

      vLLM, Ollama on GPUs

      What it is
      Open-weight models on your GPUs, registered in the gateway like any provider; the gateway decides which requests may leave the network and which stay local.
      When to use it
      Sensitive data, predictable high volume, offline sites.
    6. OpenTelemetry

      Traces and metrics

      What it is
      The gateway emits OTel spans and metrics (tokens, latency, errors per model) to the same backend as the rest of your services, so an LLM call is one span in the end-to-end trace.
      When to use it
      Finding whether a slow answer is the model, the retriever or the gateway.
    7. Langfuse

      LLM traces and evals

      What it is
      An open-source LLM observability tool that LiteLLM and other gateways log to: prompt and completion traces, cost per trace, evaluations and prompt versions. It complements OTel with the content of each call.
      When to use it
      Quality monitoring and regression checks when a model or prompt changes.
    8. State: Postgres and Redis

      Keys, spend, cache

      What it is
      Self-hosted gateways keep keys, teams and spend in Postgres and use Redis for shared rate-limit counters and the cache, so replicas agree on limits. Back both up; losing them loses budgets and keys.
      When to use it
      Any deployment with more than one gateway replica.
    9. Deploy on Kubernetes

      Replicas, HPA, private

      What it is
      The gateway as a Deployment with a horizontal autoscaler, an internal load balancer, secrets mounted from the vault, and egress only to approved provider endpoints.
      When to use it
      Platform teams that already run their services on EKS, AKS or GKE.
    10. Supply-chain hygiene

      Pin, verify, rotate

      What it is
      A gateway holds every provider key, which makes it a prime target. In March 2026 two LiteLLM releases on PyPI were malicious; pin exact versions or image digests, use the vendor's stable images, scan dependencies and follow upstream advisories.
      When to use it
      Every self-hosted gateway, whichever project you choose.
      Key figures
      • litellm 1.82.7 and 1.82.8 were live on PyPI for about 40 minutes on 24 March 2026; the official Docker image was not affected and 1.83.0 shipped through a rebuilt pipeline. Affected users were told to rotate every secret.Source: LiteLLM security update, March 2026
  9. Scenarios and use cases

    When you need one

    What it is
    The situations that justify a gateway, each with the capabilities it leans on, plus a chooser for the gateway that fits each stack.
    When to use it
    Start from the scenario, then pick the product.
    1. Enterprise AI gateway

      Every app, one door

      What it is
      A central platform team offers models to every department: one endpoint, SSO-backed keys per team, approved-model list, budgets and chargeback, central logs and one place to switch providers.
      When to use it
      Companies past the pilot stage with several AI apps in production.
      AI angle
      Leans on virtual keys, budgets, model aliases, logging and RBAC.
    2. AI agent gateway

      Agents, MCP, tools

      What it is
      Agents need models, tools and other agents. The gateway aggregates MCP servers behind one authenticated endpoint, injects each tool's credentials, limits which agent may call which tool, and logs tool calls next to model calls.
      When to use it
      Agent platforms with many MCP servers or agents acting on behalf of users.
      AI angle
      AgentCore Gateway, Azure API Management, Apigee, LiteLLM and Kong all cover this.
    3. Multi-provider cost and failover

      Cheapest model that works

      What it is
      Route each request to the cheapest model that handles it, cache repeats, spill to a second provider on 429s or outages, and keep a tested fallback for model retirements.
      When to use it
      High-volume products where token spend is a top cost line.
    4. Secure AI gateway

      Keys, access, audit

      What it is
      Provider keys only in the vault, least-privilege keys per service, PII redaction and guardrails on the way out, and an immutable audit trail of who sent which prompt to which model.
      When to use it
      Finance, health, legal and any team under an AI policy or NDA.
    5. Data residency

      Keep data in region

      What it is
      The gateway pins sensitive traffic to in-region endpoints (Bedrock geographic profiles, regional Azure deployments) or to a self-hosted model, and only lets anonymised requests reach global endpoints.
      When to use it
      EU personal data under the GDPR, public sector, contractual residency clauses.
    6. Coding agents at scale

      Budgets per developer

      What it is
      Coding agents such as Claude Code or Codex pointed at the gateway's base URL: one key and budget per developer, usage per team, and the model behind them chosen centrally. LiteLLM and Vercel AI Gateway document this setup.
      When to use it
      Rolling coding agents out to hundreds of engineers with a cost ceiling.
    7. Per-customer AI in a SaaS

      Meter and cap tenants

      What it is
      One gateway key per tenant gives usage-based billing, plan limits (tokens per month), per-tenant cache isolation and a way to block one abusive customer without touching the rest.
      When to use it
      B2B products that resell AI features inside their plans.
    8. When you can wait

      One app, one provider

      What it is
      A single app calling a single provider can rely on the provider's own limits and dashboards: Microsoft's docs note that a TPM limit on the deployment is enough until the portfolio grows. Add a gateway when the second app, team or provider arrives.
      When to use it
      Prototypes and single-purpose tools.
    9. Which gateway

      Match it to the stack

      What it is
      A starting recommendation per situation; prove it with a load test and a security review on your own traffic before committing.
      When to use it
      Shortlisting two candidates for a proof of concept.
      1. General purpose

        LiteLLM

        What it is
        The widest provider coverage, virtual keys, budgets and an admin UI, in a Python stack most AI teams already know.
        When to use it
        Python AI teams and multi-cloud platforms.
      2. High throughput

        Bifrost

        What it is
        A Go gateway built for low per-request overhead at thousands of requests per second.
        When to use it
        Latency-sensitive, high-volume production paths.
      3. Kubernetes platform

        Agent Router (Envoy)

        What it is
        AI traffic managed with the same Envoy data plane and Gateway API resources as the rest of the cluster.
        When to use it
        Platform teams that standardise on Envoy.
      4. AWS-native

        Bedrock + AgentCore

        What it is
        Bedrock with Guardrails and cross-Region inference for models, AgentCore Gateway and Identity for agents and tools; add the LiteLLM guidance when you need team budgets or non-AWS providers.
        When to use it
        Companies whose data and identity already live in AWS.
      5. Azure and Microsoft

        API Management

        What it is
        API Management's AI gateway policies in front of Foundry and other providers, with managed identities and Content Safety.
        When to use it
        Microsoft 365 and Azure-first organisations.
      6. Google Cloud

        Apigee + Model Armor

        What it is
        Apigee token policies, routing and caching with Model Armor screening in front of Gemini and other models.
        When to use it
        Gemini-centred stacks and Apigee customers.
      7. Fastest start

        A hosted gateway

        What it is
        Cloudflare AI Gateway, Vercel AI Gateway or OpenRouter: logs, budgets and fallbacks with no infrastructure, if the data may pass through the vendor.
        When to use it
        Prototypes and small teams.

Sources, read 2026-11-16: the GitHub READMEs and documentation of LiteLLM (BerriAI/litellm), Bifrost (maximhq/bifrost), Portkey Gateway (Portkey-AI/gateway), Agent Router (envoyproxy/ai-gateway), Apache APISIX and Kong AI Gateway; the LiteLLM security update of March 2026; AWS documentation for Amazon Bedrock cross-Region inference and Guardrails, Amazon Bedrock AgentCore Gateway and AgentCore Identity, and the AWS Guidance for Multi-Provider Generative AI Gateway; Microsoft Learn, "AI gateway capabilities in Azure API Management"; Google Cloud documentation for Apigee AI capabilities and the Model Armor integration; the Cloudflare AI Gateway, Vercel AI Gateway and OpenRouter documentation; the Agentic AI Foundation announcement of September 2026. Latency figures are the vendors' own benchmarks. Features change between versions and tiers: check the docs of the release you deploy.