AI & Agents · Security & Compliance · Cloud, DevOps & LLMOps
Private AI Gateway: Secure, Self-Hosted LLM Access for Every Team
By David Gadyan · Published
A self-hosted AI gateway that every company app and AI agent goes through to reach a model. Provider keys stay in an OpenBao vault and each team can open only its own. Budgets and rate limits stop spend at the cap. Guardrails mask personal data and block leaked secrets and prompt injection. Every call is logged per team for chargeback, and apps keep the OpenAI SDK while models change behind stable aliases. LiteLLM OSS with custom plugins, FastAPI, a Next.js console with Keycloak SSO, Docker and Helm, 44 tests. A second build shows the same ideas on AWS with Bedrock AgentCore Gateway, Cedar policies and Postgres row-level security.
The problem
Every team bought its own AI account. API keys sat in code and chat, nobody knew the total bill until the invoice arrived, and staff pasted customer e-mails and card numbers into prompts. Switching to a cheaper or better model meant changing every app. The fix is one gateway that every app and AI agent goes through: it holds the keys, caps the spend, checks what goes out and records every call. This gateway runs on the company's own servers or in its own cloud account, and a second build shows the same controls with managed AWS services.
The product in use
What I built
One door with a dashboard. The overview shows requests, tokens, cost, the block rate, rate-limited calls, errors and p95 latency for the chosen period, a daily chart of requests and cost, the top projects and models by spend, and the health of the gateway, the vault and Redis. The question "what do we spend on AI, and on what?" has a live answer.
Budgets and limits per team. Each team is a project with an owner, a cost centre and a data classification. It gets a rate limit, a budget for a period, a list of allowed models and limits per model. In the screenshot the support bot may make 1.2k requests per minute, spend $400 per 30 days, and call the premium model only 30 times a minute.
Keys locked in a vault, one door per team. Provider keys (OpenAI, Anthropic and others) live only in OpenBao, the open-source fork of Vault. The console issues each project its own vault login, shown once and valid for 720 hours. With it, the team's apps can read only their own gateway key under the project's path; every other path returns 403. A new provider key is rotated into every running gateway worker without a restart.
Guardrails per team. Each project picks a policy, and each check (personal data, secrets, prompt injection, deny words, answer checks) can block, mask or only log. The playground tests a policy before it goes live: a prompt with an e-mail, a phone number and a card number comes back with all three replaced by placeholders, which is what the model receives. Personal-data detection uses Microsoft Presidio on the company's own servers, so there is no per-call fee and no third party sees the text. Answers are checked too, including streamed ones.
Teams ask, security approves. A project owner asks for a model, a higher limit or more budget, with a reason. A security admin approves or rejects, and approval applies the change to the gateway in seconds. The requester, the approver, the time and the reason stay on record, and nobody can approve their own request.
Swap models, not code. Apps call stable names such as chat-default or chat-premium through the standard OpenAI SDK. The platform team decides which model sits behind each name: OpenAI, Anthropic, Bedrock, Azure, Vertex AI or the company's own GPUs with vLLM or Ollama. Fallbacks run from premium to default to local. A built-in practice model lets anyone try the whole system without a provider account.
Architecture
An app calls the gateway with its own key. LiteLLM checks the key, the team's model list, its rate limits and its budget. My plugin, written only against LiteLLM's public hooks, then loads the project's policy (cached for 30 seconds and refreshed over Redis when the console changes it), applies the per-model limit and runs the input guardrails over every user-supplied string. LiteLLM's router picks a deployment for the alias, with retries and fallbacks. Output guardrails check the answer, and a request-log row records tokens, cost, latency and findings. Prompts are stored only when a project opts in, encrypted with a vault transit key.
The features LiteLLM sells as Enterprise (vault integration, guardrail integrations, reports, single sign-on) are built here on open source, nothing is imported from its enterprise code, and there is no gateway licence fee. The FastAPI control plane keeps LiteLLM, OpenBao and Postgres consistent on every change and rolls back if a step fails. The Next.js console signs in through Keycloak, so staff use their company login, and offers four roles: platform admin, security admin, project owner and viewer.
The same gateway on AWS
Clients who prefer managed services get the same ideas on their cloud. As an example I built a company copilot on Amazon Bedrock AgentCore that answers employees' questions about company data and only ever shows each person the tables, columns and rows they are allowed to see.
- The gateway decides first. AgentCore Gateway exposes the data and context tools over MCP. Its JWT authorizer checks the employee's Cognito token, and a Cedar policy decides which tools and tables a department and clearance level may use. Restricted columns are refused before the request reaches the data, and nothing that writes is permitted.
- The database decides again. A data service in a private VPC verifies the token once more, switches to the department's Postgres role and sets the employee id for the transaction. Aurora grants limit tables and columns, and row-level security limits rows to the employee's own records, reporting line, department, region or clearance.
- The agent never handles identity. The tools take no user ids. The employee's token travels from the browser through the agent and the gateway to the database session, so a persuasive prompt cannot change who is asking. A second model also screens each message for attempts to override the rules, as defence in depth.
- Managed pieces. LangGraph with Claude on Bedrock runs in AgentCore Runtime, AgentCore Memory keeps per-employee preferences, DynamoDB holds checkpoints, and every key and password lives in AWS Secrets Manager. 129 local tests cover the access matrix, the services, the Cedar policy and the agent end to end, and 10 more run against the deployed stack.
The same pattern maps to Azure (API Management's AI gateway with token limits and Content Safety) and Google Cloud (Apigee with Model Armor). The AI Gateways mind map compares the options.
Security and testing
Apps reach only the gateway and the vault; only the gateway can reach model providers. Provider keys exist in the vault and in the gateway's memory, never in env files, Helm values or the console. The control plane can write provider keys but cannot read them back. Every change in the console writes an audit row, and the vault's own audit log records every secret read. LiteLLM is pinned by tag and image digest after the 2026 incidents with compromised releases, and the upgrade path includes a signature check. The 44 tests cover guardrails, limits and live rotation, and the 16 end-to-end tests prove the claims above: cross-project reads get 403, a revoked key stops everywhere within 5 seconds, keys rotate with no restart, over-limit calls get 429 and every blocked request is logged exactly once.
The screenshots use a demo company, and the chart history includes generated demo data.
Related projects
AI Jobs MCP: Remote MCP Server with Google OAuth on AWS Lambda
AI & Agents · Cloud, DevOps & LLMOps · Security & Compliance
CareLens: Clinical Document AI on Kubernetes and Azure (HIPAA-ready)
Cloud, DevOps & LLMOps · Security & Compliance · AI & Agents
Azure AI Automation: Document Intelligence, RAG & Governed Agents
AI & Agents · Cloud, DevOps & LLMOps · Security & Compliance