Keyless LiteLLM Gateway Across Google Cloud and Azure

Summary

Pixel shares a proof of concept for a keyless LiteLLM model gateway running on Cloud Run and Azure Container Apps. Two AI agents use their cloud identities to access Gemini and Azure OpenAI through a shared model catalogue, with centralized permissions and usage records. Cross-cloud routing helps keep model access available if one gateway goes offline.

DEV Community

One model gateway across clouds: LiteLLM on Cloud Run and Azure with zero API keys

As of November 2026. All identifiers in this post are examples. We built a proof of concept with two AI agents: one on Google Cloud, in Agent Runtime, and one on Azure, in Microsoft Foundry. Each agent has its own identity in Microsoft Entra Agent ID and stores no secret. Agents call models. This post covers how ours reach them: a model gateway that runs in both clouds, serves Gemini and Azure OpenAI from either side, and holds no API key for any of it. During a demo we switched off the gateway in one cloud. The agent's next answer came through the other one, and we barely noticed the switch. Why put a gateway between agents and models An agent can call a model directly. With two agents that is fine. With fifty it gets uncomfortable: Every agent needs credentials for every model provider it uses. There is no single place that says which agent may use which model. Usage is spread across two clouds and two billing consoles. A model gateway fixes the three at once. Agents talk to one endpoint, and the rules and the usage records live there. We used LiteLLM because it speaks to both Vertex AI and Azure OpenAI behind one OpenAI-compatible API. The catch is that a gateway usually introduces two new kinds of secret: the keys it holds for the model providers, and the keys it hands out to its callers. We wanted neither. The design There is one LiteLLM in each cloud, with the same configuration: On Google Cloud it runs on Cloud Run. On Azure it runs on Container Apps. Each one has a Redis sidecar in the same service, for its own short-lived state. Both use one shared Postgres database, on Cloud SQL, for agents, allowed models and usage. Each gateway is connected to both model families. The one on Azure can call Gemini, and the one on Google Cloud can call Azure OpenAI. An agent uses the gateway of its own cloud and gets the same catalogue either way. The model list is the same file on both sides: model_list:
- model_name: gpt-4.1-mini
litellm_params:
model: azure/gpt-4.1-mini
api_base: https://.openai.azure.com/
- model_name: gemini-3.1-flash-lite
litellm_params:
model: vertex_ai/gemini-3.1-flash-lite
vertex_project:
vertex_location: global
There is no api_key line anywhere in it. Agents sign in as themselves LiteLLM normally gives each caller a virtual key. That is one more secret per agent, which is exactly what we wanted to avoid. Each agent already has an identity in Microsoft Entra Agent ID and can get a token for any audience it has a role on. So the gateway is one more audience, a2a-llm-gateway, with one role, LLM.Invoke. The agent sends its own Entra token as the bearer. On the LiteLLM side, a custom auth hook replaces the key lookup. Its core is short: def _verify(token: str) -> dict:
key = _JWKS.get_signing_key_from_jwt(token).key
return jwt.decode(token, key, algorithms=["RS256"],
audience=AUDIENCES, issuer=ISSUER,
options={"require": ["exp", "iss", "aud"]})

async def user_api_key_auth(request, api_key):
claims = await asyncio.to_thread(_verify, api_key)
if "LLM.Invoke" not in (claims.get("roles") or []):
raise Exception("forbidden: missing app role LLM.Invoke")
team = await get_team_object(team_id=claims["azp"], ...) # the agent's ID
model = await _requested_model(request)
if model and team.models and model not in team.models:
raise Exception(f"forbidden: model '{model}' not allowed for this agent")
return UserAPIKeyAuth(team_id=claims["azp"], team_models=team.models, ...)
It is wired in with one line of configuration: general_settings:
custom_auth: custom_auth.user_api_key_auth
The caller ID in the token is the agent's identity, and LiteLLM's "team" is keyed by that same ID. So usage is recorded per agent without anyone inventing a second naming scheme. Which agent may use which model The allow-list comes from the inventory file we use to onboard agen...
Keyless LiteLLM Gateway Across Google Cloud and Azure
Powered by Readybot.io

The latest from Pixel

Incident Response: Look Beyond the Red Alert

Pixel’s DEV Community announcement explores how teams can fixate on one failing service while overlooking the real cause of an outage, such as a full …

Sach: AI Fact-Checks WhatsApp Forwards in India

Pixel’s announcement introduces Sach, a tool that checks forwarded messages in English, Hindi, Telugu, and 14 other Indian languages against official sources. It gives cited …

Java 25 Removes SecurityManager: What to Fix

Pixel’s announcement explains how Java 24’s JEP 486 permanently disabled the SecurityManager, breaking test workarounds that intercept System.exit. It recommends returning exit codes or throwing …