Skip to content

5.7.1 Introduction

ai-gateway is the system responsible for every call to a language model: it holds the provider keys, routes each request by data class and plan tier, checks prompts and answers with Model Armor, meters token use per Organization and keeps a ledger without the prompts. It belongs to the Intelligence domain of the platform. Every interface it offers follows Choosing an Interface.

Proposal for review

The contracts of this system are a proposal, not confirmed design: endpoints, events and calls were written from the refined requirements, the data model and the flows, and wait for review. What needs a decision is collected in Open Decisions.

5.7.1.1 Host Names

Environment Host
Prod ai-gateway.moonitoring.com
Stage stage-ai-gateway.moonitoring.com, behind IAP

The names follow the host name rule; on a developer's computer each service answers on a local port (Running Locally). Each domain below is a path on these hosts, followed by its version: https://ai-gateway.moonitoring.com/<domain>/v1.

5.7.1.2 Domains

Each domain of ai-gateway is one API, with its own contract and its own version.

  • 5.7.2.1 Completions

    Every call to a language model goes through ai-gateway: it holds the provider keys, routes each request by data class and plan tier, checks prompts with Model Armor, reserves and settles the Organization's usage and keeps a ledger of every call without the prompt. /completions/v1

5.7.1.3 How It Is Used

MooChat completion

  1. MooChat calls the Completions domain through gRPC and names the model profile, metered feature, and highest data class in the prompt.
  2. AI Gateway validates the prompt with Model Armor and selects a provider allowed for that data class and plan tier.
  3. Personal data is sent only to Vertex AI in the EU; public prompts may use OpenRouter.
  4. The gateway returns the completion and records the call without storing the prompt.

Grant extraction

  1. Grant sends an OCR or extraction request through the same gRPC contract.
  2. AI Gateway applies the request's data-class and usage rules before calling a provider.
  3. Grant receives the extraction result and can resume or pause its batch when a provider is degraded.

Operations and usage

  1. Before either caller's model call, the gateway reserves usage with Farm's entitlements service.
  2. After the provider responds, it settles the actual token usage; the Organization is blocked or degraded when its plan limit requires it.
  3. The Console reads the usage ledger and provider state through REST, while provider keys remain inside the gateway.
flowchart LR moochat{{MooChat}} -->|gRPC completions| gateway[AI Gateway] grant{{Grant}} -->|gRPC OCR and batches| gateway console{{Console}} -->|REST ledger| gateway gateway -->|usage checks| farm[Farm / Entitlements] gateway -->|provider calls| models[Model providers]
  • Called by: Console (REST); access (gRPC CatalogService); farm (gRPC CatalogService); grant (gRPC BatchesService, gRPC CompletionsService); moochat (gRPC BatchesService, gRPC CatalogService, gRPC CompletionsService).
  • Calls: access (gRPC OrganizationsService); farm (gRPC EntitlementsService, gRPC UsageService).
  • Publishes events consumed by: grant (1 event type), moochat (1 event type).
  • Consumes events of: farm (1 entry), moochat (1 entry).
  • Reads no other system's database: everything above goes through a contract on this page or an event.

5.7.1.4 Synchronous Calls Between Systems

CompletionsService (complete, embed) in Completions. It calls UsageService of entitlements.

5.7.1.5 Events

Events are facts in the past tense, published through the outbox and delivered at least once on the topic ai-gateway.events.v1; the envelope and the guarantees are in common/v1/events.yaml. Events marked outside reach the Pub/Sub subscriptions an Organization registers in Webhooks, pushed to its HTTPS endpoint or pulled by its own service account.

Publishes (contract: events/v1/asyncapi.yaml):

Event Published when Consumers Outside
model_call.completed.v1 A model call ended, with its usage and outcome; the ledger row. Consumed by the security detection rules and the Console's usage views. none yet no
provider.degraded.v1 A provider is answering slowly or failing; callers may fall back. moochat, grant no

Consumes:

Event From Why
agent_model_profile.changed.v1 moochat Refreshes provider, model, region and data classes per profile.
entitlement.*.v1 farm Nothing: usage is reserved and settled synchronously through the Entitlements gRPC service.