4.6 Services & Technologies
This page lists what we build with and why. Each choice names the alternative we set aside, so that a future change starts from the original reasoning instead of repeating it. A technology that is not on this page needs a decision before it is used in development, stage or prod; anything may be tried in a sandbox.
4.6.1 Cloud and Region¶
| Choice | Why | Set aside |
|---|---|---|
| Google Cloud, region europe-west8 (Milan) | Vertex AI and Google accelerators for our own models, startup credits, one bill with budgets and alerts; Milan is the region closest to our customers | AWS: no advantage for our models. Several clouds: operating cost without benefit at our size |
4.6.2 Compute and Messaging¶
| Choice | Used for | Why |
|---|---|---|
| Cloud Run services | Every API, web front end, consumer, relay and sweeper | Scales to zero, pays per request, no servers to patch |
| Cloud Run jobs | Migrations, exports and batches longer than a request | Same images and identities as the services |
| Pub/Sub | Events between services and between systems, pushed to the consumers; the ingestion queue, ordered per feed | Managed, at-least-once delivery; one stuck message blocks only its own feed |
| Cloud Scheduler | Waking the relays and sweepers, starting jobs | Calls an endpoint or a job with its own identity |
Services keep no state of their own: what must survive a restart lives in the database or in Cloud Storage. Nothing runs while there is no work; how services wake and scale is in Running on Demand.
4.6.3 Data¶
| Choice | Used for | Why | Set aside |
|---|---|---|---|
| Cloud SQL for PostgreSQL, Enterprise edition, europe-west8; highly available in prod | The database of every system | Inside our projects: private network, the same identities, audit logs (with pgAudit), backups and budgets as everything else; data stays in Milan | Neon: scales to zero, but runs on AWS in Frankfurt, outside our network and audit, reachable only over the internet with an address allow-list, and billed apart |
| SeaORM | The data layer of every Rust system: its entities, queries and transactions on PostgreSQL | Async and typed, in the same language as the system; the same entities build the system's Dev ORM | |
| Cloud Storage | Raw farm files, release files, datasets | Files are written once and addressed by their content | Uploading through our API: the API would sit in the path of every byte |
| BigQuery | Querying the audit log and large datasets | Reads Cloud Storage and log buckets in place |
The data model itself is described in Data.
4.6.4 Identity and Security¶
| Choice | Used for |
|---|---|
| Google Workspace | Staff accounts and groups |
Identity Platform, behind access |
Sign-in of Organization users |
| Identity-Aware Proxy | Staff access to the Console and to stage |
| Privileged Access Manager | Temporary, approved access to prod |
| Secret Manager | Every secret, one per service and purpose (see Secrets Management) |
| Cloud KMS with HSM keys | Signing releases of the installed clients, and the tokens of access with an asymmetric key: access signs through KMS and publishes the public key at /jwks, so the private key never leaves Google |
| Cloud Armor | Filtering requests at the edge |
4.6.5 Models¶
| Choice | Why | Set aside |
|---|---|---|
| Vertex AI (Gemini, EU endpoints) as the primary provider | EU endpoints in our own project, batch at half price, credits. Any prompt with personal data goes only here | OpenRouter as primary: adds a fee and has no EU residency on its standard plan |
| OpenRouter | Only for prompts without personal data that need a model Google does not offer | |
One AI gateway, the ai-gateway system |
Provider keys live in one place; requests are routed by data class and plan, and token use is counted per Organization | A key in every system |
4.6.6 Delivery¶
| Choice | Why | Set aside |
|---|---|---|
| GitHub for code, connected through Developer Connect | Developer Connect serves GitHub and GitLab alike, so the code host can change without touching the pipelines | Self-hosted GitLab: €42 to €82 a month plus upkeep; kept as an option |
| Cloud Build for CI, Cloud Deploy for releases | Runs inside Google Cloud with our service accounts: no keys outside, approvals and logs in the same audit | GitHub Actions for builds and deployments: needs federated identities and secrets held by GitHub |
| GitHub Actions for small checks | Format, lint and test jobs that need no Google Cloud identity and no secret, such as the checks of a library | GitHub Actions for anything that builds what reaches an environment, deploys, or holds a secret: that goes through Cloud Build. One exception until Cloud Build runs the checks: a read-only GitHub App token that lets the checks fetch our own private repositories, such as moonitoring-common-rs |
| Artifact Registry | Images and packages, built once and promoted by digest | |
| Sentry, EU data region | Errors, panics and crash dumps of every service, worker and installed client, grouped by release, and the only alerts on errors | Error Reporting alone: it groups only what reaches Cloud Logging, so it never sees the crash of a client on a farm computer |
| Terraform | All infrastructure (see Infrastructure as Code) | Changes by hand in the console |
4.6.7 Languages¶
Rust for internal critical systems, Python for machine learning and tooling, Rust with Slint for graphical interfaces, Nuxt for simple web sites and public web apps such as MooGrant. The rules are in Programming Languages.
| Choice | Used for |
|---|---|
| axum on tokio | The REST APIs of the Rust systems |
| tonic | gRPC between services, with the code generated from the contracts by tonic-build |
| utoipa | Describing each REST handler, so that a test compares the served API with its contract in this documentation |
| SeaORM and its migrator | The data layer and the migrations of every Rust system |
4.6.8 Running on Demand¶
Every part of every system scales to zero, in every environment, and wakes when its work arrives (decided 2026-11-05). An always-on worker would cost about €29 a month each, whether it does anything or not: a Cloud Run worker pool needs a whole vCPU, and each system has three of them.
- Minimum instances are 0 everywhere, with billing per request and startup CPU boost on, so a cold Rust service answers in about a second. A minimum instance in prod needs a reason written in its pull request; stage and development never have one.
<system>-consumeris a Cloud Run service that Pub/Sub pushes to, with internal ingress and the OIDC token of a push service account. A 2xx answer acknowledges the message; anything else is retried with growing delays and sent to the dead-letter topic after five attempts. The push subscription's acknowledgement deadline and the consumer's request timeout are both 600 seconds, the longest Pub/Sub allows; work that may take longer starts a Cloud Run job and acknowledges.- The backend publishes its own outbox rows right after the commit, before it answers, so an event leaves within milliseconds; a row is marked published only when Pub/Sub has acknowledged it.
<system>-relayis a Cloud Run service that Cloud Scheduler calls every minute and publishes what is left, so a crash between commit and publish delays an event by a minute at most and loses nothing. Events with an ordering key leave in outbox order: while an older event of the same key waits, the newer one waits for the relay too. A write recorded after the answer has started, such as the usage of a streamed completion ofai-gateway, leaves through the relay, up to a minute later. <system>-sweeperis a Cloud Run service that Cloud Scheduler calls at its interval: every five minutes foraccess,farm(heartbeats),news(scheduled articles) andai-gateway(polling its batches), and every hour forreleases, whose trust chain it renews. A scheduled job would be billed for a full minute at each run; a call is billed for the milliseconds it takes.- Relays, consumers and sweepers have internal ingress and admit only their caller: the push service account, or the service account of Cloud Scheduler.
Scaling out is bounded by the database, so connections are a budget. Each instance keeps a pool of at most four connections, two in stage and development, and each service has a maximum number of instances, so that across an environment the pools stay below 80% of the database's connection limit; the rest is kept for migrations and the Dev ORM. A service takes up to 80 requests at once, so load grows the work per instance before it grows the instances. Past that limit the answer is a larger instance or a read replica, not more connections.
Bytes never pass through a service: files go to and from Cloud Storage with signed URLs, and release files are served by Cloud CDN.
Stage costs little while nobody uses it: its Cloud SQL instance can be stopped outside working hours, it uses software keys of the same algorithms as the HSM keys of prod, and it keeps every trace. Prod samples one trace in ten, decided when a request starts, and its load balancer logs only failed requests. Every error still reaches Sentry and the logs with its trace id, sampled or not, so no error is lost; keeping every trace that has an error would need a tail-sampling collector running all the time, which costs more than it tells us.
Builds reuse what did not change: compiled dependencies are cached between builds (a cargo-chef layer in the registry and sccache in Cloud Storage), and a pull request runs the checks without building images; images are built once, on main.
4.6.9 E-mail¶
Transactional e-mail goes through Brevo (decided 2026-11-10): the invitations of access, the codes of releases' testers, and any future message a system sends to a person. Brevo is a European company (data in the EU, GDPR), its free plan covers 300 messages a day, which is far above our volume, and its HTTP API is called from Cloud Run with an API key kept in Secret Manager; its webhooks report bounces and complaints.
- Each system sends its own mail through a
Mailerport of its domain, with one adapter for Brevo and one that records messages for local runs and tests; no system shares another's sender or templates. - Messages are plain and in the person's language (Italian or English), from a no-reply address of
moonitoring.comwith SPF, DKIM and DMARC set for Brevo. - No marketing mail and no newsletter goes through this path.
- Alternatives considered: Google Workspace's SMTP relay (free and inside Google, but no API, no bounce events, and it needs a fixed egress address), SendGrid through the Google Cloud Marketplace (American, no permanent free plan), Amazon SES (cheapest at volume, outside both Google and the EU).