> ## Documentation Index
> Fetch the complete documentation index at: https://team.k0labs.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Routing
> Routing conventions and best practices for the k0rdent platform
# Multi-Region API Routing Architecture
**Status:** Draft
**Date:** 2026-02-08
***
## 1. Region Naming Convention
### 1.1 Industry Research
Every major platform has converged on a similar hierarchy but with different encoding choices:
| Provider | Format | Example | Hierarchy |
| -------------------- | ----------------------------- | ------------------------------------------- | --------------------------------- |
| AWS | `{continent}{cardinal}-{n}` | `us-east-1` | continent-direction-number |
| AWS (short) | `{continent}{cardinal}{n}` | `use1` | compressed, no delimiters |
| AWS (AZ) | `{region}{letter}` | `us-east-1a` | region + zone letter |
| GCP | `{continent}-{subregion}{n}` | `us-central1` | continent-subregion-number |
| GCP (zone) | `{region}-{letter}` | `us-central1-a` | region + zone letter |
| Azure | `{region}` | `eastus`, `westus2` | flat slug, no delimiters |
| Azure (regional DNS) | `{service}-{region}-01` | `contoso-westus2-01.regional.azure-api.net` | service-region-instance |
| Cloudflare | `{iata}` | `DFW`, `IAD` | 3-letter IATA, uppercase |
| Vercel | `{iata}{n}` | `sfo1`, `iad1`, `dub1` | city code + number |
| Hetzner | `{location}{n}` | `fsn1`, `nbg1`, `ash1` | location code + number |
| Hetzner (DC) | `{location}-dc{n}` | `fsn1-dc14` | location + datacenter number |
| Fly.io | `{iata}` | `dfw`, `iad`, `ams` | 3-letter IATA airport code |
| Kubernetes | `topology.kubernetes.io/zone` | `us-east-2a` | follows cloud provider convention |
### 1.2 Format
IATA airport codes map directly to metro areas where datacenters operate. Developers recognize them immediately. The number distinguishes multiple facilities in the same metro.
```
{iata}{n} or Airport Code + Datacenter Number
sfo1 San Francisco metro, datacenter 1
lax1 Los Angeles metro, datacenter 1
lax2 Los Angeles metro, datacenter 2
ams1 Amsterdam metro, datacenter 1
fra1 Frankfurt metro, datacenter 1
nrt1 Tokyo Narita metro, datacenter 1
```
### 1.3 Why IATA
Fly.io, Cloudflare, Fastly, and Wikimedia all use IATA codes for physical infrastructure. k0rdent sells bare metal in specific physical locations — the same category. Cloud-style codes (`us-west-1`) encode abstract geography because you never know the specific datacenter. With bare metal, the physical location *is* the product.
IATA codes are globally unique, assigned by an international body, 3 characters, and human-memorable. `sfo1` carries more information in 4 characters than `usca1` does in 5 — it tells you the metro, not just the state.
### 1.4 Subdomain Format
```
sfo1.api.example.com → San Francisco datacenter 1
lax1.api.example.com → Los Angeles datacenter 1
api.example.com → Global (resolved via session + settings)
```
### 1.5 Region Registry
```typescript theme={null}
const REGIONS = {
sfo1: { code: 'sfo1', label: 'San Francisco 1', country: 'US', timezone: 'America/Los_Angeles', provider: 'iren' },
lax1: { code: 'lax1', label: 'Los Angeles 1', country: 'US', timezone: 'America/Los_Angeles', provider: 'iren' },
ams1: { code: 'ams1', label: 'Amsterdam 1', country: 'NL', timezone: 'Europe/Amsterdam', provider: 'equinix' },
} as const;
```
One code used everywhere — subdomains, request IDs, database columns, Kubernetes labels, log correlation. No translation tables ([FR-18](#fr-18)).
***
## 2. Request IDs vs Resource IDs
### 2.1 Request IDs
Every API request gets a unique Request ID for tracing, debugging, and correlation across services. Request IDs embed the region to enable grep-based debugging across the entire stack.
**Format:** `req_{region}-{timestamp}-{entropy}`
```
req_sfo1-1770564159296-7d4b9e1f3a5b
req_lax1-1770564205843-3c2f9a8e7b4d
req_ams1-1770564298172-9f3e8d2c1a5b
```
**Components:**
* `req_` — Platform prefix for easy grepping in logs
* `sfo1` — Region code where request was processed
* `1770564159296` — Unix timestamp in milliseconds (sortable, debuggable)
* `7d4b9e1f3a5b` — Random entropy (12 hex characters)
**Why embed region in Request IDs:**
* Grep any log file for `sfo1` and see all activity in that region
* Trace a request across services without additional lookups
* Instant debugging: see the request ID, know which region handled it
* Correlate metrics, traces, and logs by region prefix
**Usage:**
* Returned in `X-Request-Id` response header
* Logged by all services handling the request
* Used for distributed tracing correlation
* NOT exposed to end users (internal debugging only)
### 2.2 Resource IDs
Resources (clusters, servers, organizations, etc.) get globally unique, opaque IDs that do NOT contain region information. This decouples resource identity from physical location.
**Format:** `{prefix}_{base62}`
| Resource | Prefix | Example |
| ------------ | -------- | ------------------------------------ |
| Organization | `org_` | `org_8TcVx2WkZddNmK3Pt9JwX7BzWrLM` |
| Server | `srv_` | `srv_3KpQm9WnXccFjH2Ls8DkT6VzRqYU` |
| Cluster | `cls_` | `cls_6NZtkvWLBbbmHfPi7L6oz7KZpqET` |
| Stack | `stk_` | `stk_5MfRp4WjYbbHmG8Nt2LvS9CxPqZK` |
| Workflow Run | `run_` | `run_7NhTq6WlAbbKmF5Rt3MxU8DzSqWJ` |
| Pool | `pool_` | `pool_2LgPn8WmXccGjE7Mt4KwV9BySrTL` |
| Allocation | `alloc_` | `alloc_9QjSr3WnZddMmH6Pt5LxW2CzUrYK` |
| API Key | `key_` | `key_4KfQm7WkYccJmG3Nt8MvX9BzSqWL` |
| Event | `evt_` | `evt_6MgRp2WlXbbKmF9Rt5NxU3DzTqZJ` |
**Special ID:** `org_system` is reserved for platform-level admin operations. TBD if this is needed still. Originally it was for something else.
**Components:**
* `{prefix}_` — Resource type for debugging clarity (e.g., `cls_`, `srv_`, `org_`)
* `{base62}` — 26-character base62-encoded unique identifier (case-sensitive)
**Why NOT embed region:**
* Resources can migrate between regions without ID changes
* Multi-region resources don't have a single "home" region
* Simpler, shorter IDs (30 chars vs 40+)
* Region is stored as metadata on the resource, queryable via API
* Resource identity is independent of physical topology
**Region resolution:**
Region information is resolved through the [routing cascade](#24-region-resolution-cascade):
1. Subdomain (`sfo1.api.example.com`)
2. Header (`X-Region: sfo1`)
3. Query param (`?region=sfo1`)
4. Request body (`{ "region": "sfo1" }`)
5. Session context (project/org default region)
6. Database lookup (stored on resource record)
***
## 3. API Design
### 2.1 Core Principle
Region is a **routing concern**, not a resource hierarchy. The API path describes *what*. The subdomain/header/session describes *where* ([FR-06](#fr-06)).
```
✗ /v1/region/us-ca-1/zones/z1a/locations/loc_abc2fd456/clusters/cls_abc789 ← couples path to topology
✓ /v1/region/us-ca-1/clusters/cls_abc789 ← clean, region resolved separately
```
### 2.2 Atlas API (Operator Console)
```
Base: /v1/region/global
GET /infrastructure/servers List servers (filter by ?region=sfo1)
GET /infrastructure/servers/:serverId Server details
POST /infrastructure/servers/:serverId/provision Provision server
GET /infrastructure/regions List all regions
GET /regions/:regionCode Region details (capacity, health)
GET /compute/clusters All clusters across all orgs
GET /organizations All organizations
```
Regions are queryable resources, not path hierarchy. Atlas operators use subdomain pinning or query filters to scope: `sfo1.api.example.com/v1/region/global/servers` returns only servers in that region.
### 2.3 Arc API (Customer Console)
```
Base: /v1/region/{region}
POST /compute/clusters Create cluster (region in body)
GET /compute/clusters List my org's clusters
GET /compute/clusters/:clusterId Cluster details
DELETE /compute/clusters/:clusterId Delete cluster
GET /compute/clusters/:clusterId/kubeconfig Get kubeconfig
GET /projects List projects
POST /projects Create project
GET /me Current user
PATCH /me/settings Update settings
GET /notifications List notifications
PATCH /notifications/:id/read Mark read
```
Organization resolved from session, never in the URL ([FR-09](#fr-09)). Region specified in request body for creates: `POST /compute/clusters { "name": "prod", "region": "sfo1" }`. Once a resource exists, its region is embedded in its ID ([FR-08](#fr-08)).
### 2.4 Region Resolution Cascade
Every request resolves a region before reaching application code ([FR-01](#fr-01)):
```
┌──────────────────────────────────────────────────────────────────┐
│ REQUEST ARRIVES AT EDGE or GATEWAY │
│ │
│ GET /v1/region/{region}/clusters/cls_6NZtkvWLBbbmHfPi7L6oz7KZpqET │
│ Host: api.example.com │
│ Cookie: session=... │
│ X-Request-Id: req_sfo1-1770564159296-7d4b9e1f3a5b │
└──────────────────────────────┬───────────────────────────────────┘
│
┌──────────▼──────────┐
│ 1. SUBDOMAIN CHECK │
│ sfo1.api.example.com?│──→ Yes: region = sfo1
│ api.example.com? │──→ No subdomain, continue
└──────────┬──────────┘
│ (no subdomain)
┌──────────▼──────────┐
│ 2. HEADER CHECK │
│ X-Region: sfo1? │──→ Yes: region = sfo1
└──────────┬──────────┘
│ (no header)
┌──────────▼──────────┐
│ 3. QUERY PARAM │
│ ?region=sfo1 │──→ Yes: region = sfo1
└──────────┬──────────┘
│ (no query param)
┌──────────▼──────────┐
│ 4. REQUEST BODY │
│ { "region": "sfo1" │
│ ... } │──→ Yes: region = sfo1
└──────────┬──────────┘
│ (no body region, creates only)
┌──────────▼──────────┐
│ 5. SESSION CONTEXT │
│ prj.defaultRegion │──→ Yes: region = project default
│ (inherits from │
│ org.defaultRegion)│
└──────────┬──────────┘
│ (no default, e.g. multi-region org)
┌──────────▼──────────┐
│ 6. DATABASE LOOKUP │
│ Query resource │
│ record for region │──→ Yes: region from DB
└──────────┬──────────┘
│ (no resource or multi-region query)
┌──────────▼──────────┐
│ 7. FALLBACK │
│ GET → fan-out │──→ Query all org regions
│ Mutation → error │──→ 400: region required
└─────────────────────┘
```
Explicit always wins ([FR-02](#fr-02)). A single-region org user never thinks about regions — the session default handles everything ([FR-10](#fr-10)).
### 2.5 Persona Experiences
**Arc developer, single-region org:**
```bash theme={null}
k0r clusters list # Region from session
k0r clusters create --name prod # Region from session
# Developer never sees or types a region code
```
**Arc developer, multi-region org:**
```bash theme={null}
k0r clusters create --name prod --region sfo1 # Must specify
k0r clusters list # All regions
k0r clusters list --region lax1 # Filter
k0r clusters get cls_6NZtkvWLBbbmHfPi7L6oz7KZpqET # Region from DB
```
**Atlas operator:**
```bash theme={null}
k0r atlas servers list # Global
k0r atlas servers list --region sfo1 # Filter
k0r config set region sfo1 # Pin default
curl https://api.internal.example.com/v1/region/global/infrastructure/servers
```
***
## 3. Database Topology
### 3.1 Option A: Single Primary + Regional Read Replicas (MVP)
```
┌─────────────────────────────────────────────────────────┐
│ MOTHERSHIP REGION │
│ │
│ ┌──────────────┐ ┌──────────────────────────────┐ │
│ │ PG Primary │ │ Atlas App + Gateway │ │
│ │ (all writes) │ │ Auth Service │ │
│ │ │◄───│ Workflow Orchestrator │ │
│ └──────┬───────┘ └──────────────────────────────┘ │
│ │ replication │
└─────────┼────────────────────────────────────────────────┘
│
┌─────┴──────┬──────────────┐
▼ ▼ ▼
┌────────┐ ┌────────┐ ┌────────┐
│ Replica │ │ Replica │ │ Replica │
│ sfo1 │ │ lax1 │ │ ams1 │
│ │ │ │ │ │
│ Arc App │ │ Arc App │ │ Arc App │
│ (reads) │ │ (reads) │ │ (reads) │
└────────┘ └────────┘ └────────┘
```
Infrastructure mutations (create cluster, provision server) go through the workflow orchestrator to the primary ([FR-14](#fr-14)). User-facing writes (settings, notifications) go cross-region to the primary — 20-80ms extra, acceptable for MVP. See [§8.1](#81-single-primary-write-latency) for evolution triggers.
### 3.2 Option B: Hybrid — Regional for Fast Writes, Central for State of Record
```
┌─────────────────────────────────────────────────┐
│ MOTHERSHIP REGION │
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ PG Primary (State of Record) │ │
│ │ - Infrastructure state (servers, clusters)│ │
│ │ - Organization/tenant config │ │
│ │ - Billing, audit log │ │
│ │ - Auth (existing) │ │
│ └──────────────────────────────────────────┘ │
└─────────────────────────────────────────────────┘
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ sfo1 │ │ lax1 │ │ ams1 │
│ │ │ │ │ │
│ Redis/SQLite │ │ Redis/SQLite │ │ Redis/SQLite │
│ - User prefs │ │ - User prefs │ │ - User prefs │
│ - Notif read │ │ - Notif read │ │ - Notif read │
│ - UI state │ │ - UI state │ │ - UI state │
│ - Session │ │ - Session │ │ - Session │
│ cache │ │ cache │ │ cache │
│ │ │ │ │ │
│ PG Replica │ │ PG Replica │ │ PG Replica │
│ (reads only) │ │ (reads only) │ │ (reads only) │
└──────────────┘ └──────────────┘ └──────────────┘
```
**Operational overhead:** Medium. Central PG for source of truth (you already manage this). Regional Redis or lightweight store for fast user-facing writes that are either eventually consistent (notification read status) or can be synced lazily.
**The insight:** "Mark notification as read" and "update my theme preference" don't need the same consistency guarantees as "create a Kubernetes cluster." Separating the write path by consistency requirement lets you keep the simple central database for things that matter while giving snappy UX for things that don't.
**Verdict:** Good upgrade path from Option A. Start with Option A, identify which user-facing writes are actually causing UX pain from cross-region latency, then selectively move those to regional stores.
### 3.3 Recommendation: Start A, Evolve to B
```
MVP (Now): Option A — Single primary, regional read replicas
Accept 20-80ms on user settings writes
Monitor write latencies with OpenTelemetry
Trigger to evolve: p95 write latency > 200ms for user-facing operations
OR customer complaints about "slow" UI interactions
OR expansion to regions with > 150ms to primary
Scale-up (Later): Option B — Add regional Redis for user state
Notification read status → regional Redis, async sync
User preferences → regional Redis, async sync
All infra state stays in central PG
```
### 3.4 Write Latency by Type
| Write | Latency Sensitivity | Path |
| ---------------------- | ------------------------ | -------------------------------- |
| Create cluster | Low (async workflow) | Arc → Workflow Queue → Primary |
| Provision server | Low (async workflow) | Atlas → Workflow Queue → Primary |
| Update user settings | High | Arc → Primary (cross-region) |
| Mark notification read | High | Arc → Primary (cross-region) |
| Login/session | High (auth handles this) | Auth → Primary |
***
## 4. Routing Architecture
### 4.1 Request Flow
```
┌──────────────────────────────────────────────────────────────┐
│ 1. DNS (GeoDNS) │
│ Resolves api.example.com or api.example.com │
│ Routes to nearest/specified regional gateway │
└────────────────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 2. Regional Gateway │
│ • TLS termination │
│ • Region resolution (see §2.4) │
│ • Authentication check │
│ • Request ID stamping (req_{region}-{ts}-{entropy}) │
└────────────────────────┬─────────────────────────────────────┘
▼
┌──────────────────────────────────────────────────────────────┐
│ 3. Service Mesh │
│ Routes to appropriate backend: │
│ • Local Arc service (regional) │
│ • Mothership Atlas service (global) │
└──────────────────────────────────────────────────────────────┘
```
### 4.2 Request Flow Examples
**Arc user creates a cluster (multi-region org, no subdomain):**
```
1. POST https://api.example.com/v1/region/{region}/compute/clusters
Body: { "name": "prod-gpu", "region": "lax1", "template": "k8s-gpu-a100" }
Cookie: session=abc123
2. DNS resolves api.example.com → nearest gateway (sfo1 for this user)
3. Gateway (sfo1):
a. No subdomain → no region override
b. No X-Region header → no region override
c. No resource ID (it's a create) → no ID-based resolution
d. No ?region param → no param override
e. Check request body: region = "lax1" → resolved!
f. Validate: org allows lax1? ✓
g. Stamp Request ID: req_lax1-1770564159296-7d4b9e1f3a5b
h. Route to lax1 regional services via mesh
4. Regional Arc service (lax1):
a. Validates template exists and has capacity
b. Writes cluster record to PG primary (cross-region to mothership)
c. Enqueues workflow: "create-cluster" with request context
d. Returns 202 Accepted with cluster ID and status URL
5. Workflow orchestrator (mothership):
a. Picks up workflow
b. Talks to CAPI/Metal3 in lax1 regional K8s cluster
c. Updates cluster status as it progresses
```
**Arc user marks a notification as read:**
```
1. PATCH https://api.example.com/v1/region/global/notifications/ntf_sfo1-1770564159296-a3f8/read
Cookie: session=abc123
2. Gateway (nearest):
a. Session → org.defaultRegion = sfo1
b. Stamp Request ID: req_sfo1-1770564159296-...
c. Route to local Arc service
3. Arc service (sfo1):
a. MVP: Write to PG primary (20-80ms extra, acceptable)
b. Future (Option B): Write to regional Redis, async sync to PG
c. Returns 200 OK
```
**Atlas operator lists all servers globally:**
```
1. GET https://api.example.com/v1/region/global/infrastructure/servers
X-Region: * (or omitted — global is default for Atlas)
2. Gateway:
a. Atlas route + no region pin → global query
b. Route to mothership Atlas service
3. Atlas service (mothership):
a. Queries PG primary directly (it's co-located)
b. Returns all servers across all regions
c. Response includes server.region for each
4. Alternatively, with subdomain pin:
GET https://sfo1.api.example.com/v1/region/global/infrastructure/servers
→ Only returns servers in sfo1
```
### 4.3 Gateway Implementation (Hono)
```typescript theme={null}
import { createMiddleware } from 'hono/factory';
const REGION_PATTERN = /^[a-z]{3}\d+$/; // sfo1, lax2, ams1
export const regionResolver = createMiddleware(async (c, next) => {
let region: string | null = null;
let source = 'none';
// 1. Subdomain
const host = c.req.header('host') ?? '';
const sub = host.match(/^([a-z0-9]+)\.api\./);
if (sub && REGION_PATTERN.test(sub[1])) {
region = sub[1];
source = 'subdomain';
}
// 2. Header
if (!region) {
const h = c.req.header('x-region');
if (h && REGION_PATTERN.test(h)) { region = h; source = 'header'; }
}
// 3. Query param
if (!region) {
const q = c.req.query('region');
if (q && REGION_PATTERN.test(q)) { region = q; source = 'query-param'; }
}
// 4. Request body (creates)
if (!region && c.req.method === 'POST') {
try {
const body = await c.req.json();
if (body.region && REGION_PATTERN.test(body.region)) {
region = body.region; source = 'request-body';
}
} catch {}
}
// 5. Session default
if (!region) {
const session = c.get('session');
if (session?.project?.defaultRegion) {
region = session.project.defaultRegion; source = 'project-session';
} else if (session?.org?.defaultRegion) {
region = session.org.defaultRegion; source = 'session';
}
}
// 6. Database lookup (for single-resource GET/PATCH/DELETE)
// If request path contains a resource ID, query DB for its region
// This step is deferred to the service layer in most cases
// 7. Fallback
if (!region) {
if (c.req.method === 'GET') { region = 'global'; source = 'fan-out'; }
else return c.json({ error: 'region_required' }, 400);
}
const requestId = generateRequestId(region);
c.set('region', region);
c.set('regionSource', source);
c.set('requestId', requestId);
c.header('X-Request-Id', requestId);
c.header('X-Region', region);
await next();
});
```
### 4.4 SDK Configuration
```typescript theme={null}
import { K0rdentClient } from '@k0rdent/sdk';
// Single-region org — region derived from auth
const client = new K0rdentClient({ apiKey: 'key_4KfQm7WkYccJmG3Nt8MvX9BzSqWL' });
// Pin to a region
const client = new K0rdentClient({ apiKey: 'key_4KfQm7WkYccJmG3Nt8MvX9BzSqWL', region: 'sfo1' });
// Per-request override
const clusters = await client.clusters.list({ region: 'lax1' });
```
***
## 5. Request ID Generation
Request IDs are generated at the gateway for every API request. They provide distributed tracing and enable grep-based debugging across services.
### 5.1 Generation Function
```typescript theme={null}
import { randomBytes } from 'crypto';
function generateRequestId(region: string): string {
const timestamp = Date.now(); // Unix milliseconds
const entropy = randomBytes(6).toString('hex'); // 12 hex chars
return `req_${region}-${timestamp}-${entropy}`;
}
// Examples:
generateRequestId('sfo1'); // req_sfo1-1770564159296-7d4b9e1f3a5b
generateRequestId('lax1'); // req_lax1-1770564205843-3c2f9a8e7b4d
generateRequestId('ams1'); // req_ams1-1770564298172-9f3e8d2c1a5b
```
### 5.2 Region Extraction from Request IDs
```typescript theme={null}
function extractRegionFromRequestId(requestId: string): string {
// Region is always immediately after the req_ prefix
const match = requestId.match(/^req_([a-z]{3}\d+)-/);
return match?.[1] ?? 'unknown';
}
extractRegionFromRequestId('req_sfo1-1770564159296-7d4b9e1f3a5b'); // 'sfo1'
extractRegionFromRequestId('req_ams1-1770564205843-3c2f9a8e7b4d'); // 'ams1'
```
The region code in Request IDs is the same string used in subdomains, database columns, Kubernetes labels, and log correlation ([FR-18](#fr-18)). This enables powerful debugging workflows:
```bash theme={null}
# Find all activity in sfo1 across all logs
grep "sfo1" /var/log/gateway/*.log
# Trace a specific request across services
grep "req_sfo1-1770564159296-7d4b9e1f3a5b" /var/log/**/*.log
# Find all requests to a specific region in the last hour
journalctl --since "1 hour ago" | grep "req_sfo1-"
```
***
## 6. Functional Requirements
Any alternative proposal must satisfy these. If it can't, the burden is on the proposer to explain why the requirement is wrong.
### Critical Requirements
| ID | Requirement | Rationale |
| ----------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| FR-01 | Resolve target region for every request before it reaches application code. | Routing is infrastructure, not application logic. |
| FR-02 | Follow deterministic priority: Subdomain → Header → Query → Body → Session → DB Lookup → Fan-out/Error. | Developers must predict where requests go. |
| FR-06 | Region never appears as a path segment. | Decouples API contract from topology changes. |
| FR-08 | Resource IDs are globally unique and opaque (base62, 26 chars). Region stored as metadata. | Enables resource migration between regions without ID changes. |
| FR-09 | Request IDs embed region, timestamp, and entropy. Format: `req_{region}-{ts}-{entropy}`. | Enables grep-based debugging and distributed tracing. |
| FR-10 | Single-region org users never need to specify a region. | The DX bar. If they think about regions, we failed. |
| FR-11 | Arc requests only return resources for the authenticated org, regardless of gateway. | Tenant isolation is non-negotiable. |
| FR-14 | Infrastructure mutations go through the workflow orchestrator to primary DB. | Eventual consistency on "does this cluster exist" is not acceptable. |
All functional requirements including [FR-03 through FR-19](#appendix-a-all-functional-requirements) →
***
## 7. Non-Functional Requirements
MVP and production targets. The gap between them defines when to [scale](#8-scaling-roadmap).
### Critical Targets
| ID | Metric | MVP | Production |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ---------------------------------------------- |
| NFR-02 | Same-region read (Arc list/get) | \< 100ms p95 | \< 50ms p95 |
| NFR-03 | Cross-region write (settings, notifications) | \< 150ms p95 | \< 100ms p95 |
| NFR-07 | Per-region read availability | 99.5% monthly | 99.9% monthly |
| NFR-10 | Mothership availability | 99.5% monthly | 99.9% monthly |
| NFR-13 | Primary DB down | All writes fail. Reads continue. `503` + `Retry-After`. | Managed failover within 60-120s. RTO \< 5 min. |
| NFR-19 | Every request emits OTel span: `region`, `region_source`, `request_id`, `org_id`, `latency_ms`, `status_code` | Required | Required |
All non-functional requirements including [NFR-01 through NFR-23](#appendix-b-all-non-functional-requirements) →
***
## 8. Scaling Roadmap
Each item is a deliberate MVP limitation. The table below summarizes what to monitor, what triggers action, and what to build. Details follow.
| # | Monitor | Trigger | Build | Effort |
| --------------------------------------- | -------------------------------- | ----------------------------------- | --------------------------------------------- | -------------------- |
| [8.1](#81-single-primary-write-latency) | Cross-region write p95 | > 200ms | Regional Redis for user state | 2-3 sprints |
| [8.2](#82-single-primary-availability) | Write availability, failover RTO | RTO > 5 min or SLA > 99.5% | Managed PG failover → standby → multi-primary | 1 → 2 → 6+ sprints |
| [8.3](#83-single-region-clusters) | Customer demand | HA across failure domains requested | Federation layer over single-region clusters | 4-6 sprints |
| [8.4](#84-manual-region-selection) | Gateway latency from distance | p95 > 100ms or 3+ regions active | GeoDNS → edge gateways → edge compute | 1 → 2-3 → 4+ sprints |
| [8.5](#85-centralized-session-auth) | Session validation latency | > 50ms per request or > 500 RPS | Regional Redis session cache | 1-2 sprints |
***
### 8.1 Single-Primary Write Latency
**What you'll see:** [NFR-03](#nfr-03) cross-region write latency for user settings and notification reads climbing toward 200ms p95. UI feels sluggish on interactions that trigger writes. Customers in distant regions experience noticeably worse responsiveness than those near the mothership.
**Why it happens:** All writes go to the single PG primary in the mothership region. A user in `ams1` marking a notification as read incurs a transatlantic round-trip. This is acceptable at 20-80ms but degrades as regions get farther from the primary or write volume increases.
**What to build:** Regional Redis (or Turso/SQLite) for user-scoped state — notification read status, user preferences, UI state. These writes hit the local store immediately and async-sync to central PG. Infrastructure state (clusters, servers, org config) stays in central PG where consistency matters. **Effort: 2-3 sprints.** Requires defining consistency model per data type, deploying regional Redis, and implementing sync workers.
**Dependencies:** None. Can be deployed independently per region.
***
### 8.2 Single-Primary Availability
**What you'll see:** [NFR-13](#nfr-13) mothership primary goes down. All writes across all regions fail immediately. Reads continue from replicas. Duration depends on recovery — manual intervention could take 10-30 minutes. This is the architecture's single biggest risk.
**Why it happens:** One PG primary handles all writes. No standby, no automatic failover. The managed database service provides backups but not instant promotion.
**What to build (staged):**
**Stage 1 — Managed failover.** Enable Multi-AZ on managed PostgreSQL. Standby in a second availability zone within the mothership region. Automatic failover in 60-120s. No application code changes. **Effort: 1 sprint** (infrastructure configuration).
**Stage 2 — Cross-region standby.** Promote a replica in a second region to synchronous standby. If mothership fails, DNS update points writes to the standby. Requires connection drain and cache invalidation. **Effort: 2 sprints.**
**Stage 3 — Multi-primary.** CockroachDB, Spanner, or PostgreSQL BDR. Local writes in every region. Only justified by contractual SLA requirements from very large customers. **Effort: 6+ sprints** with dedicated database engineering.
**Dependencies:** Stage 1 is independent. Stage 2 requires monitoring from [NFR-19](#nfr-19). Stage 3 requires significant application changes.
***
### 8.3 Single-Region Clusters
**What you'll see:** Customers ask for high-availability workloads that survive a full region failure. Competitors offer multi-region Kubernetes. Sales loses deals where cross-region resilience is a hard requirement.
**Why it happens:** A cluster lives in exactly one region ([FR-08](#fr-08) embeds region in the cluster ID). All nodes are co-located. If `sfo1` goes down, clusters in `sfo1` are down.
**What to build:** A federation layer. A "multi-region deployment" is a logical resource that owns multiple single-region clusters (e.g., one in `sfo1`, one in `lax1`). The routing architecture doesn't change — each underlying cluster still has a single region. The federation layer handles cross-cluster orchestration, health monitoring, and failover. **Effort: 4-6 sprints.**
**Dependencies:** Requires [§8.4](#84-manual-region-selection) GeoDNS for meaningful cross-region traffic steering. Requires fat-pipe interconnects between DCs for viable cross-region networking.
***
### 8.4 Manual Region Selection
**What you'll see:** `api.example.com` resolves to a single gateway (or round-robin). Users in Amsterdam hit a US gateway before being routed to `ams1`. [NFR-02](#nfr-02) read latency inflated by unnecessary cross-region hop.
**Why it happens:** No geographic DNS routing. `api.example.com` points to one place. Developers pick their region via subdomain, header, or SDK config — there's no automatic "nearest region" behavior.
**What to build (staged):**
**Stage 1 — GeoDNS.** Route53 latency-based routing or Cloudflare load balancing. `api.example.com` resolves to the nearest healthy regional gateway. DNS configuration only, no code changes. **Effort: 1 sprint.**
**Stage 2 — Edge gateways.** Lightweight gateways at CDN edge (Cloudflare Workers, CloudFront Functions). TLS termination + region resolution at edge. Reduces first-byte latency. **Effort: 2-3 sprints.**
**Stage 3 — Edge compute.** Move read-heavy operations (list caches, notification counts) to edge. Requires cache invalidation design. **Effort: 4+ sprints.**
**Dependencies:** Stage 1 requires 3+ active regions to be meaningful. Stage 2 requires [NFR-19](#nfr-19) observability to measure improvement. Stage 3 pairs with [§8.1](#81-single-primary-write-latency) regional state.
***
### 8.5 Centralized Session Auth
**What you'll see:** Session validation adds measurable latency to every request because the auth DB is in the mothership region. At scale, the auth service becomes a throughput bottleneck.
**Why it happens:** Stateful sessions live in mothership PG. Every request that needs session validation from a non-mothership region pays a cross-region round-trip.
**What to build:** Regional Redis session cache, populated on login, validated locally. Session revocation propagated via pub/sub with \< 5s delay. This uses the same regional Redis infrastructure as [§8.1](#81-single-primary-write-latency) — same deployment, same operational cost. **Effort: 1-2 sprints** if regional Redis already exists from 8.1.
**Dependencies:** Pairs naturally with [§8.1](#81-single-primary-write-latency). Deploy together for shared infrastructure cost.
***
### 8.6 Phase Summary
```
Phase 1 — MVP
├── Single PG primary, regional read replicas
├── Manual region selection (subdomain/header/SDK)
├── Single-region clusters only
└── Monitor: write latency, replication lag, availability
Phase 2 — Regional Performance (triggered by latency NFRs)
├── Regional Redis for user state (8.1)
├── Regional session cache (8.5)
├── GeoDNS (8.4 stage 1)
└── Managed PG failover (8.2 stage 1)
Phase 3 — Scale & Resilience (triggered by SLA requirements)
├── Cross-region PG standby (8.2 stage 2)
├── Edge gateways (8.4 stage 2)
└── Multi-region federation (8.3)
Phase 4 — Global Platform (triggered by global customer base)
├── Multi-primary database (8.2 stage 3)
├── Edge compute (8.4 stage 3)
└── Cross-region clusters (8.3 extension)
```
Each phase triggered by measurable thresholds, not calendar dates. Observability ([NFR-19](#nfr-19) through [NFR-23](#nfr-23)) provides the data to know when.
***
## 9. Decision Summary
| Decision | Choice | Reference |
| ------------------------------------ | ---------------------------------------------------------------- | ---------------------------------------------------------- |
| Region code format | IATA + number → `sfo1`, `lax2` | [FR-17](#fr-17) |
| Physical topology (zone, rack, cage) | Metadata on server records, not routing | [FR-19](#fr-19) |
| Subdomain format | `sfo1.api.example.com` | [§1.4](#14-subdomain-format) |
| API prefix | `/v1/{service}` → `/v1/region/global`, `/v1/region/{region}` | [§2.2](#22-atlas-api-operator-console) |
| Region in API path | Never | [FR-06](#fr-06) |
| Resource ID format | `{prefix}_{base62}` (26 chars, opaque) | [§2.2](#22-resource-ids), [FR-08](#fr-08) |
| Request ID format | `req_{region}-{timestamp}-{entropy}` | [§2.1](#21-request-ids), [FR-09](#fr-09) |
| Database topology (MVP) | Single primary + regional read replicas | [§3](#3-database-topology), [NFR-03](#nfr-03) |
| Resolution priority | Subdomain → Header → Query → Body → Session → DB → Fan-out/Error | [FR-02](#fr-02) |
| Cross-region clusters | Not for MVP | [§8.3](#83-single-region-clusters) |
| Write availability risk | Accepted for MVP, managed failover first | [NFR-13](#nfr-13), [§8.2](#82-single-primary-availability) |
| Monitoring baseline | OTel on every request from day one | [NFR-19](#nfr-19) |
***
## Appendix A: All Functional Requirements
### Region Resolution
| ID | Requirement | Rationale |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| FR-01 | Resolve target region for every request before it reaches application code. | Routing is infrastructure, not application logic. |
| FR-02 | Follow deterministic priority: Subdomain → Header → Query → Body → Session → DB Lookup → Fan-out/Error. | Developers must predict where requests go. |
| FR-03 | Mutations without a resolved region return `400`, never silently default. | Wrong-region creates are infrastructure incidents. |
| FR-04 | List operations without a region fan out to all org-accessible regions. | `GET /clusters` returns all clusters, not a random subset. |
| FR-05 | Resolution adds \< 5ms overhead. | It's a parse, not a network call. |
### API Contract
| ID | Requirement | Rationale |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- |
| FR-06 | Region never appears as a path segment. | Decouples API contract from topology changes. |
| FR-07 | OpenAPI spec is identical across all regional gateways. | One SDK, one set of docs. |
| FR-08 | Resource IDs are globally unique and opaque (`{prefix}_{base62}`, 26 chars). Region stored as metadata. | Enables resource migration between regions without ID changes. |
| FR-09 | Request IDs embed region, timestamp, and entropy (`req_{region}-{ts}-{entropy}`). | Enables grep-based debugging and distributed tracing. |
| FR-10-ORG | Organization resolved from session, never from URL. | Prevents enumeration, simplifies API surface. |
| FR-10 | Single-region org users never need to specify a region. | The DX bar. If they think about regions, we failed. |
### Multi-Tenancy
| ID | Requirement | Rationale |
| ---------------------------------- | ------------------------------------------------------------------------------------ | ---------------------------------------- |
| FR-11 | Arc requests only return resources for the authenticated org, regardless of gateway. | Tenant isolation is non-negotiable. |
| FR-12 | Resource creation rejected if target region not in org's allowed set. | API enforces region access, not just UI. |
| FR-13 | Atlas requests require platform-level auth, enforced at gateway before routing. | Atlas exposes cross-org data. |
### Data Consistency
| ID | Requirement | Rationale |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- |
| FR-14 | Infrastructure mutations go through the workflow orchestrator to primary DB. | Eventual consistency on "does this cluster exist" is not acceptable. |
| FR-15 | Read replicas serve Arc list/get operations. Reads don't hit primary unless read-after-write is needed. | Local read latency is the point of replicas. |
| FR-16 | After creation, resource visible in same-session lists within 5 seconds. | Read-after-write for the creating user. |
### Naming
| ID | Requirement | Rationale |
| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------- |
| FR-17 | Region codes use IATA + number format (e.g., `sfo1`, `lax2`). | Industry standard for physical infrastructure, instantly recognizable. |
| FR-18 | Same region string used in subdomains, IDs, DB columns, K8s labels, and logs. | One code everywhere, no translation tables. |
| FR-19 | Physical topology below region (zone, rack, cage) is metadata on resources, not part of the routing hierarchy or API contract. | Provider topology changes don't break API contracts. |
***
## Appendix B: All Non-Functional Requirements
### Latency
| ID | Metric | MVP | Production | Measured At |
| ------------------------------------ | -------------------------------------------- | ------------ | ------------ | ---------------------------------- |
| NFR-01 | Gateway resolution overhead | \< 5ms p99 | \< 2ms p99 | Request arrival → region resolved |
| NFR-02 | Same-region read (Arc list/get) | \< 100ms p95 | \< 50ms p95 | Gateway → response, local replica |
| NFR-03 | Cross-region write (settings, notifications) | \< 150ms p95 | \< 100ms p95 | Round-trip including primary write |
| NFR-04 | Infrastructure mutation acceptance | \< 500ms p95 | \< 300ms p95 | Time to `202 Accepted` + enqueue |
| NFR-05 | Fan-out list (all org regions) | \< 500ms p95 | \< 250ms p95 | Parallel query + merge |
| NFR-06 | Replication lag | \< 5s p99 | \< 1s p99 | pg\_stat\_replication |
### Availability
| ID | Metric | MVP | Production |
| ------------------------------------ | ----------------------------------- | ------------- | ------------- |
| NFR-07 | Per-region reads | 99.5% monthly | 99.9% monthly |
| NFR-08 | Per-region writes | 99.0% monthly | 99.5% monthly |
| NFR-09 | Global endpoint (`api.example.com`) | 99.5% monthly | 99.9% monthly |
| NFR-10 | Mothership availability | 99.5% monthly | 99.9% monthly |
### Failover
| ID | Scenario | MVP Behavior | Production Behavior |
| ------------------------------------ | --------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| NFR-11 | Regional gateway down | `{region}.api.example.com` returns 503. Global endpoint routes to healthy regions. | Automatic DNS failover within 60s. |
| NFR-12 | Regional replica down | Reads fall back to primary (higher latency). | Automatic replica promotion within 30s. |
| NFR-13 | Primary DB down | **All writes fail.** Reads continue from replicas. API returns `503` with `Retry-After`. | Managed failover restores writes within 60-120s. RTO \< 5 min. |
| NFR-14 | Mesh partition | Local reads continue. Writes fail with clear error. No silent data loss. | Regional write buffer retries for 5 min, then fails with audit trail. |
| NFR-15 | Degraded mode | Responses include `X-Region`, `X-Request-Id`. Degraded responses add `X-Degraded: true` + reason. | Same, plus `GET /health/region` with replica lag, primary connectivity, uptime. |
### Throughput
| ID | Metric | MVP | Production |
| ------------------------------- | ------------------------ | ----------------------------- | --------------- |
| NFR-16 | RPS per regional gateway | 100 sustained | 1,000 sustained |
| NFR-17 | Concurrent fan-out ops | 10 | 100 |
| NFR-18 | Region registry lookup | In-memory, zero network calls | Same |
### Observability
| ID | Requirement | MVP | Production |
| ------------------------------------ | ------------------------------------------------------------------------------------------------------------- | ------------ | ---------- |
| NFR-19 | Every request emits OTel span: `region`, `region_source`, `request_id`, `org_id`, `latency_ms`, `status_code` | Required | Required |
| NFR-20 | Cross-region write latency tracked separately from same-region reads, per-region p50/p95/p99 | Required | Required |
| NFR-21 | Replication lag monitored per replica, alert at 5s (MVP) / 1s (prod) | Required | Required |
| NFR-22 | Fan-out tracks per-region sub-request latency | Nice-to-have | Required |
| NFR-23 | Gateway exposes Prometheus `/metrics`: resolution latency, routing source distribution, error rates | Required | Required |