> ## Documentation Index
> Fetch the complete documentation index at: https://team.k0labs.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Standards

> API design conventions, naming conventions, and response envelope standards

import { Accordion, Accordions } from 'fumadocs-ui/components/accordion'
import { Callout } from 'fumadocs-ui/components/callout'

This document defines the foundational standards for all k0rdent APIs: request/response structure, naming conventions, field design, and operational patterns across Atlas, Arc, and shared services.

<Callout type="warn">
  **Draft:** This documentation is currently a work in progress and subject to change.
</Callout>

<Callout type="info">
  **Quick Navigation:** - For complete endpoint implementations, see [API
  Specifications](/docs/specification) - For code examples and patterns, see
  [Data Ownership](/docs/data-ownership) - For authentication and security, see
  [Auth](/docs/auth)
</Callout>

### Table of Contents

* [API Response Contract](#api-response-contract)
* [Field Design Rules](#field-design-rules)
* [Implementation Reference](#implementation-reference)
* [Naming Conventions](#naming-conventions)
* [Action Endpoints](#action-endpoints)
* [Bulk Operations](#bulk-operations)
* [Error Handling](#error-handling)
* [Audit Logging](#audit-logging)
* [Multi-Tenancy Patterns](#multi-tenancy-patterns)

***

## API Response Contract

All API responses use a consistent discriminated union envelope structure with a `success` boolean discriminator.

### Type Definition

```typescript theme={null}
// Base response wrapper with discriminated union
type ApiResponse<T, M = {}> =
  | { success: true; data: T; meta: BaseMeta & M } // Success with extendable meta
  | { success: false; error: ApiErrorBody; meta: BaseMeta } // Error with fixed meta
```

### Base Meta Object

The `meta` object is always present and contains request tracking information:

```typescript theme={null}
// Base meta - always present in all responses
interface BaseMeta {
  requestId: string // Required: for log correlation
  timestamp: string // Required: ISO 8601, for distributed debugging
}
```

<Callout type="info">
  **Design Decision:** `requestId` and `timestamp` are always included because
  they provide negligible overhead while being critical for distributed
  debugging and log correlation across services.
</Callout>

### Extension Meta Types

Success responses can extend meta with additional context:

```typescript theme={null}
// Pagination meta (for list responses)
interface PaginationMeta {
  pagination: {
    total: number
    page: number
    pageSize: number
    hasMore: boolean
    nextCursor?: string // For cursor-based pagination
  }
}

// Workflow meta (for long-running tasks via Trigger.dev)
interface WorkflowMeta {
  workflowId?: string // Trigger.dev run ID for polling status
  estimatedDuration?: number // Hint for UI progress (ms)
}

// Debug meta (development/troubleshooting)
interface DebugMeta {
  duration: number // Request duration in milliseconds
}

// Deprecation warnings
interface DeprecationMeta {
  warnings: DeprecationWarning[]
}

interface DeprecationWarning {
  code: 'DEPRECATED_FIELD' | 'DEPRECATED_ENDPOINT'
  message: string
  field?: string
  sunset?: string // ISO 8601 date when feature will be removed
  migration?: string // URL to migration guide
}
```

### Error Response Structure

```typescript theme={null}
interface ApiErrorBody {
  code: string // Machine-readable: "VALIDATION_ERROR", "NOT_FOUND"
  message: string // Human-readable description
  details?: Record<string, unknown> // Additional error context
}
```

### Response Examples

<Accordions>
  <Accordion title="Basic Success Response">
    ```typescript theme={null}
    interface Server { 
      id: string
      name: string
      state: string
    }

    type GetServerResponse = ApiResponse<Server>

    const response: GetServerResponse = {
    success: true,
      data: {
        id: "srv_abc123",
        name: "compute-node-a7f3e2",
        state: "available"
      },
      meta: {
        requestId: "kl4c6-1766196422377-0f705e3ef475",
        timestamp: "2025-01-09T12:00:00.000Z"
      }
    }

    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="List Response with Pagination">
    ```typescript theme={null}
    type ListServersResponse = ApiResponse<Server[], PaginationMeta>

    const response: ListServersResponse = {
      success: true,
      data: [
        { id: "srv_123", name: "node-01", state: "available" },
        { id: "srv_456", name: "node-02", state: "allocated" }
      ],
      meta: {
        requestId: "kl4c6-1766196422377-0f705e3ef475",
        timestamp: "2025-01-09T12:00:00.000Z",
        pagination: {
          total: 156,
          page: 1,
          pageSize: 25,
          hasMore: true,
          nextCursor: "srv_456"
        }
      }
    }
    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Workflow Operation Response">
    ```typescript theme={null}
    type ProvisionServerResponse = ApiResponse<Server, WorkflowMeta>

    const response: ProvisionServerResponse = {
      success: true,
      data: {
        id: "srv_abc123",
        name: "compute-node-a7f3e2",
        state: "provisioning"
      },
      meta: {
        requestId: "kl4c6-1766196422377-0f705e3ef475",
        timestamp: "2025-01-09T12:00:00.000Z",
        workflowId: "run_deploy_789",
        estimatedDuration: 300000 // 5 minutes in ms
      }
    }

    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Error Response">
    ```typescript theme={null}
    const errorResponse: GetServerResponse = {
      success: false,
      error: {
        code: "VALIDATION_ERROR",
        message: "Invalid request body",
        details: {
          fields: {
            name: "Name is required",
            bmcAddress: "Invalid BMC address format"
          }
        }
      },
      meta: {
        requestId: "kl4c6-1766196422377-0f705e3ef475",
        timestamp: "2025-01-09T12:00:00.000Z"
      }
    }
    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Response with Deprecation Warning">
    ```json theme={null}
    {
      "success": true,
      "data": {
        "id": "srv_abc123",
        "name": "node-01",
        "bmcAddress": "ipmi://10.0.100.50"
      },
      "meta": {
        "requestId": "kl4c6-1766196422377-0f705e3ef475",
        "timestamp": "2025-01-09T12:00:00.000Z",
        "warnings": [
          {
            "code": "DEPRECATED_FIELD",
            "field": "bmcAddress",
            "message": "bmcAddress is deprecated, use bmc.address instead",
            "sunset": "2026-06-01",
            "migration": "https://docs.example.com/migration/bmc-fields"
          }
        ]
      }
    }
    ```
  </Accordion>
</Accordions>

### Meta Object Design Decisions

| Field        | Decision       | Rationale                                               |
| ------------ | -------------- | ------------------------------------------------------- |
| `requestId`  | Always include | Essential for log correlation across services           |
| `timestamp`  | Always include | Negligible overhead; critical for distributed debugging |
| `apiVersion` | Omit           | URL path (`/v1/`) is the version; redundant in response |
| `rateLimit`  | Add later      | When rate limiting is implemented                       |

***

## Implementation Reference

### Technology Stack

| Layer      | Technology               | Purpose                                |
| ---------- | ------------------------ | -------------------------------------- |
| Framework  | Next.js 16+ (App Router) | Server components, API routes          |
| API Layer  | Hono                     | Lightweight, type-safe API routes      |
| Validation | Zod                      | Runtime validation, schema definitions |
| Database   | Drizzle ORM              | Type-safe queries, migrations          |
| Auth       | BetterAuth               | Session management, OAuth              |
| Workflows  | Trigger.dev              | Durable task execution                 |

### Zod Schema Definitions

Define all request/response schemas using Zod for validation and type inference:

<Accordions>
  <Accordion title="Schema Definition Example">
    ```typescript title="packages/api-schemas/src/atlas/servers.ts" theme={null}
    import { z } from 'zod'

    // Response schemas
    export const serverSchema = z.object({
        id: z.string(),
        name: z.string(),
        hostname: z.string(),
        state: z.enum(['available', 'allocated', 'maintenance', 'failed']),
        tags: z.record(z.string()).optional(),
        infrastructure: z.object({
        provisioningState: z.string(),
        powerState: z.enum(['on', 'off', 'unknown']),
        hardware: z.object({
        cpu: z.object({ model: z.string(), cores: z.number() }),
        memory: z.object({ totalGb: z.number() }),
        createdAt: z.string().datetime(),
      }).optional(),
    })

    export type Server = z.infer<typeof serverSchema>

    // Request body schemas
    export const createServerBodySchema = z.object({
      name: z.string().min(1).max(63),
      hostname: z.string(),
      bmcAddress: z.string().ip(),
      bmcCredentialId: z.string(),
      bootMacAddress: z.string().optional(),
      tags: z.record(z.string()).optional(),
    })

    export type CreateServerBody = z.infer<typeof createServerBodySchema>

    // Query parameter schemas
    export const listServersQuerySchema = z.object({
      state: z.enum(['available', 'allocated', 'maintenance', 'failed']).optional(),
      pool: z.string().optional(),
      limit: z.coerce.number().min(1).max(100).default(20),
      cursor: z.string().optional(),
    })

    export type ListServersQuery = z.infer<typeof listServersQuerySchema>
    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Hono Route Implementation">
    ```typescript title="apps/atlas/app/api/[...route]/route.ts" theme={null}
    import { Hono } from 'hono'
    import { handle } from 'hono/vercel'
    import { zValidator } from '@hono/zod-validator'
    import {
      listServersQuerySchema,
      createServerBodySchema,
      serverSchema,
    } from '@k0rdent/api-schemas'

    const app = new Hono().basePath('/v1')

    // Middleware: Extract auth context
    app.use('\*', async (c, next) => {
    const userId = c.req.header('X-User-ID')
    const orgId = c.req.header('X-Org-ID')
    const requestId = c.req.header('X-Request-ID') || generateRequestId()

    c.set('userId', userId)
    c.set('orgId', orgId)
    c.set('requestId', requestId)

    await next()
    })

    // List servers
    app.get('/servers', zValidator('query', listServersQuerySchema), async (c) => {
    const { state, pool, limit, cursor } = c.req.valid('query')
    const requestId = c.get('requestId')

    const servers = await db.query.servers.findMany({
      where: and(
      state ? eq(schema.inventory.state, state) : undefined,
      pool ? eq(schema.inventory.poolId, pool) : undefined
      ),
      limit: limit + 1,
      // cursor-based pagination logic
    })

    const hasMore = servers.length {'>'} limit
    const data = hasMore ? servers.slice(0, -1) : servers

    return c.json({
      success: true,
      data,
      meta: {
        requestId,
        timestamp: new Date().toISOString(),
        pagination: {
          total: await getTotal(state, pool),
          page: 1,
          pageSize: limit,
          hasMore,
          nextCursor: hasMore ? encodeCursor(data[data.length - 1]) : undefined,
        },
      },
      })
    })

    // Create server (async workflow)
    app.post('/servers', zValidator('json', createServerBodySchema), async (c) => {
    const body = c.req.valid('json')
    const userId = c.get('userId')
    const requestId = c.get('requestId')

    // Create server record
    const server = await db.insert(schema.servers)
    .values({
      name: body.name,
      hostname: body.hostname,
      bmcAddress: body.bmcAddress,
      bmcCredentialId: body.bmcCredentialId,
    })
    .returning()

    // Trigger async registration workflow
    const workflow = await tasks.trigger('server-registration', {
      serverId: server[0].id,
      triggeredBy: userId,
    })

    return c.json({
      success: true,
      data: {
        id: server[0].id,
        name: server[0].name,
        state: 'registering',
      },
      meta: {
        requestId,
        timestamp: new Date().toISOString(),
        operationId: workflow.id,
        estimatedDuration: 120000, // 2 minutes
      },
      }, 202)
    })

    export const GET = handle(app)
    export const POST = handle(app)
    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Response Helper Functions">
    ```typescript title="lib/api/response.ts" theme={null}
    import { Context } from 'hono'

    export function success<T>(
      c: Context,
      data: T,
      meta?: Record<string, unknown>
    ) {
      return c.json({
        success: true,
        data,
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
          ...meta,
        },
      })
    }

    export function error(
      c: Context,
      code: string,
      message: string,
      status: number = 400,
      details?: Record<string, unknown>
    ) {
      c.set('errorCode', code)  // For audit middleware
      return c.json({
        success: false,
        error: { code, message, details },
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
        },
      }, status)
    }

    export function paginated<T>(
      c: Context,
      data: T[],
      pagination: {
        total: number
        page: number
        pageSize: number
        hasMore: boolean
        nextCursor?: string
      }
    ) {
      return c.json({
        success: true,
        data,
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
          pagination,
        },
      })
    }
    ```
  </Accordion>
</Accordions>

<Callout type="info">
  **Pattern: Workflows by Default.** All operations that interact with
  infrastructure (BMC, K8s) are executed as Trigger.dev workflows and return
  immediately with a workflow run ID. Clients poll workflow status via the
  Workflows API or receive webhooks.
</Callout>

***

## Field Design Rules

Design fields for extension from day one. The cost of refactoring primitive fields into objects later is high and often requires breaking changes.

<Callout type="warn">
  **Principle: Objects Over Primitives.** Always wrap values that might grow
  into structured objects. It's better to have nested objects early than to
  break APIs later when you need to add context.
</Callout>

### Use Objects Over Primitives

Wrap values that might grow into objects immediately.

```typescript theme={null}
// ❌ Will require breaking change
interface Server {
  status: 'available' | 'provisioning' | 'error'
  bmcAddress: string
}

// ✅ Extensible
interface Server {
  status: {
    state: 'available' | 'provisioning' | 'error'
    reason?: string
    since?: string
    conditions?: Condition[]
  }
  bmc: {
    address: string
    protocol?: 'ipmi' | 'redfish'
    vendor?: string
  }
}
```

### Never Use Booleans for State

States often grow beyond two values.

```typescript theme={null}
// ❌ Trouble waiting to happen
interface Server {
  isOnline: boolean
  isHealthy: boolean
}

// ✅ Extensible
interface Server {
  power: {
    state: 'on' | 'off' | 'unknown'
  }
  health: {
    state: 'healthy' | 'degraded' | 'unhealthy' | 'unknown'
  }
}
```

### Use IDs with Optional Expansion

Don't embed full objects. Use IDs and let clients request expansion.

```json theme={null}
// ❌ Embedded - can't change cardinality later
{
  "id": "srv_123",
  "cluster": {
    "id": "cls_456",
    "name": "prod-cluster"
  }
}
```

```json title="GET /v1/compute/servers/srv_123" theme={null}
// ✅ ID reference
{
  "id": "srv_123",
  "clusterId": "cls_456"
}
```

```json title="GET /v1/compute/servers/srv_123?expand=cluster" theme={null}
// ✅ Optional expansion
{
  "id": "srv_123",
  "clusterId": "cls_456",
  "cluster": {
    "id": "cls_456",
    "name": "prod-cluster"
  }
}
```

***

## API Versioning

All API endpoints include version in the URL path: `/v1/...`

Domain-based routing separates Atlas and Arc APIs:

### Version Format

```http theme={null}
# Atlas APIs (api.internal.example.com)
https://api.internal.example.com/v1/region/global/compute/servers
https://api.internal.example.com/v1/region/global/compute/clusters
https://api.internal.example.com/v1/region/global/organizations

# Arc APIs (api.example.com)
https://api.example.com/v1/region/{region}/projects
https://api.example.com/v1/region/{region}/compute/clusters
https://api.example.com/v1/region/{region}/stacks

# Shared Services (both domains)
/v1/region/global/auth
/v1/region/global/notifications
/v1/region/global/webhooks
```

### Version Policy

* **Major versions** (`v1`, `v2`) for breaking changes
* **No minor versions** in URL - use feature flags and deprecation warnings instead
* **Deprecation timeline:** 6 months notice before removing deprecated endpoints
* **Version in URL, not response:** The `apiVersion` field is omitted from responses because the URL path is the source of truth

<Callout type="info">
  When deprecating fields or endpoints, include warnings in the `meta.warnings`
  array with sunset dates and migration guides.
</Callout>

### Deprecation Example

```json theme={null}
{
  "success": true,
  "data": { ... },
  "meta": {
    "requestId": "...",
    "timestamp": "...",
    "warnings": [
      {
        "code": "DEPRECATED_ENDPOINT",
        "message": "This endpoint is deprecated. Use /v2/servers instead.",
        "sunset": "2026-06-01",
        "migration": "https://docs.example.com/migration/v2-servers"
      }
    ]
  }
}
```

***

## Naming Conventions

Consistent naming across URLs, fields, and resources improves developer experience and reduces confusion.

### URL Paths

Hono uses colon-prefixed route parameters.

| Rule                         | Example                   |
| ---------------------------- | ------------------------- |
| Route params with colon      | `/v1/clusters/:clusterId` |
| Lowercase, hyphenated        | `/v1/ai-services`         |
| Plural nouns for collections | `/servers`, `/clusters`   |
| Singular for singletons      | `/me`, `/health`          |

> NOTE: Lowercase, hyphenated is only for url paths and not the same for the database, response, request body, and other contexts.

### Resource IDs

Resources (clusters, servers, organizations, etc.) get globally unique, opaque IDs that do NOT contain region information. This decouples resource identity from physical location.

**Format:** `{prefix}_{base62}`

| Resource     | Prefix | Example                            |
| ------------ | ------ | ---------------------------------- |
| Organization | `org_` | `org_8TcVx2WkZddNmK3Pt9JwX7BzWrLM` |
| Server       | `srv_` | `srv_3KpQm9WnXccFjH2Ls8DkT6VzRqYU` |
| Cluster      | `cls_` | `cls_6NZtkvWLBbbmHfPi7L6oz7KZpqET` |
| Stack        | `stk_` | `stk_5MfRp4WjYbbHmG8Nt2LvS9CxPqZK` |
| Workflow Run | `run_` | `run_7NhTq6WlAbbKmF5Rt3MxU8DzSqWJ` |
| Pool         | `poo_` | `poo_2LgPn8WmXccGjE7Mt4KwV9BySrTL` |
| Allocation   | `all_` | `all_9QjSr3WnZddMmH6Pt5LxW2CzUrYK` |
| API Key      | `key_` | `key_4KfQm7WkYccJmG3Nt8MvX9BzSqWL` |
| Event        | `evt_` | `evt_6MgRp2WlXbbKmF9Rt5NxU3DzTqZJ` |

**Special ID:** `org_system` is reserved for platform-level admin operations. TBD if this is needed still. Originally it was for something else.

### Field Names

| Rule                          | Example                               |
| ----------------------------- | ------------------------------------- |
| camelCase                     | `createdAt`, `nodeCount`              |
| Suffix IDs with `Id`          | `clusterId`, `organizationId`         |
| Use past tense for timestamps | `createdAt`, `updatedAt`, `deletedAt` |

***

## Pagination

Use cursor-based pagination for real-time data, offset/limit for stable datasets.

### Query Parameters

```http theme={null}
GET /v1/compute/servers?limit=50&offset=0
GET /v1/compute/servers?limit=50&cursor=srv_abc123
```

### Response

```typescript theme={null}
interface PaginatedResponse<T> {
  success: true
  data: T[]
  meta: {
    requestId: string
    timestamp: string
    pagination: {
      total: number
      limit: number
      offset?: number
      cursor?: string
      nextCursor?: string
      hasMore: boolean
    }
  }
}
```

***

## Filtering and Sorting

### Query String Format

Use consistent query parameter patterns for filtering and sorting:

```http theme={null}
GET /v1/compute/servers?status=available&sort=-createdAt&limit=50
```

| Parameter       | Format                | Example                         |
| --------------- | --------------------- | ------------------------------- |
| Filter          | `field=value`         | `status=available`              |
| Multiple values | `field=val1,val2`     | `status=available,provisioning` |
| Sort ascending  | `sort=field`          | `sort=name`                     |
| Sort descending | `sort=-field`         | `sort=-createdAt`               |
| Multiple sorts  | `sort=field1,-field2` | `sort=status,-createdAt`        |

### Implementation with Zod

```typescript theme={null}
export const listQuerySchema = z.object({
  // Filtering
  status: z.string().optional(),
  type: z.string().optional(),

  // Sorting
  sort: z.string().optional(),

  // Pagination
  limit: z.coerce.number().min(1).max(100).default(25),
  offset: z.coerce.number().min(0).default(0),
  cursor: z.string().optional(),
})
```

***

## Action Endpoints

For operations beyond CRUD, use a unified action endpoint with POST method. Actions represent commands that change resource state asynchronously.

### Design Principles

1. **Unified Endpoint:** Single `/actions` endpoint handles all action types (power, provision, deprovision, inspect, maintenance)
2. **Type-Safe Parameters:** Each action type has its own request schema with action-specific options
3. **Async by Default:** Actions return `202 Accepted` with workflow/operation IDs for tracking
4. **Audit Trail:** Logs show "POST /actions with type=power action=off" for clear tracking
5. **Granular Permissions:** Easy to scope permissions like `servers:lifecycle` vs `servers:update`

### Endpoint Pattern

```http theme={null}
POST /v1/compute/servers/:serverId/actions     # Actions: power, provision, deprovision, inspect, maintenance
```

### Action Request Schema

```typescript theme={null}
// Base action schema with discriminated union for type safety
export const serverActionBodySchema = z.discriminatedUnion('type', [
  // Power actions
  z.object({
    type: z.literal('power'),
    action: z.enum(['on', 'off', 'reboot', 'cycle']),
    force: z.boolean().default(false),
  }),

  // Provision actions
  z.object({
    type: z.literal('provision'),
    imageUrl: z.string().url(),
    imageChecksum: z.string().optional(),
    checksumType: z.enum(['md5', 'sha256', 'sha512']).optional(),
    rootDeviceHints: z
      .object({
        deviceName: z.string().optional(),
        minSizeGiB: z.number().optional(),
      })
      .optional(),
  }),

  // Deprovision actions
  z.object({
    type: z.literal('deprovision'),
    wipeDisks: z.boolean().default(true),
  }),

  // Inspect actions
  z.object({
    type: z.literal('inspect'),
    full: z.boolean().default(false),
  }),

  // Maintenance mode actions
  z.object({
    type: z.literal('maintenance'),
    enabled: z.boolean(),
    reason: z.string().optional(),
  }),
])
```

### Implementation Example

<Accordions>
  <Accordion title="Unified Actions Endpoint">
    ```typescript theme={null}
    app.post(
      '/servers/:id/actions',
      zValidator('json', serverActionBodySchema),
      async (c) => {
        const { id } = c.req.param()
        const body = c.req.valid('json')
        const userId = c.get('userId')
        const requestId = c.get('requestId')

        // Validate server exists
        const server = await db.query.servers.findFirst({
          where: eq(schema.servers.id, id),
          with: { cache: true, inventory: true },
        })

        if (!server) {
          return c.json({
            success: false,
            error: {
              code: 'NOT_FOUND',
              message: 'Server not found',
            },
            meta: {
              requestId,
              timestamp: new Date().toISOString(),
            },
          }, 404)
        }

        // Handle different action types
        switch (body.type) {
          case 'power': {
            // Trigger power action workflow
            const workflow = await tasks.trigger('server-power-action', {
              serverId: id,
              action: body.action,
              force: body.force,
              triggeredBy: userId,
            })

            return c.json({
              success: true,
              data: {
                serverId: id,
                actionType: 'power',
                action: body.action,
                previousState: server.cache?.powerState || 'unknown',
                targetState: body.action === 'off' ? 'off' : 'on',
              },
              meta: {
                requestId,
                timestamp: new Date().toISOString(),
                workflowId: workflow.id,
                estimatedDuration: 30000, // 30 seconds
              },
            }, 202)
          }

          case 'provision': {
            // Validate state transition
            if (server.inventory.state !== 'available') {
              return c.json({
                success: false,
                error: {
                  code: 'INVALID_STATE_TRANSITION',
                  message: `Cannot provision server in '${server.inventory.state}' state`,
                  details: {
                    currentState: server.inventory.state,
                    requiredState: 'available',
                  },
                },
                meta: {
                  requestId,
                  timestamp: new Date().toISOString(),
                },
              }, 409)
            }

            // Update state immediately
            await db.update(schema.inventory)
              .set({ state: 'provisioning' })
              .where(eq(schema.inventory.id, server.inventory.id))

            // Trigger provision workflow
            const workflow = await tasks.trigger('server-provision', {
              serverId: id,
              imageUrl: body.imageUrl,
              imageChecksum: body.imageChecksum,
              checksumType: body.checksumType,
              rootDeviceHints: body.rootDeviceHints,
              triggeredBy: userId,
            })

            return c.json({
              success: true,
              data: {
                serverId: id,
                actionType: 'provision',
                previousStatus: 'available',
                targetStatus: 'provisioned',
                image: extractImageName(body.imageUrl),
              },
              meta: {
                requestId,
                timestamp: new Date().toISOString(),
                workflowId: workflow.id,
                estimatedDuration: 300000, // 5 minutes
              },
            }, 202)
          }

          case 'deprovision': {
            // Trigger deprovision workflow
            const workflow = await tasks.trigger('server-deprovision', {
              serverId: id,
              wipeDisks: body.wipeDisks,
              triggeredBy: userId,
            })

            return c.json({
              success: true,
              data: {
                serverId: id,
                actionType: 'deprovision',
                wipeDisks: body.wipeDisks,
              },
              meta: {
                requestId,
                timestamp: new Date().toISOString(),
                workflowId: workflow.id,
                estimatedDuration: 180000, // 3 minutes
              },
            }, 202)
          }

          case 'inspect': {
            // Trigger inspect workflow
            const workflow = await tasks.trigger('server-inspect', {
              serverId: id,
              full: body.full,
              triggeredBy: userId,
            })

            return c.json({
              success: true,
              data: {
                serverId: id,
                actionType: 'inspect',
                full: body.full,
              },
              meta: {
                requestId,
                timestamp: new Date().toISOString(),
                workflowId: workflow.id,
                estimatedDuration: 60000, // 1 minute
              },
            }, 202)
          }

          case 'maintenance': {
            // Update maintenance mode immediately
            await db.update(schema.servers)
              .set({
                maintenanceMode: body.enabled,
                maintenanceReason: body.reason,
              })
              .where(eq(schema.servers.id, id))

            return c.json({
              success: true,
              data: {
                serverId: id,
                actionType: 'maintenance',
                enabled: body.enabled,
                reason: body.reason,
              },
              meta: {
                requestId,
                timestamp: new Date().toISOString(),
              },
            }, 200)
          }
        }

    }
    )

    ```
  </Accordion>
</Accordions>

***

## Bulk Operations

Bulk operations allow applying actions to multiple resources simultaneously. All bulk actions use **partial success semantics** - individual resource failures do not fail the entire bulk operation.

### Design Principles

1. **Partial Success:** Individual failures don't abort the entire bulk operation
2. **Explicit IDs:** Use explicit ID lists for predictability and safety
3. **Per-Resource Results:** Response includes success/failure status for each resource
4. **207 Multi-Status:** Always return 207 to indicate mixed results possible
5. **Dedicated Endpoints:** Use `/bulk` pattern for consistency

### Endpoint Pattern

```http theme={null}
POST /v1/compute/servers/bulk                     # Atlas API
POST /v1/notifications/inbox/bulk          # Shared service
POST /v1/webhooks/subscriptions/bulk       # Shared service
```

The action type is specified in the request body, making the API flexible and maintainable.

### Request Schema

```typescript theme={null}
export const bulkRequestSchema = z.object({
  // Action type
  action: z.enum([
    'register',
    'power',
    'provision',
    'deprovision',
    'delete',
  ]),

  // Explicit resource IDs
  ids: z.array(z.string()).min(1).max(1000),

  // Optional dry-run mode
  dryRun: z.boolean().optional(),

  // Action-specific configuration
  params: z.record(z.unknown()).optional(),
})
```

### Response Structure

```typescript theme={null}
interface BulkOperationResponse {
  success: true // Always true for bulk ops
  data: {
    action: string
    requested: number
    succeeded: number
    failed: number
    dryRun?: boolean
    wouldAffect?: number // Dry-run only
    results: Array<{
      id: string
      status: 'success' | 'failed'
      error?: {
        code: string
        message: string
      }
    }>
  }
  meta: {
    requestId: string
    timestamp: string
  }
}
```

### Implementation Example

<Accordions>
  <Accordion title="Bulk Operation Implementation">
    ```typescript theme={null}
    app.post(
      '/servers/bulk',
      zValidator('json', bulkRequestSchema),
      async (c) => {
        const body = c.req.valid('json')
        const userId = c.get('userId')
        const requestId = c.get('requestId')

        // Resolve target servers
        const servers = await getServersByIds(body.ids)

        if (servers.length === 0) {
          return error(c, 'NO_SERVERS_FOUND', 'No servers found', 404)
        }

        // Dry-run mode - preview only
        if (body.dryRun) {
          return c.json({
            success: true,
            data: {
              action: body.action,
              dryRun: true,
              wouldAffect: servers.length,
              preview: servers.slice(0, 20).map(s => ({
                id: s.id,
                name: s.name,
                state: s.state,
              })),
            },
            meta: {
              requestId,
              timestamp: new Date().toISOString(),
            },
          })
        }

        // Execute action on each server
        const results = await Promise.allSettled(
          servers.map(async (server) => {
            try {
              // Trigger appropriate workflow based on action
              const workflow = await tasks.trigger(`server-${body.action}`, {
                serverId: server.id,
                params: body.params,
                triggeredBy: userId,
              })

              return {
                id: server.id,
                status: 'success' as const,
                workflowId: workflow.id,
              }
            } catch (err) {
              return {
                id: server.id,
                status: 'failed' as const,
                error: {
                  code: err.code || 'INTERNAL_ERROR',
                  message: err.message,
                },
              }
            }
          })
        )

        // Aggregate results
        const succeeded = results.filter(r =>
          r.status === 'fulfilled' && r.value.status === 'success'
        ).length
        const failed = results.length - succeeded

        return c.json({
          success: true,
          data: {
            action: body.action,
            requested: servers.length,
            succeeded,
            failed,
            results: results.map(r =>
              r.status === 'fulfilled' ? r.value : r.reason
            ),
          },
          meta: {
            requestId,
            timestamp: new Date().toISOString(),
          },
        }, 207)
      }
    )
    ```
  </Accordion>
</Accordions>

### Safety Features

#### Dry-Run Mode

Preview which resources would be affected without executing:

```http theme={null}
POST /v1/servers/bulk
Content-Type: application/json
```

```json theme={null}
{
  "action": "power",
  "ids": ["srv_123", "srv_456"],
  "dryRun": true,
  "params": { "action": "off" }
}
```

**Response:**

```json theme={null}
{
  "success": true,
  "data": {
    "action": "power",
    "dryRun": true,
    "wouldAffect": 2,
    "preview": [
      { "id": "srv_123", "name": "node-01", "state": "on" },
      { "id": "srv_456", "name": "node-02", "state": "on" }
    ]
  },
  "meta": {
    "requestId": "kl4c6-1766196422377-0f705e3ef475",
    "timestamp": "2025-01-13T12:00:00.000Z"
  }
}
```

#### Rate Limiting

Bulk operations are throttled to prevent resource overload. Default: 10 requests/min.

***

## Implementation Reference

<Accordions>
  <Accordion title="Hono Response Helpers">
    ```typescript title="lib/api/response.ts" theme={null}
    import { Context } from 'hono';

    export function success<T>(c: Context, data: T, meta?: Partial<ApiMeta>) {
      return c.json({
        success: true,
        data,
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
          ...meta,
        },
      });
    }

    export function error(c: Context, code: string, message: string, status: number = 400) {
      c.set('errorCode', code);  // For audit middleware
      return c.json({
        success: false,
        error: { code, message },
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
        },
      }, status);
    }

    export function paginated<T>(
      c: Context,
      data: T[],
      pagination: PaginationMeta
    ) {
      return c.json({
        success: true,
        data,
        meta: {
          requestId: c.get('requestId'),
          timestamp: new Date().toISOString(),
          pagination,
        },
      });
    }
    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Zod Schema Definitions">
    ```typescript title="lib/api/schemas.ts" theme={null}
    import { z } from 'zod';

    export const apiMetaSchema = z.object({
      requestId: z.string(),
      timestamp: z.string().datetime(),
      pagination: z
        .object({
          total: z.number(),
          limit: z.number(),
          offset: z.number().optional(),
          cursor: z.string().optional(),
          hasMore: z.boolean(),
        })
        .optional(),
      warnings: z
        .array(
          z.object({
            code: z.enum(['DEPRECATED_FIELD', 'DEPRECATED_ENDPOINT']),
            message: z.string(),
            field: z.string().optional(),
            sunset: z.string().optional(),
            migration: z.string().optional(),
          })
        )
        .optional(),
    })

    export const apiErrorSchema = z.object({
      code: z.string(),
      message: z.string(),
      details: z
        .array(
          z.object({
            field: z.string().optional(),
            code: z.string(),
            message: z.string(),
          })
        )
        .optional(),
    })

    export function apiResponseSchema<T extends z.ZodType>(dataSchema: T) {
      return z.discriminatedUnion('success', [
        z.object({
          success: z.literal(true),
          data: dataSchema,
          meta: apiMetaSchema,
        }),
        z.object({
          success: z.literal(false),
          error: apiErrorSchema,
          meta: apiMetaSchema,
        }),
      ]);
    }
    ```
  </Accordion>
</Accordions>

***

## Error Handling

### Typed Error Classes

Define semantic error types for consistent error responses:

```typescript title="lib/errors.ts" theme={null}
export class AppError extends Error {
  constructor(
    public code: string,
    message: string,
    public statusCode: number = 500,
    public details?: unknown
  ) {
    super(message)
    this.name = 'AppError'
  }
}

export class ValidationError extends AppError {
  constructor(message: string, details?: unknown) {
    super('VALIDATION_ERROR', message, 400, details)
  }
}

export class NotFoundError extends AppError {
  constructor(resource: string, id: string) {
    super('NOT_FOUND', `${resource} not found: ${id}`, 404)
  }
}

export class ForbiddenError extends AppError {
  constructor(message: string = 'Access denied') {
    super('FORBIDDEN', message, 403)
  }
}

export class ConflictError extends AppError {
  constructor(message: string) {
    super('CONFLICT', message, 409)
  }
}

export class InvalidStateTransitionError extends ConflictError {
  constructor(resource: string, currentState: string, targetState: string) {
    super(
      `Cannot transition ${resource} from ${currentState} to ${targetState}`
    )
    this.code = 'INVALID_STATE_TRANSITION'
  }
}
```

### Global Error Handler

```typescript title="middleware/error-handler.ts" theme={null}
export function errorHandler(err: Error, c: Context) {
  const requestId = c.get('requestId')

  // Log error with context
  console.error({
    requestId,
    error: err.message,
    stack: err.stack,
    code: err instanceof AppError ? err.code : 'INTERNAL_ERROR',
    userId: c.get('userId'),
    path: c.req.path,
  })

  // Return typed error response
  if (err instanceof AppError) {
    return c.json(
      {
        success: false,
        error: {
          code: err.code,
          message: err.message,
          details: err.details,
        },
        meta: {
          requestId,
          timestamp: new Date().toISOString(),
        },
      },
      err.statusCode
    )
  }

  // Don't leak internal errors to client
  return c.json(
    {
      success: false,
      error: {
        code: 'INTERNAL_ERROR',
        message: 'An unexpected error occurred',
      },
      meta: {
        requestId,
        timestamp: new Date().toISOString(),
      },
    },
    500
  )
}

// Usage in Hono
app.onError(errorHandler)
```

### Usage in Routes

```typescript theme={null}
app.get('/servers/:id', async (c) => {
  const { id } = c.req.param()

  const server = await db.query.servers.findFirst({
    where: eq(schema.servers.id, id),
  })

  if (!server) {
    throw new NotFoundError('Server', id)
  }

  return success(c, server)
})

app.post('/servers/:id/provision', async (c) => {
  const { id } = c.req.param()
  const body = c.req.valid('json')

  const server = await db.query.servers.findFirst({
    where: eq(schema.servers.id, id),
    with: { inventory: true },
  })

  if (server.inventory.state !== 'available') {
    throw new InvalidStateTransitionError(
      'server',
      server.inventory.state,
      'provisioning'
    )
  }

  // ... continue with provisioning
})
```

***

## Audit Logging

SOC 2 compliant audit logging for all API requests. Every significant action must be traceable to a user and timestamp.

### What to Log

| Event Type                            | Log?        | Rationale                         |
| ------------------------------------- | ----------- | --------------------------------- |
| All mutations (POST/PUT/PATCH/DELETE) | ✅ Always    | Core audit trail                  |
| Failed authentication (401)           | ✅ Always    | Security monitoring               |
| Failed authorization (403)            | ✅ Always    | Access control audit              |
| Server errors (5xx)                   | ✅ Always    | Incident response                 |
| Reads on sensitive resources          | ✅ Always    | Compliance (see below)            |
| General reads (GET)                   | ⚠️ Optional | High volume; enable for debugging |
| Health/metrics endpoints              | ❌ Never     | Noise                             |

<Callout type="info">
  For multi-tenant security architecture and authorization patterns, see [Auth
  Architecture](/docs/auth).
</Callout>

### Sensitive Entities Requiring Audit Logs

These entities require audit logging on **all operations**, including reads:

| Entity                   | Why Sensitive         | Example Events                                            |
| ------------------------ | --------------------- | --------------------------------------------------------- |
| **API Keys**             | Credential access     | `api_key.created`, `api_key.viewed`, `api_key.revoked`    |
| **BMC Credentials**      | Infrastructure access | `bmc_credential.created`, `bmc_credential.accessed`       |
| **Cluster Credentials**  | Kubeconfig access     | `cluster_credential.downloaded`                           |
| **SSH Keys**             | Server access         | `ssh_key.created`, `ssh_key.deleted`                      |
| **Secrets**              | User-managed secrets  | `secret.created`, `secret.accessed`, `secret.deleted`     |
| **Organization Members** | Access control        | `member.invited`, `member.role_changed`, `member.removed` |
| **Billing/Payment**      | Financial data        | `payment_method.added`, `invoice.viewed`                  |

### Audit Event Schema

```typescript theme={null}
interface AuditEvent {
  id: string // evt_<nanoid>
  timestamp: string // ISO 8601
  requestId: string // Correlation ID

  // Actor
  actor: {
    type: 'user' | 'service' | 'system'
    id: string
    email?: string // For user actors
    service?: string // For service actors
  }

  // Action
  action: string // e.g., "server.provision", "api_key.created"
  method: string // HTTP method
  path: string // Request path

  // Resource
  resource: {
    type: string // e.g., "server", "cluster", "api_key"
    id: string
    name?: string
  }

  // Context
  organizationId: string | null

  // Outcome
  outcome: 'success' | 'failure'
  statusCode: number
  errorCode?: string

  // Changes (for mutations)
  changes?: {
    before?: Record<string, unknown>
    after?: Record<string, unknown>
  }

  // Request metadata
  ip: string
  userAgent: string
  duration: number // ms
}
```

<Accordions>
  <Accordion title="Audit Middleware Implementation">
    ```typescript title="middleware/audit.ts" theme={null}
    import { createMiddleware } from 'hono/factory';
    import { db } from '@/lib/db';
    import { auditEvents } from '@/lib/db/schema';
    import { generateId } from '@/lib/utils';

    // Endpoints to skip
    const SKIP_PATHS = ['/health', '/metrics', '/v1/auth/session'];

    // Sensitive resources - always log reads
    const SENSITIVE_RESOURCES = [
    '/api-keys',
    '/credentials',
    '/secrets',
    '/kubeconfig',
    '/members',
    '/billing',
    ];

    function shouldLog(method: string, path: string, status: number): boolean {
    // Skip health checks
    if (SKIP_PATHS.some(p => path.startsWith(p))) return false;

    // Always log mutations
    if (method !== 'GET') return true;

    // Always log auth failures
    if (status === 401 || status === 403) return true;

    // Always log server errors
    if (status >= 500) return true;

    // Always log reads on sensitive resources
    if (SENSITIVE_RESOURCES.some(r => path.includes(r))) return true;

    return false;
    }

    export const auditMiddleware = createMiddleware(async (c, next) => {
      const startTime = Date.now()
      const requestId = c.get('requestId')

    await next()

    const method = c.req.method
    const path = c.req.path
    const status = c.res.status

    if (!shouldLog(method, path, status)) return

    const userId = c.get('userId')
    const userEmail = c.get('userEmail')
    const orgId = c.get('orgId')

    await db.insert(auditEvents).values({
        id: `evt_${generateId()}`,
        timestamp: new Date().toISOString(),
        requestId,
        actorType: userId ? 'user' : 'anonymous',
        actorId: userId ?? 'anonymous',
        actorEmail: userEmail,
        action: `${method} ${path}`,
        method,
        path,
        resourceType: extractResourceType(path),
        resourceId: extractResourceId(path),
        organizationId: orgId,
        outcome: status < 400 ? 'success' : 'failure',
        statusCode: status,
        errorCode: c.get('errorCode'),
        ip:
        c.req.header('x-forwarded-for') ?? c.req.header('x-real-ip') ?? 'unknown',
        userAgent: c.req.header('user-agent') ?? 'unknown',
        duration: Date.now() - startTime,
      })
    })

    function extractResourceType(path: string): string {
    const segments = path.split('/').filter(Boolean);
    // Find the resource type (usually after 'v1')
    const resourceIndex = segments.findIndex(s => s === 'v1') + 1;
    return segments[resourceIndex] ?? 'unknown';
    }

    function extractResourceId(path: string): string | null {
    // Look for ID patterns like srv*xxx, cls_xxx, etc.
    const idMatch = path.match(/[a-z]+*[a-zA-Z0-9]+/);
    return idMatch?.[0] ?? null;
    }

    ```
  </Accordion>
</Accordions>

<Accordions>
  <Accordion title="Semantic Event Emission">
    ```typescript title="lib/audit/emit.ts" theme={null}
    import { Context } from 'hono';
    import { db } from '@/lib/db';
    import { auditEvents } from '@/lib/db/schema';
    import { generateId } from '@/lib/utils';

    interface SemanticEvent {
      action: string; // e.g., "server.provisioned"
      resource: {
        type: string;
        id: string;
        name?: string;
      };
      changes?: {
        before?: Record<string, unknown>;
        after?: Record<string, unknown>;
      };
      metadata?: Record<string, unknown>;
    }

    export async function emitAuditEvent(c: Context, event: SemanticEvent) {
      await db.insert(auditEvents).values({
        id: `evt_${generateId()}`,
        timestamp: new Date().toISOString(),
        requestId: c.get('requestId'),
        actorType: 'user',
        actorId: c.get('userId'),
        actorEmail: c.get('userEmail'),
        action: event.action,
        method: c.req.method,
        path: c.req.path,
        resourceType: event.resource.type,
        resourceId: event.resource.id,
        resourceName: event.resource.name,
        organizationId: c.get('orgId'),
        outcome: 'success',
        statusCode: 200,
        changes: event.changes ? JSON.stringify(event.changes) : null,
        metadata: event.metadata ? JSON.stringify(event.metadata) : null,
        ip: c.req.header('x-forwarded-for') ?? 'unknown',
        userAgent: c.req.header('user-agent') ?? 'unknown',
        duration: 0,
      });
    }

    // Usage in handler
    export async function provisionServer(c: Context) {
    const serverId = c.req.param('serverId');
    const body = await c.req.json();

    const server = await getServer(serverId);
    const beforeState = { status: server.status.state };

    // ... perform provisioning ...

    await emitAuditEvent(c, {
          action: 'server.provisioned',
          resource: { type: 'server', id: serverId, name: server.name },
          changes: {
          before: beforeState,
          after: { status: 'provisioning' },
        },
        metadata: { templateId: body.templateId },
      });

      return success(c, updatedServer);
    }

    ```
  </Accordion>
</Accordions>

### Audit Event Naming Convention

Use past-tense, dot-namespaced actions:

```text theme={null}
# Resource lifecycle
server.created
server.updated
server.deleted

# State transitions
server.provisioned
server.deprovisioned
cluster.scaled

# Access events
api_key.created
api_key.accessed
api_key.revoked
cluster_credential.downloaded
secret.accessed

# Security events
member.invited
member.role_changed
member.removed
auth.login_failed
auth.login_success
```

***

## Multi-Tenancy Patterns

### Row-Level Security (RLS)

Use PostgreSQL RLS for defense-in-depth isolation:

```sql theme={null}
-- Enable RLS on multi-tenant tables
ALTER TABLE atlas.clusters ENABLE ROW LEVEL SECURITY;

-- Policy: Users see only their org's data
CREATE POLICY clusters_org_isolation ON atlas.clusters
  FOR ALL
  USING (organization_id = current_setting('app.current_org_id')::TEXT);

-- Policy: Service accounts bypass RLS (for background jobs)
CREATE POLICY clusters_service_bypass ON atlas.clusters
  FOR ALL
  USING (current_setting('app.is_service', true)::BOOLEAN = true);
```

### Setting Context Per Request

```typescript title="middleware/rls-context.ts" theme={null}
export async function setRLSContext(db: Database, orgId: string) {
  await db.execute(sql`SELECT set_config('app.current_org_id', ${orgId}, true)`)
}

// In route middleware
app.use('*', async (c, next) => {
  const orgId = c.req.header('X-Org-ID')
  if (orgId) {
    await setRLSContext(db, orgId)
  }
  await next()
})
```

<Callout type="warn">
  **Critical:** RLS context is set per-transaction. For connection pooling,
  always set context at the start of each request. Drizzle's `transaction()`
  helper ensures this.
</Callout>

***

## Decision Log

### Response Envelope Pattern

| Decision                                                                          | Rationale                                                                                                                                                                                                                       | Trade-off                                                                        |
| --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------- |
| **Discriminated union** with `success: boolean` over separate success/error types | • TypeScript discriminated unions provide excellent type narrowing<br />• Client code: `if (response.success)` gets correct types<br />• Consistent structure across all endpoints<br />• Easier to generate TypeScript clients | Slightly more verbose than HTTP-only error signaling.<br />Type safety worth it. |

### Resource ID Format

| Decision                                                                   | Rationale                                                                                                                                           | Trade-off                                                                          |
| -------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| **Prefixed nanoid** (`srv_abc123`, `cls_xyz789`) over UUIDs or numeric IDs | • Human-readable in logs<br />• Immediately identify resource type<br />• URL-safe<br />• Short enough for display<br />• Low collision probability | Slightly longer than pure nanoid.<br />Worth it for debugging and log correlation. |

### Action Endpoints

| Decision                                                                     | Rationale                                                                                                                                                                                                                                                                                                  | Trade-off                                                           |
| ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- |
| **Dedicated POST endpoints** (`/power`, `/provision`) over overloading PATCH | • Semantic clarity: `POST /power action=reboot` clearer than `PATCH { online: true }`<br />• Action-specific parameters (e.g., `force`, `imageUrl`)<br />• Better audit trail: "POST /power action=off" vs "PATCH with field changes"<br />• Granular permissions: `servers:lifecycle` vs `servers:update` | Slightly more endpoints.<br />Worth it for clarity and permissions. |

### Bulk Operation Responses

| Decision                                                                                                       | Rationale                                                                                                                                                                                                                                       | Trade-off                                                     |
| -------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- |
| **Always 207 Multi-Status** with per-resource results (not 200 OK with mixed results or fail-entire-operation) | • Partial success is common in bulk operations<br />• Client needs to know which specific resources succeeded/failed<br />• Failing entire operation for one resource is poor UX<br />• 207 status code semantically correct for mixed outcomes | None significant.<br />Standard practice for bulk operations. |

### Async Operation Default

| Decision                                                        | Rationale                                                                                                                                                                                                                                                               | Trade-off                                                                     |
| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| **Return 202 immediately** (not synchronous with long timeouts) | • Infrastructure operations take 30s to 30min<br />• Prevents HTTP timeouts and connection issues<br />• Allows UI to show progress<br />• Supports horizontal scaling (request and execution on different instances)<br />• Better observability via workflow tracking | Requires more client code.<br />Mitigated by SDKs and clear polling patterns. |

***

## Testing

### Test Structure

All API endpoints should have integration tests covering:

1. **Happy path:** Successful requests with expected responses
2. **Validation:** Invalid inputs return appropriate errors
3. **Authorization:** Unauthorized users receive 403
4. **State transitions:** Invalid state transitions are rejected
5. **Edge cases:** Empty lists, missing resources, etc.

### Test Helpers

```typescript title="tests/helpers/api.ts" theme={null}
export function createTestClient(options?: {
  userId?: string
  orgId?: string
  roles?: string[]
}) {
  return {
    async get(path: string) {
      const req = new Request(`http://localhost${path}`, {
        method: 'GET',
        headers: {
          'X-User-ID': options?.userId || 'test-user',
          'X-Org-ID': options?.orgId || 'test-org',
          'X-Roles': JSON.stringify(options?.roles || ['admin']),
        },
      })
      return app.fetch(req)
    },

    async post(path: string, body: unknown) {
      const req = new Request(`http://localhost${path}`, {
        method: 'POST',
        headers: {
          'Content-Type': 'application/json',
          'X-User-ID': options?.userId || 'test-user',
          'X-Org-ID': options?.orgId || 'test-org',
          'X-Roles': JSON.stringify(options?.roles || ['admin']),
        },
        body: JSON.stringify(body),
      })
      return app.fetch(req)
    },
  }
}
```

### Example Test

```typescript theme={null}
// tests/api/servers.test.ts
import { describe, it, expect, beforeAll, afterAll } from 'vitest'
import { createTestClient } from '../helpers/api'

describe('Servers API', () => {
  let client: ReturnType<typeof createTestClient>

  beforeAll(async () => {
    client = createTestClient({ roles: ['provider_operator'] })
  })

  it('lists available servers', async () => {
    const res = await client.get('https://api.internal.example.com/v1/compute/servers?state=available')
    expect(res.status).toBe(200)

    const body = await res.json()
    expect(body.success).toBe(true)
    expect(body.data).toBeInstanceOf(Array)
    expect(body.meta.requestId).toBeDefined()
  })

  it('rejects invalid state filter', async () => {
    const res = await client.get('https://api.internal.example.com/v1/compute/servers?state=invalid')
    expect(res.status).toBe(400)

    const body = await res.json()
    expect(body.success).toBe(false)
    expect(body.error.code).toBe('VALIDATION_ERROR')
  })
})
```

***

## Related Documentation

* **[Specification](/docs/specification)** - Complete API specification with detailed endpoint examples
* **[Auth Architecture](/docs/auth)** - Authentication, authorization, and multi-tenant security patterns
* **[Data Ownership](/docs/data-ownership)** - Implementation patterns, workflow orchestration, and development guidance
* **[Data Model](/docs/data-model)** - Database schema and relationships

```
```