diff --git a/oseps/0004-secure-container-runtime.md b/oseps/0004-secure-container-runtime.md new file mode 100644 index 00000000..6468b69d --- /dev/null +++ b/oseps/0004-secure-container-runtime.md @@ -0,0 +1,873 @@ +--- +title: Pluggable Secure Container Runtime Support +authors: + - "@hittyt" +creation-date: 2026-02-05 +last-updated: 2026-02-06 +status: draft +--- + +# OSEP-0004: Pluggable Secure Container Runtime Support + + +- [Summary](#summary) +- [Motivation](#motivation) + - [Goals](#goals) + - [Non-Goals](#non-goals) +- [Requirements](#requirements) +- [Proposal](#proposal) + - [Notes/Constraints/Caveats](#notesconstraintscaveats) + - [Risks and Mitigations](#risks-and-mitigations) +- [Design Details](#design-details) + - [API Schema Extension](#api-schema-extension) + - [Server Configuration](#server-configuration) + - [Infrastructure Prerequisites](#infrastructure-prerequisites) + - [Runtime Mapper](#runtime-mapper) + - [Runtime Validator](#runtime-validator) + - [Docker Mode Implementation](#docker-mode-implementation) + - [Kubernetes Mode Implementation](#kubernetes-mode-implementation) + - [BatchSandboxProvider](#batchsandboxprovider) + - [AgentSandboxProvider](#agentsandboxprovider) + - [Pooled Sandbox Handling](#pooled-sandbox-handling) +- [Test Plan](#test-plan) +- [Drawbacks](#drawbacks) +- [Alternatives](#alternatives) +- [Infrastructure Needed](#infrastructure-needed) +- [Upgrade & Migration Strategy](#upgrade--migration-strategy) + + +## Summary + +This proposal introduces pluggable secure container runtime support for OpenSandbox, enabling sandboxes to run in secure container runtimes such as gVisor, Firecracker, and Kata Containers. This provides hardware-level isolation for executing untrusted AI-generated code, protecting the host system from potential malicious behavior. + +The design emphasizes abstraction: users only need to specify a runtime type (e.g., `secure_runtime="gvisor"`), while the server configuration handles the mapping to underlying runtime parameters (Docker `--runtime` or Kubernetes `runtimeClassName`). + +## Motivation + +OpenSandbox is designed to execute untrusted code generated by AI models (such as Claude, GPT-4, Gemini). While standard container isolation (runc) provides process-level isolation, it may not be sufficient for scenarios where: + +1. **Untrusted Code Execution**: AI-generated code could potentially contain malicious behavior, including container escape attempts +2. **Multi-tenant Environments**: Different users' sandboxes may require stronger isolation guarantees +3. **Compliance Requirements**: Some industries require hardware-level virtualization for security compliance + +Secure container runtimes like gVisor, Firecracker, and Kata Containers provide additional isolation layers: + +| Runtime | Isolation Mechanism | Use Case | +|---------|-------------------|----------| +| gVisor | User-space kernel (syscall interception) | General workloads, low overhead | +| Firecracker | Lightweight microVM | High isolation, fast boot | +| Kata Containers | Full VM with optimized hypervisor | Maximum isolation, compatibility | + +### Goals + +1. **Pluggable Architecture**: Support multiple secure container runtimes through a unified abstraction layer +2. **User-Friendly API**: Users specify runtime type without needing to know underlying implementation details (RuntimeClass, Docker runtime names) +3. **Dual-Mode Compatibility**: Work seamlessly in both Local Docker and Kubernetes deployment modes +4. **Extensibility**: Allow administrators to register custom secure runtimes through configuration +5. **Graceful Fallback**: Default to standard runc when no secure runtime is specified +6. **Validation**: Verify runtime availability before sandbox creation, with clear error messages + +### Non-Goals + +1. **Runtime Installation**: OpenSandbox will not install or configure secure container runtimes; this is the responsibility of infrastructure administrators +2. **Runtime-Specific Features**: Exposing all features of each secure runtime (e.g., gVisor platforms, Kata hypervisors) is out of scope for the initial implementation +3. **Performance Optimization**: Tuning secure runtimes for optimal performance is left to operators +4. **Mixed Runtime Pools**: Running sandboxes with different runtimes in the same pool is not supported initially + +## Requirements + +| ID | Requirement | Priority | +|----|-------------|----------| +| R1 | Users can specify secure runtime type via SDK/API | Must Have | +| R2 | Server configuration maps type to underlying runtime | Must Have | +| R3 | Support gVisor, Firecracker, Kata as predefined types | Must Have | +| R4 | Validate runtime availability before sandbox creation | Must Have | +| R5 | Work in both Docker and Kubernetes modes | Must Have | +| R6 | Default to runc when secure runtime not specified | Must Have | +| R7 | Allow custom runtime registration via configuration | Should Have | +| R8 | Clear error messages when runtime unavailable | Should Have | +| R9 | Runtime-specific options passthrough | Could Have | + +## Proposal + +We propose adding a `secureRuntime` field to the `CreateSandboxRequest` that accepts a simple type string (e.g., `"gvisor"`). The server configuration maintains mappings from this type to: + +- **Docker mode**: `--runtime` parameter (e.g., `runsc`) +- **Kubernetes mode**: `runtimeClassName` (e.g., `gvisor`) + +This abstraction allows users to write deployment-agnostic code while administrators configure the actual runtime mappings. + +``` +User API Server Config Backend +┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ +│ secure_runtime │ → │ [secure_runtimes│ → │ Docker: │ +│ = "gvisor" │ │ .gvisor] │ │ --runtime= │ +│ │ │ docker_runtime │ │ runsc │ +│ │ │ = "runsc" │ ├─────────────────┤ +│ │ │ k8s_runtime_ │ │ Kubernetes: │ +│ │ │ class="gvisor"│ → │ runtimeClass- │ +│ │ │ │ │ Name: gvisor│ +└─────────────────┘ └─────────────────┘ └─────────────────┘ +``` + +### Notes/Constraints/Caveats + +1. **Infrastructure Dependency**: Secure runtimes must be pre-installed and configured on the host (Docker) or cluster (Kubernetes) before use + +2. **Performance Overhead**: Secure runtimes add latency and resource overhead compared to runc: + - gVisor: ~10-50ms additional startup, minimal memory overhead + - Kata Containers: Performance varies significantly by hypervisor backend: + + | Hypervisor | Cold Start | Memory Overhead | Notes | + |-----------|-----------|----------------|-------| + | QEMU | ~500ms | ~20-50MB | Default, most feature-complete | + | Cloud Hypervisor (CLH) | ~200ms | ~10-20MB | Lightweight, Rust-based | + | Firecracker | ~125ms | ~5MB | Minimal footprint, limited features | + | Dragonball | ~100-200ms | ~10MB | Optimized for cloud-native | + + The actual hypervisor is determined by the `RuntimeClass` handler configured by the SRE administrator (e.g., `kata-qemu`, `kata-clh`, `kata-fc`). + + > **Note**: Firecracker is not a standalone OCI runtime. In this OSEP, `secure_runtime="firecracker"` maps to Kata Containers with the Firecracker hypervisor backend (`kata-fc`). See [Server Configuration](#server-configuration) for details. + +3. **Compatibility**: Not all container images work with all secure runtimes: + - gVisor: Some syscalls may not be implemented; check [gVisor compatibility](https://gvisor.dev/docs/user_guide/compatibility/) + - Kata (QEMU/CLH): Generally most compatible but highest overhead + - Kata + Firecracker (`kata-fc`): Limited device support; some workloads requiring specific kernel features may not work + +4. **execd Injection**: The execd binary injection mechanism must work within secure runtime constraints + +5. **Pooled Sandbox Constraint (Kubernetes)**: In Kubernetes mode with resource pools (Pool CRD), the `runtimeClassName` is fixed at pool creation time by the SRE administrator. Pre-warmed Pods cannot change their runtime after creation. This means: + - The Pool's `runtimeClassName` determines the secure runtime for all sandboxes allocated from that pool + - If a user requests `secure_runtime="kata"` but the pool was created with `runtimeClassName: gvisor`, the request must be rejected + - SRE administrators must create separate pools for different secure runtimes if multiple types are needed + - The abstraction (`secure_runtime` type) only applies at the SDK/API layer; the infrastructure layer (Pool CRD) still works directly with `runtimeClassName` + +### Risks and Mitigations + +| Risk | Impact | Mitigation | +|------|--------|------------| +| Runtime unavailable at creation time | Sandbox creation fails | Pre-validation with clear error messages | +| Syscall compatibility issues | Application may not work | Document known limitations per runtime | +| Performance degradation | Slower sandbox creation | Allow users to choose based on security/performance tradeoff | +| Configuration complexity | Operational burden | Provide sensible defaults and clear documentation | + +## Design Details + +### API Schema Extension + +Extension to `specs/sandbox-lifecycle.yml`: + +```yaml +components: + schemas: + SecureRuntimeSpec: + type: object + required: [type] + properties: + type: + type: string + description: | + Secure container runtime type. + Predefined values: gvisor, firecracker, kata + Custom runtime names defined in server configuration are also accepted. + + The server maps this type to the underlying runtime: + - Docker mode: maps to --runtime parameter + - Kubernetes mode: maps to runtimeClassName + options: + type: object + additionalProperties: true + description: | + Optional. Runtime-specific configuration options. + These options are passed to the underlying runtime. + Supported options depend on the runtime type. + + CreateSandboxRequest: + properties: + # ... existing fields ... + secureRuntime: + oneOf: + - type: string + description: Simple runtime type specification + - $ref: '#/components/schemas/SecureRuntimeSpec' + description: | + Optional. Specifies the secure container runtime. + Can be a simple string (e.g., "gvisor") or an object with options. + If not specified, uses the server's default runtime (typically runc). +``` + +**SDK Usage Examples**: + +```python +from opensandbox import Sandbox, SecureRuntime + +# Simple string form - most common usage +sandbox = await Sandbox.create( + image="python:3.11", + entrypoint=["python", "-c", "print('hello')"], + secure_runtime="gvisor", +) + +# Object form with options (for advanced use cases) +sandbox = await Sandbox.create( + image="python:3.11", + entrypoint=["python", "-c", "print('hello')"], + secure_runtime=SecureRuntime( + type="gvisor", + options={"platform": "ptrace"}, + ), +) +``` + +### Server Configuration + +Extension to `~/.sandbox.toml`: + +```toml +[runtime] +type = "docker" # or "kubernetes" +execd_image = "opensandbox/execd:v1.0.5" + +# Secure container runtime configuration +[secure_runtimes] +# Server-level default (empty string = use standard runc) +default = "" + +# Predefined secure runtimes +[secure_runtimes.gvisor] +enabled = true +docker_runtime = "runsc" # Docker: --runtime=runsc +k8s_runtime_class = "gvisor" # K8s: pod.spec.runtimeClassName + +# Firecracker Integration Note: +# Firecracker is a VMM (Virtual Machine Monitor), not an OCI-compliant container +# runtime. It cannot serve as a CRI implementation directly. There are two ways +# to use Firecracker in Kubernetes: +# +# 1. Kata Containers + Firecracker (recommended) +# Kata uses Firecracker as a hypervisor backend via the "kata-fc" RuntimeClass +# handler. This is the mature, production-ready approach. +# +# 2. firecracker-containerd +# A containerd runtime shim that uses Firecracker directly. This project is +# less actively maintained and not recommended for production use. +# +# This OSEP recommends approach (1): Firecracker via Kata Containers. +[secure_runtimes.firecracker] +enabled = true +docker_runtime = "" # Not supported in Docker mode +k8s_runtime_class = "kata-fc" # Kata Containers with Firecracker hypervisor + +[secure_runtimes.kata] +enabled = true +docker_runtime = "kata-runtime" +k8s_runtime_class = "kata-qemu" + +# Custom runtime example +[secure_runtimes.my-secure-vm] +enabled = true +docker_runtime = "my-vm-runtime" +k8s_runtime_class = "my-vm-class" +``` + +### Infrastructure Prerequisites + +OpenSandbox does not install secure runtimes. The following must be configured by infrastructure administrators. + +#### Docker Mode - gVisor Setup + +**Step 1: Install gVisor runsc** + +```bash +# Ubuntu/Debian +curl -fsSL https://gvisor.dev/archive.key | sudo gpg --dearmor -o /usr/share/keyrings/gvisor-archive-keyring.gpg +echo "deb [signed-by=/usr/share/keyrings/gvisor-archive-keyring.gpg] https://storage.googleapis.com/gvisor/releases release main" | \ + sudo tee /etc/apt/sources.list.d/gvisor.list +sudo apt-get update && sudo apt-get install -y runsc +``` + +**Step 2: Configure Docker daemon** + +```json +// /etc/docker/daemon.json +{ + "runtimes": { + "runsc": { + "path": "/usr/bin/runsc", + "runtimeArgs": [ + "--platform=systrap", + "--network=host" + ] + } + } +} +``` + +```bash +sudo systemctl restart docker +``` + +**Step 3: Verify installation** + +```bash +docker run --runtime=runsc hello-world +``` + +#### Docker Mode - Kata Containers Setup + +```bash +# Install Kata Containers +bash -c "$(curl -fsSL https://raw.githubusercontent.com/kata-containers/kata-containers/main/utils/kata-manager.sh) install-docker-system" +``` + +```json +// /etc/docker/daemon.json +{ + "runtimes": { + "kata-runtime": { + "path": "/usr/bin/kata-runtime" + } + } +} +``` + +#### Kubernetes Mode - RuntimeClass Setup + +Cluster administrators must create RuntimeClass resources: + +```yaml +# gVisor RuntimeClass +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: gvisor +handler: runsc # Matches containerd handler name + +--- +# Kata Containers (QEMU backend) RuntimeClass +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: kata-qemu +handler: kata-qemu + +--- +# Kata Containers (Firecracker backend) RuntimeClass +# This is what secure_runtime="firecracker" maps to +apiVersion: node.k8s.io/v1 +kind: RuntimeClass +metadata: + name: kata-fc +handler: kata-fc +``` + +containerd configuration (`/etc/containerd/config.toml`): + +```toml +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runsc] + runtime_type = "io.containerd.runsc.v1" + +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu] + runtime_type = "io.containerd.kata-qemu.v2" + +[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-fc] + runtime_type = "io.containerd.kata-fc.v2" +``` + +### Runtime Mapper + +New module `server/src/services/runtime_mapper.py`: + +```python +class SecureRuntimeMapper: + """Maps user API runtime type to underlying runtime parameters.""" + + def __init__(self, config: AppConfig): + self.config = config + self.runtime_mode = config.runtime.type # "docker" or "kubernetes" + + def resolve(self, secure_runtime_type: str) -> str: + """ + Resolve runtime type to backend-specific identifier. + + Args: + secure_runtime_type: User-specified type (e.g., "gvisor") + + Returns: + Docker mode: Returns docker_runtime (e.g., "runsc") + K8s mode: Returns k8s_runtime_class (e.g., "gvisor") + + Raises: + ValueError: If runtime type is not configured or disabled + """ + runtime_config = self.config.secure_runtimes.get(secure_runtime_type) + if not runtime_config: + raise ValueError( + f"Secure runtime '{secure_runtime_type}' is not configured. " + f"Available: {list(self.config.secure_runtimes.keys())}" + ) + + if not runtime_config.enabled: + raise ValueError( + f"Secure runtime '{secure_runtime_type}' is disabled in configuration." + ) + + if self.runtime_mode == "docker": + if not runtime_config.docker_runtime: + raise ValueError( + f"Secure runtime '{secure_runtime_type}' is not supported in Docker mode. " + f"It is only available in Kubernetes mode." + ) + return runtime_config.docker_runtime + else: # kubernetes + return runtime_config.k8s_runtime_class +``` + +### Runtime Validator + +New module `server/src/services/runtime_validator.py`: + +```python +class RuntimeValidator: + """Validates that secure runtimes are available in the environment.""" + + def __init__(self, docker_client=None, k8s_client=None): + self.docker_client = docker_client + self.k8s_client = k8s_client + + def validate_docker_runtime(self, runtime_name: str) -> bool: + """Check if Docker daemon has the specified runtime configured.""" + try: + info = self.docker_client.info() + available_runtimes = info.get("Runtimes", {}).keys() + return runtime_name in available_runtimes + except Exception as e: + logger.warning(f"Failed to check Docker runtimes: {e}") + return False + + def validate_k8s_runtime_class(self, runtime_class_name: str) -> bool: + """Check if the RuntimeClass exists in the Kubernetes cluster.""" + try: + self.k8s_client.read_runtime_class(runtime_class_name) + return True + except ApiException as e: + if e.status == 404: + return False + raise +``` + +### Docker Mode Implementation + +Changes to `server/src/services/docker.py`: + +```python +class DockerSandboxService(SandboxService): + def __init__(self, config: Optional[AppConfig] = None): + # ... existing initialization ... + self.runtime_mapper = SecureRuntimeMapper(self.app_config) + self.runtime_validator = RuntimeValidator(docker_client=self.docker_client) + + def _get_container_runtime(self, request: CreateSandboxRequest) -> Optional[str]: + """ + Resolve secure runtime configuration to Docker runtime name. + + Returns None for standard runc, or runtime name (e.g., "runsc"). + """ + if not request.secure_runtime: + default = self.app_config.secure_runtimes.default + if not default: + return None # Use standard runc + secure_runtime_type = default + else: + secure_runtime_type = ( + request.secure_runtime + if isinstance(request.secure_runtime, str) + else request.secure_runtime.type + ) + + docker_runtime = self.runtime_mapper.resolve(secure_runtime_type) + + if not self.runtime_validator.validate_docker_runtime(docker_runtime): + raise HTTPException( + status_code=400, + detail={ + "code": "SECURE_RUNTIME_UNAVAILABLE", + "message": f"Secure runtime '{secure_runtime_type}' is not available. " + f"Please ensure '{docker_runtime}' is installed and configured " + f"in Docker daemon (/etc/docker/daemon.json)." + } + ) + + return docker_runtime + + async def create_sandbox(self, request: CreateSandboxRequest) -> CreateSandboxResponse: + # ... existing code ... + + runtime = self._get_container_runtime(request) + + container = self.docker_client.containers.run( + image=request.image.uri, + # ... other parameters ... + runtime=runtime, # "runsc", "kata-runtime", or None + ) +``` + +### Kubernetes Mode Implementation + +Both Kubernetes workload providers must support `runtimeClassName` injection. The core resolution and validation logic is shared; each provider applies it to its own Pod spec path. + +#### Shared Logic + +The `_resolve_and_validate_runtime_class` helper is shared by both providers (e.g., via mixin or utility function): + +```python +def _resolve_and_validate_runtime_class( + runtime_mapper: SecureRuntimeMapper, + runtime_validator: RuntimeValidator, + secure_runtimes_config: SecureRuntimesConfig, + request: CreateSandboxRequest, +) -> Optional[str]: + """ + Resolve user's secure_runtime to a runtimeClassName and validate it. + Returns None if no secure runtime is requested. + """ + if not request.secure_runtime: + default = secure_runtimes_config.default + if not default: + return None # Use cluster default + secure_runtime_type = default + else: + secure_runtime_type = ( + request.secure_runtime + if isinstance(request.secure_runtime, str) + else request.secure_runtime.type + ) + + runtime_class = runtime_mapper.resolve(secure_runtime_type) + + if not runtime_validator.validate_k8s_runtime_class(runtime_class): + raise HTTPException( + status_code=400, + detail={ + "code": "SECURE_RUNTIME_UNAVAILABLE", + "message": f"RuntimeClass '{runtime_class}' does not exist. " + f"Please ensure the RuntimeClass is created by cluster administrator." + } + ) + + return runtime_class +``` + +#### BatchSandboxProvider + +Changes to `server/src/services/k8s/batchsandbox_provider.py`: + +- **CRD**: `sandbox.opensandbox.io/v1alpha1` BatchSandbox +- **Pod spec path**: `spec.template.spec` +- **Two creation paths**: + - Template mode: inject `runtimeClassName` into Pod spec before template merge + - Pool mode (`poolRef`): runtime is determined by the Pool's `runtimeClassName`, validate consistency only (see [Pooled Sandbox Handling](#pooled-sandbox-handling)) + +```python +class BatchSandboxProvider: + def create_workload(self, request: CreateSandboxRequest, ...): + # ... existing code ... + + if pool_ref: + # Pool mode: runtime comes from pool, validate consistency + self._validate_pool_runtime_consistency(pool_runtime_class, request) + else: + # Template mode: inject runtimeClassName + runtime_class = _resolve_and_validate_runtime_class( + self.runtime_mapper, self.runtime_validator, + self.config.secure_runtimes, request, + ) + if runtime_class: + # spec.template.spec.runtimeClassName + runtime_manifest["spec"]["template"]["spec"]["runtimeClassName"] = runtime_class + + # ... template merge ... +``` + +Generated BatchSandbox CR: + +```yaml +apiVersion: sandbox.opensandbox.io/v1alpha1 +kind: BatchSandbox +metadata: + name: sandbox-abc123 +spec: + template: + spec: + runtimeClassName: "gvisor" # Injected by server + containers: + - name: sandbox-container + image: python:3.11 +``` + +#### AgentSandboxProvider + +Changes to `server/src/services/k8s/agent_sandbox_provider.py`: + +- **CRD**: `agents.x-k8s.io/v1alpha1` Sandbox +- **Pod spec path**: `spec.podTemplate.spec` +- **No pool concept**: always inject directly into Pod spec + +```python +class AgentSandboxProvider: + def create_workload(self, request: CreateSandboxRequest, ...): + # ... existing code ... + + # Resolve secure runtime + runtime_class = _resolve_and_validate_runtime_class( + self.runtime_mapper, self.runtime_validator, + self.config.secure_runtimes, request, + ) + + pod_spec = self._build_pod_spec(request, ...) + if runtime_class: + pod_spec["runtimeClassName"] = runtime_class + + # spec.podTemplate.spec + runtime_manifest["spec"]["podTemplate"]["spec"] = pod_spec + + # ... template merge ... +``` + +Generated AgentSandbox CR: + +```yaml +apiVersion: agents.x-k8s.io/v1alpha1 +kind: Sandbox +metadata: + name: sandbox-abc123 +spec: + podTemplate: + spec: + runtimeClassName: "gvisor" # Injected by server + initContainers: + - name: execd-init + image: opensandbox/execd:v1.0.5 + containers: + - name: sandbox-container + image: python:3.11 +``` + +#### Provider Comparison + +| Aspect | BatchSandboxProvider | AgentSandboxProvider | +|--------|---------------------|---------------------| +| CRD Kind | `BatchSandbox` | `Sandbox` | +| Pod Spec Path | `spec.template.spec` | `spec.podTemplate.spec` | +| Pool Support | Yes (`poolRef`) | No | +| Runtime Injection Point | Before template merge | Before template merge | +| Pool Runtime Conflict | 409 Conflict on mismatch | N/A | + +#### Pooled Sandbox Handling + +In Kubernetes mode, when a sandbox uses a Pool (via `poolRef`), the runtime is determined by the Pool's `runtimeClassName`, not by the user's `secure_runtime` parameter. The server must validate consistency between the two. + +**Pool configuration by SRE administrator:** + +```yaml +apiVersion: sandbox.opensandbox.io/v1alpha1 +kind: Pool +metadata: + name: gvisor-pool +spec: + template: + spec: + runtimeClassName: "gvisor" # Fixed at pool creation + containers: + - name: sandbox-container + image: python:3.11 + capacitySpec: + bufferMax: 10 + bufferMin: 2 + poolMax: 20 + poolMin: 5 +``` + +**Server-side validation:** + +```python +def _validate_pool_runtime_consistency( + self, pool_runtime_class: Optional[str], request: CreateSandboxRequest +): + """ + Validate that user's secure_runtime matches the pool's runtimeClassName. + + In pooled mode, pre-warmed Pods already have a fixed runtime. + The user's request must be compatible. + """ + if not request.secure_runtime: + return # No explicit preference, accept pool's runtime + + requested_type = ( + request.secure_runtime + if isinstance(request.secure_runtime, str) + else request.secure_runtime.type + ) + requested_class = self.runtime_mapper.resolve(requested_type) + + if pool_runtime_class and requested_class != pool_runtime_class: + raise HTTPException( + status_code=409, + detail={ + "code": "RUNTIME_POOL_MISMATCH", + "message": f"Requested secure runtime '{requested_type}' " + f"(runtimeClass='{requested_class}') conflicts with " + f"pool's runtimeClass='{pool_runtime_class}'. " + f"Use a pool configured with the matching runtime, " + f"or omit secure_runtime to accept the pool's default." + } + ) +``` + +**Behavior matrix:** + +| User `secure_runtime` | Pool (`poolRef`) | Pool `runtimeClassName` | Result | +|----------------------|-----------------|------------------------|--------| +| Not specified | Yes | `gvisor` | OK - use pool's gVisor runtime | +| `"gvisor"` | Yes | `gvisor` | OK - consistent | +| `"kata"` | Yes | `gvisor` | **409 Conflict** - mismatch | +| `"gvisor"` | Yes | Not set (runc) | **409 Conflict** - pre-warmed Pods use runc, cannot switch to gVisor | +| `"gvisor"` | No | N/A | OK - non-pooled creation, inject runtimeClassName | +| Not specified | No | N/A | OK - use server default (typically runc) | + +### Compatibility Matrix + +| Secure Runtime | Local Docker | Kubernetes | Notes | +|---------------|--------------|------------|-------| +| gVisor (runsc) | Full support | Full support | Docker `--runtime=runsc`; K8s via RuntimeClass | +| Kata Containers | Full support | Full support | Docker `--runtime=kata-runtime`; K8s via RuntimeClass | +| Firecracker | Not supported | Via Kata (`kata-fc`) | Not a Docker OCI runtime; use Kata with Firecracker hypervisor backend in K8s | +| Custom runtimes | Via config | Via RuntimeClass | Requires pre-installation | + +## Test Plan + +### Unit Tests + +| Test Case | Description | +|-----------|-------------| +| Config parsing | Verify SecureRuntimesConfig correctly parses TOML | +| Type mapping | Verify RuntimeMapper returns correct identifiers for Docker/K8s | +| Unknown type handling | Verify proper error for unconfigured runtime types | +| Default value handling | Verify fallback to runc when secure_runtime not specified | + +### Integration Tests + +| Test Case | Description | +|-----------|-------------| +| Docker + gVisor | Create sandbox on Docker host with runsc configured | +| Docker + Kata | Create sandbox on Docker host with kata-runtime configured | +| Docker runtime unavailable | Verify clear error when runtime not installed | +| Docker + Firecracker rejected | Verify clear error: Firecracker not supported in Docker mode | +| K8s + gVisor RuntimeClass | Create sandbox in cluster with gVisor RuntimeClass | +| K8s + kata-fc RuntimeClass | Create sandbox with `secure_runtime="firecracker"` maps to kata-fc | +| K8s RuntimeClass missing | Verify clear error when RuntimeClass doesn't exist | +| Pooled sandbox runtime match | Create sandbox with `secure_runtime` matching pool's runtimeClassName | +| Pooled sandbox runtime mismatch | Verify 409 Conflict when `secure_runtime` conflicts with pool | + +### E2E Tests + +| Test Case | Description | +|-----------|-------------| +| Python SDK full flow | Create sandbox with `secure_runtime="gvisor"`, execute code | +| Runtime isolation verification | Verify syscall interception in gVisor sandbox | +| Fallback behavior | Verify standard runc when no runtime specified | +| execd injection under gVisor | Verify execd binary injection works within gVisor runtime | + +## Drawbacks + +1. **Operational Complexity**: Administrators must install and configure secure runtimes +2. **Performance Overhead**: Secure runtimes add startup latency and memory overhead +3. **Compatibility Issues**: Some workloads may not work with certain runtimes +4. **Documentation Burden**: Requires comprehensive setup guides for each runtime + +## Alternatives + +### Alternative 1: Hardcoded Runtime Support + +**Approach**: Only support gVisor with hardcoded configuration. + +**Pros**: +- Simpler implementation +- Less configuration + +**Cons**: +- No flexibility for different runtimes +- Cannot adapt to different environments + +**Decision**: Rejected. The pluggable approach provides necessary flexibility for different deployment scenarios. + +### Alternative 2: Expose RuntimeClass Directly + +**Approach**: Let users specify `runtimeClassName` directly in the API. + +**Pros**: +- Full control for users +- Simpler server implementation +- Aligns with what SRE administrators already use when configuring Pool CRDs + +**Cons**: +- Leaks Kubernetes concepts into the SDK user-facing API +- User code becomes deployment-specific (Docker mode has no `runtimeClassName`) + +**Decision**: Rejected at the SDK/API layer. The abstraction (`secure_runtime` type) keeps user code deployment-agnostic. + +However, this abstraction only applies to the **SDK user layer**. At the **infrastructure layer** (K8s Pool CRD, Docker daemon config), platform-specific concepts like `runtimeClassName` are unavoidable. SRE administrators must set `runtimeClassName` directly when configuring Pool resources. The server is responsible for validating consistency between the user's `secure_runtime` request and the pool's `runtimeClassName` (see [Pooled Sandbox Handling](#pooled-sandbox-handling)). + +### Alternative 3: Automatic Runtime Detection + +**Approach**: Automatically detect and use the most secure available runtime. + +**Pros**: +- Zero user configuration +- Always uses best available isolation + +**Cons**: +- Unpredictable behavior across environments +- May break workloads with runtime incompatibilities +- Performance impact without user consent + +**Decision**: Rejected. Explicit user choice is preferred for security/performance tradeoffs. + +## Infrastructure Needed + +- **Testing Environments**: + - Docker host with gVisor (runsc) configured + - Docker host with Kata Containers (kata-runtime) configured + - Kubernetes cluster with gVisor RuntimeClass (`runsc`) + - Kubernetes cluster with Kata QEMU RuntimeClass (`kata-qemu`) + - Kubernetes cluster with Kata + Firecracker RuntimeClass (`kata-fc`) + +- **CI/CD Updates**: + - Add integration tests for secure runtime validation + - Add E2E tests with gVisor-enabled environment + +- **Documentation**: + - User guide: How to use secure runtimes + - Admin guide: How to set up gVisor/Kata/Firecracker + - API reference updates + +## Upgrade & Migration Strategy + +### Backward Compatibility + +- **No breaking changes**: The `secureRuntime` field is optional +- **Default behavior unchanged**: Without `secureRuntime`, sandboxes use standard runc +- **Existing configurations work**: No changes required to existing deployments + +### Migration Path + +1. **Phase 1**: Deploy server with secure_runtimes configuration (runtimes disabled) +2. **Phase 2**: Install and configure secure runtimes on infrastructure +3. **Phase 3**: Enable runtimes in configuration +4. **Phase 4**: Users can opt-in via `secure_runtime` parameter + +### Documentation Updates + +- Add secure container section to SDK documentation +- Add infrastructure setup guide +- Add troubleshooting guide for runtime issues