A sandbox's container is ephemeral. Files written into the container's writable layer (/tmp, the working directory, anywhere not on a mount) live only as long as the container itself. If the container is destroyed - by an explicit Destroy, by a host crash, or by the loss of the owner node in cluster mode - that data is gone. By default the sandbox placement is gone too: the API returns 410 Gone. Sandboxes that opt into failover.policy = "recreate" are best-effort recreated on another worker from their replicated create spec.

This is the same model used by GitHub Codespaces, Daytona workspaces, and Coder workspaces. Persistent state lives on a separate, declared volume; the container is treated as cattle. Add cluster mode and the sandbox is also treated as cattle - a fresh Create is the way to recover.

Durability class

durability is a top-level create field on every runtime, not a WASM-only knob. It selects how hard the daemon tries to keep a sandbox alive across its own restarts.

Class	Behavior
ephemeral	Lost on daemon restart.
passivatable	Checkpointed locally on drain and rehydrated on start.
durable	Local checkpoint plus cross-node failover push.

The only runtime-specific behavior is the default when you omit the field, because container/VM runtimes survive a daemon restart natively while WASM workers do not:

Runtime	Default when omitted
docker, firecracker, gvisor	passivatable
wasm	ephemeral

durable is currently implemented for WASM only; requesting it on another runtime is rejected at create time. The field name, values, and SDK shape are identical everywhere — see WASM Sandbox for the checkpoint mechanics that back it on WASM.

TypeScript

const sandbox = await client.create({
  image: 'ubuntu:22.04',
  durability: 'passivatable',
})

Python

sandbox = client.create({
    'image': 'ubuntu:22.04',
    'durability': 'passivatable',
})

Go

sandbox, err := client.Create(ctx, sdktypes.CreateSandboxOptions{
    Image:      "ubuntu:22.04",
    Durability: "passivatable",
})

Rust

let sandbox = client.create(CreateOptions {
    image:      "ubuntu:22.04".to_string(),
    durability: Some("passivatable".to_string()),
    ..Default::default()
})?;

Java

Sandbox sandbox = client.create(
    new CreateOptions()
        .setImage("ubuntu:22.04")
        .setDurability("passivatable")
);

Durability class governs survival across daemon restarts. The rest of this page covers the orthogonal concerns of workspace state (mounts) and cluster owner death (failover policy).

What survives, what does not

Asset	Single-node Destroy	Single-node host crash	Cluster-mode owner death
Container writable layer	Lost	Lost	Lost
Mounted volumes (S3, NFS, SSHFS, rclone)	Survive (external)	Survive (external)	Survive (external) - a new sandbox can re-mount them
Sandbox ID	Reused on next Create	Preserved if record persisted	Default: lost (410 Gone). With failover.policy="recreate": preserved if recreation succeeds
Environment variables, image, resource limits	N/A	Restored on Start	Default: lost. With recreate: restored from replicated spec
Idle lifecycle policy	N/A	Restored on Start	Default: lost. With recreate: restored
Exposed ports (preview, TCP, TLS)	Removed	Restored on Start	Default: lost. With recreate: replayed when possible
Caddy routes	Removed	Restored on Start	Default: lost. With recreate: rebuilt by the new owner and ingress reconciler
Host port number	N/A	Same host, same port	Default: lost. With recreate: preferred raw TCP host ports are replayed when available
Public URL	Removed	Restored on Start	Default: lost. With recreate: stable ingress host remains, owner route changes underneath

Make workspace state durable

Mount external storage at the path you want to keep across container lifetimes. The mount is established by the host (not the sandbox), so credentials never enter the container, and the data persists independently of any single sandbox or node.

TypeScript

const sandbox = await client.create({
  image: 'ubuntu:22.04',
  mounts: [
    {
      type:   's3',
      target: '/workspace',
      source: 's3://my-bucket/project-42',
      credentials: {
        access_key_id:     'AKIA...',
        secret_access_key: 'secret...',
      },
    },
  ],
})

Python

sandbox = client.create({
    'image': 'ubuntu:22.04',
    'mounts': [
        {
            'type':   's3',
            'target': '/workspace',
            'source': 's3://my-bucket/project-42',
            'credentials': {
                'access_key_id':     'AKIA...',
                'secret_access_key': 'secret...',
            },
        },
    ],
})

Go

sandbox, err := client.Create(ctx, sdktypes.CreateSandboxOptions{
    Image: "ubuntu:22.04",
    Mounts: []sdktypes.MountSpec{
        {
            Type:   "s3",
            Target: "/workspace",
            Source: "s3://my-bucket/project-42",
            Credentials: map[string]string{
                "access_key_id":     "AKIA...",
                "secret_access_key": "secret...",
            },
        },
    },
})

Rust

use std::collections::HashMap;

let mut creds = HashMap::new();
creds.insert("access_key_id".to_string(), "AKIA...".to_string());
creds.insert("secret_access_key".to_string(), "secret...".to_string());

let sandbox = client.create(CreateOptions {
    image: "ubuntu:22.04".to_string(),
    mounts: Some(vec![aerolvm_sdk::MountSpec {
        type_:       "s3".to_string(),
        target:      "/workspace".to_string(),
        source:      "s3://my-bucket/project-42".to_string(),
        credentials: Some(creds),
    }]),
    ..Default::default()
})?;

Java

import ai.aerol.microvm.model.MountSpec;
import java.util.List;
import java.util.Map;

Sandbox sandbox = client.create(
    new CreateOptions()
        .setImage("ubuntu:22.04")
        .setMounts(List.of(
            new MountSpec()
                .setType("s3")
                .setTarget("/workspace")
                .setSource("s3://my-bucket/project-42")
                .setCredentials(Map.of(
                    "access_key_id",     "AKIA...",
                    "secret_access_key", "secret..."
                ))
        ))
);

Anything written under /workspace outlives the container. Anything written elsewhere does not. See External Storage for the full mount surface (S3, NFS, SSHFS, rclone).

Cluster-mode owner death

When SB_ENABLE_CLUSTER=true and a sandbox's owner node dies, the default policy is:

1. After SB_DEAD_OWNER_GRACE (default 30s) the cluster leader orphans every placement that the dead node owned (OwnerNodeID = "" in the replicated FSM) and removes the node from the raft configuration.
2. Any further API call for one of those sandboxes returns 410 Gone.
3. Clients should issue a fresh Create to land a new sandbox on the surviving fleet.

For workloads that can tolerate losing writable-layer state but need the same sandbox ID and route intents, opt in at create time:

TypeScript

const sandbox = await client.create({
  image: 'ubuntu:22.04',
  failover: { policy: 'recreate' },
})

Python

sandbox = client.create({
    'image': 'ubuntu:22.04',
    'failover': {'policy': 'recreate'},
})

Go

sandbox, err := client.Create(ctx, sdktypes.CreateSandboxOptions{
    Image: "ubuntu:22.04",
    Failover: &sdktypes.Failover{Policy: sdktypes.FailoverPolicyRecreate},
})

Rust

let sandbox = client.create(CreateOptions {
    image: "ubuntu:22.04".to_string(),
    failover: Some(aerolvm_sdk::Failover { policy: "recreate".to_string() }),
    ..Default::default()
})?;

Java

Sandbox sandbox = client.create(
    new CreateOptions()
        .setImage("ubuntu:22.04")
        .setFailover(new Failover().setPolicy("recreate"))
);

The cluster replicates each sandbox's create spec (image, env, resource limits, mounts, lifecycle, failover policy), the sealed registry/mount credential bag, and every exposed-port route intent. When an opted-in owner dies, the leader reassigns that placement to a live worker that satisfies the spec; the new worker recreates the sandbox with the original ID and replays exposed ports.

failover.policy="recreate" is rejected for local-only images because no other worker can pull them. Writable-layer state is still lost, so put durable state on a mount.

If you ever need to re-expose a port manually (for example, the original ExposePort call landed during a network partition and was dropped), the API is idempotent:

TypeScript

await client.exposePort(sandboxId, { port: 3000, protocol: 'http' })

Python

client.expose_port(sandbox_id, port=3000, protocol='http')

Go

if err := client.ExposePort(ctx, sandboxID, sdktypes.ExposePortOptions{
    Port:     3000,
    Protocol: "http",
}); err != nil {
    log.Fatal(err)
}

Rust

client.expose_port(&sandbox_id, ExposePortOptions {
    port:     3000,
    protocol: "http".to_string(),
    ..Default::default()
})?;

Java

client.exposePort(sandboxId, new ExposePortOptions()
    .setPort(3000)
    .setProtocol("http"));

Cluster network security

Cluster traffic uses two transports:

• Gossip (SB_GOSSIP_BIND_ADDR, default :7001) - SWIM membership and role/identity metadata.
• Capacity heartbeats (GET /v1/capacity) - authenticated worker capacity snapshots used by server-role schedulers.
• Raft (SB_RAFT_BIND_ADDR, default :7000) - placement-map consensus.

When a node successfully joins gossip, it is added to the Raft configuration as a voter until SB_CLUSTER_MAX_AUTO_VOTERS is reached, then as a non-voter. Anyone who can reach these ports can still join the cluster and influence placement replication. Run them on a private network the operator controls (VPC peering, WireGuard mesh, dedicated cluster subnet); never expose them to the public internet.

To authenticate gossip and reject foreign joiners, set SB_GOSSIP_SECRET_KEY to a base64-encoded 16, 24, or 32-byte key (AES-128/192/256-GCM). The same key must be configured on every node in the cluster.

# Generate a 32-byte key once and distribute it to every node.
openssl rand -base64 32

# Then on each node:
export SB_GOSSIP_SECRET_KEY="<the-base64-key>"

When SB_GOSSIP_SECRET_KEY is unset, the daemon logs a warning at boot. Raft itself remains plaintext today; the gossip key is what gates the join, so a peer that can't decrypt the gossip handshake never reaches Raft. Wrap Raft in TLS at the network layer (e.g., a private overlay) for defense in depth.

Limitations

• Sandbox HA is opt-in. Owner-node death orphans default placements; clients see 410 Gone and must create a fresh sandbox. Only sandboxes created with failover.policy="recreate" are reassigned and recreated.
• Ingress has a convergence window. Public HTTP/TLS URLs and raw TCP host ports for live sandboxes are replicated through the cluster route map, but non-owner ingress routes refresh periodically. Immediately after a new ExposePort call, a backend can briefly return 404/502 until it has reconciled.
• Writable-layer state is not replicated. Treat anything outside a mount as cache. If you need it after a crash, write it to a mount.
• Single-node deployments inherit the same model. A host crash loses the writable layer just as owner death does. Mount external storage for anything that must survive.

Recommended pattern

1. Pick the directory your workload writes durable state to (often a project root, a database data dir, or a model cache).
2. Declare it as a mount target at Create time.
3. Treat everything else inside the container as scratch.
4. In cluster mode, design the client to call Create again on 410 Gone. The mount keeps the durable state intact; a new sandbox picks up where the old one left off.

This keeps the durability story identical across single-node, cluster-mode, and any future runtime: the spec describes the sandbox, the mounts hold the data, the container is replaceable - and, in cluster mode, the sandbox itself is replaceable too.