Sandbox troubleshooting and recovery

Start with durable state, not the last toast or terminal line. Preserve the request ID, sandbox ID, operation ID, resource ID, last event sequence, and UTC timestamp. Never include tokens, submitted secret values, terminal/port tickets, signed URLs, or volatile database credentials in support material.

First-response checklist

stackshift --output json sandbox get <sandbox-id>
stackshift --output json sandbox operation <operation-id>
stackshift sandbox events <sandbox-id> --after-sequence <last-sequence>

Compare:

desired_state with observed_state and readiness.
generation with observed_generation.
active_operation.status, step, attempts, deadline, and failure fields.
Effective capability decisions and their reason/remediation.
Recent events after the last sequence you successfully processed.

A transport timeout means the outcome is unknown. Fetch the operation or resource before retrying.

Stable API errors

Code	Meaning	Safe response
`invalid_request`	Malformed input or field validation failed	Correct reported field errors; keep the same logical intent
`authentication_required`	Token missing, expired, or invalid	Re-authenticate; do not print the token
`permission_denied`	Actor lacks required scope/role	Request the narrow permission; a UI control cannot bypass it
`not_found`	ID is wrong, inaccessible, or tombstoned	Verify account/profile and exact ID; do not recreate blindly
`conflict`	Resource/action conflicts with current state	Refetch resource and active operation
`idempotency_key_required`	Mutation omitted its key	Resubmit once with a stable key
`idempotency_conflict`	Same key was reused with different input	Retrieve the original result; use a new key only for a genuinely new action
`version_conflict`	`If-Match` is stale	Refetch, review the new version, then decide whether to retry
`quota_exceeded`	Account limit would be exceeded	Read the included quota name/current/requested/limit/unit; release capacity or change plan/request
`capacity_unavailable`	Qualified server capacity cannot admit the request	Retry only when `retryable` and with backoff; do not create duplicates
`capability_unavailable`	Target/image/backend cannot provide a required feature	Use the remediation or a compatible template/profile
`invalid_state_transition`	Action is invalid from current state	Wait for the active operation or perform the prerequisite transition
`deployment_server_unavailable`	Configured worker cannot currently accept work	Preserve the operation and wait/retry according to `retryable`
`operation_deadline_exceeded`	Durable work exceeded its deadline	Inspect compensation/recovery state; do not assume rollback completed
`operation_failed`	Operation reached terminal failure	Inspect failure code/message, attempts, and recovery availability
`policy_denied`	Effective security/plan policy rejects the action	Narrow the request or obtain an authorized policy change
`snapshot_incompatible`	Image, backend, protocol, members, or target mismatch	Choose a compatible sleeping target or correct fork inputs
`snapshot_corrupt`	Manifest/chunk integrity verification failed	Do not restore; retain evidence and use another verified snapshot
`rate_limited`	Request rate exceeded	Honor server guidance and use bounded exponential backoff with jitter
`internal_error`	Unexpected control-plane failure	Retry only if marked retryable; report request/operation IDs

Lifecycle does not converge

queued or scheduling: inspect quota, capacity, target availability, and active operation deadline.
provisioning or starting: inspect image digest compatibility, supervisor protocol, storage, network-policy application, and worker failure code.
degraded: the sandbox exists but a health, policy, storage, port, service, or reconciliation condition needs attention. Avoid destructive retry loops.
failed: this is not successful cleanup. If recovery_state is available or required, review the failed operation and run recovery once.

stackshift --timeout 15m sandbox recover <sandbox-id> --wait

If recovery fails, preserve both operation IDs. Destroy only after reviewing retained data and confirming that recovery is no longer required.

Execution or terminal fails

Symptom	Check
Exit code from `sandbox exec`	The CLI propagates remote exit codes 1–125; inspect stderr and execution state
`sandbox_not_running`	Resume and wait for readiness before starting work
`timed_out`	Increase `--command-timeout` only after confirming the process is expected to run longer
`oom_killed`	Inspect memory limit and usage; reduce workload or request an allowed larger profile
Output missing	Resume logs from `next_offset`; handle truncation and completion explicitly
Terminal rejected	Check readiness, writer lease ownership, ticket expiry, permission, and current operation
Terminal disconnects	Request a new short-lived ticket and reconnect using the retained sequence; do not reuse an expired ticket

Use --debug only long enough to capture request diagnostics, then redact authorization, cookies, signed URLs, and headers before sharing.

File write conflicts

writer lease conflicts mean another active writer owns mutation authority. Coordinate before forced takeover. A checksum/version conflict means the file changed after it was read; reload, merge, and retry with the new expected checksum. Path errors usually indicate a non-absolute or disallowed mount path, traversal, or a symlink resolving outside the allowed root. Do not work around path validation by executing privileged shell commands.

Network or preview is unavailable

Confirm the process is listening on the intended port and readiness passes.
Confirm a port resource exists and is ready; listening alone does not create ingress.
Confirm access mode and obtain a fresh private token/authenticated session.
Inspect effective egress/inbound policy and port access logs.
For outbound failures, resolve every redirect/DNS answer and check protected destination rules.

A policy-denied private, loopback, link-local, metadata, or control-plane destination should not be retried through alternate encodings or DNS aliases.

Service is not ready

Inspect both the service resource and its operation. Check engine support, sidecar capacity/quota, owned volume encryption/state, image digest, sandbox availability, and service failure fields. The connection URL appears only inside the sandbox’s volatile delivery file; its absence from API/UI output is expected. After service creation, restart executions/processes that need its environment. After resume or restore, wait for service ready before migrations or application startup.

Snapshot, restore, or volume failure

Snapshot creation: inspect member inventory, quiesce result, storage capacity, checksums, and terminal status.
Restore: target must be sleeping; verify image/backend/supervisor compatibility and keep the safety snapshot enabled.
Corruption: never bypass checksum failure. Preserve the snapshot metadata and select a different ready snapshot.
Volume attach/detach: use an absolute mount path; detach while the sandbox sleeps.
Volume grow: request a capacity strictly larger than the current value; shrink is unsupported.
Delete: snapshot must not be legally held/in use; volume must detach; destructive CLI commands require --yes.

CLI exit codes

Code	Meaning
`0`	Success
`1`	General or unmapped remote failure
`2`	CLI usage/validation error
`3`	Authentication or permission failure
`4`	Resource not found
`5`	Conflict or failed precondition
`6`	Network, server, cancellation timeout, or request deadline failure
`7`	Partial result
`130`	User cancellation signal

Escalation bundle

Provide only: request ID, operation ID, sandbox/resource IDs, UTC time range, CLI/SDK version, API error code, redacted command shape, desired/observed/readiness state, generation values, last event sequence, and relevant safe failure messages. State whether retry used the same idempotency key.

Security and persistence

Review recovery and data-loss rules.

API and SDKs

Review operation and error contracts.

​First-response checklist

​Stable API errors

​Lifecycle does not converge

​Execution or terminal fails

​File write conflicts

​Network or preview is unavailable

​Service is not ready

​Snapshot, restore, or volume failure

​CLI exit codes

​Escalation bundle