> ## Documentation Index > Fetch the complete documentation index at: https://docs.stackshift.cloud/llms.txt > Use this file to discover all available pages before exploring further. # Sandbox troubleshooting and recovery > Diagnose authentication, admission, lifecycle, execution, networking, storage, secret, snapshot, stream, and destructive-action failures. Start with durable state, not the last toast or terminal line. Preserve the request ID, sandbox ID, operation ID, resource ID, last event sequence, and UTC timestamp. Never include tokens, submitted secret values, terminal/port tickets, signed URLs, or volatile database credentials in support material. ## First-response checklist ```bash theme={null} stackshift --output json sandbox get stackshift --output json sandbox operation stackshift sandbox events --after-sequence ``` Compare: 1. `desired_state` with `observed_state` and readiness. 2. `generation` with `observed_generation`. 3. `active_operation.status`, `step`, attempts, deadline, and failure fields. 4. Effective capability decisions and their reason/remediation. 5. Recent events after the last sequence you successfully processed. A transport timeout means the outcome is unknown. Fetch the operation or resource before retrying. ## Stable API errors | Code | Meaning | Safe response | | ------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | | `invalid_request` | Malformed input or field validation failed | Correct reported field errors; keep the same logical intent | | `authentication_required` | Token missing, expired, or invalid | Re-authenticate; do not print the token | | `permission_denied` | Actor lacks required scope/role | Request the narrow permission; a UI control cannot bypass it | | `not_found` | ID is wrong, inaccessible, or tombstoned | Verify account/profile and exact ID; do not recreate blindly | | `conflict` | Resource/action conflicts with current state | Refetch resource and active operation | | `idempotency_key_required` | Mutation omitted its key | Resubmit once with a stable key | | `idempotency_conflict` | Same key was reused with different input | Retrieve the original result; use a new key only for a genuinely new action | | `version_conflict` | `If-Match` is stale | Refetch, review the new version, then decide whether to retry | | `quota_exceeded` | Account limit would be exceeded | Read the included quota name/current/requested/limit/unit; release capacity or change plan/request | | `capacity_unavailable` | Qualified server capacity cannot admit the request | Retry only when `retryable` and with backoff; do not create duplicates | | `capability_unavailable` | Target/image/backend cannot provide a required feature | Use the remediation or a compatible template/profile | | `invalid_state_transition` | Action is invalid from current state | Wait for the active operation or perform the prerequisite transition | | `deployment_server_unavailable` | Configured worker cannot currently accept work | Preserve the operation and wait/retry according to `retryable` | | `operation_deadline_exceeded` | Durable work exceeded its deadline | Inspect compensation/recovery state; do not assume rollback completed | | `operation_failed` | Operation reached terminal failure | Inspect failure code/message, attempts, and recovery availability | | `policy_denied` | Effective security/plan policy rejects the action | Narrow the request or obtain an authorized policy change | | `snapshot_incompatible` | Image, backend, protocol, members, or target mismatch | Choose a compatible sleeping target or correct fork inputs | | `snapshot_corrupt` | Manifest/chunk integrity verification failed | Do not restore; retain evidence and use another verified snapshot | | `rate_limited` | Request rate exceeded | Honor server guidance and use bounded exponential backoff with jitter | | `internal_error` | Unexpected control-plane failure | Retry only if marked retryable; report request/operation IDs | ## Lifecycle does not converge * `queued` or `scheduling`: inspect quota, capacity, target availability, and active operation deadline. * `provisioning` or `starting`: inspect image digest compatibility, supervisor protocol, storage, network-policy application, and worker failure code. * `degraded`: the sandbox exists but a health, policy, storage, port, service, or reconciliation condition needs attention. Avoid destructive retry loops. * `failed`: this is not successful cleanup. If `recovery_state` is `available` or `required`, review the failed operation and run recovery once. ```bash theme={null} stackshift --timeout 15m sandbox recover --wait ``` If recovery fails, preserve both operation IDs. Destroy only after reviewing retained data and confirming that recovery is no longer required. ## Execution or terminal fails | Symptom | Check | | ----------------------------- | ---------------------------------------------------------------------------------------------------------- | | Exit code from `sandbox exec` | The CLI propagates remote exit codes 1–125; inspect stderr and execution state | | `sandbox_not_running` | Resume and wait for readiness before starting work | | `timed_out` | Increase `--command-timeout` only after confirming the process is expected to run longer | | `oom_killed` | Inspect memory limit and usage; reduce workload or request an allowed larger profile | | Output missing | Resume logs from `next_offset`; handle truncation and completion explicitly | | Terminal rejected | Check readiness, writer lease ownership, ticket expiry, permission, and current operation | | Terminal disconnects | Request a new short-lived ticket and reconnect using the retained sequence; do not reuse an expired ticket | Use `--debug` only long enough to capture request diagnostics, then redact authorization, cookies, signed URLs, and headers before sharing. ## File write conflicts `writer lease` conflicts mean another active writer owns mutation authority. Coordinate before forced takeover. A checksum/version conflict means the file changed after it was read; reload, merge, and retry with the new expected checksum. Path errors usually indicate a non-absolute or disallowed mount path, traversal, or a symlink resolving outside the allowed root. Do not work around path validation by executing privileged shell commands. ## Network or preview is unavailable 1. Confirm the process is listening on the intended port and readiness passes. 2. Confirm a port resource exists and is `ready`; listening alone does not create ingress. 3. Confirm access mode and obtain a fresh private token/authenticated session. 4. Inspect effective egress/inbound policy and port access logs. 5. For outbound failures, resolve every redirect/DNS answer and check protected destination rules. A policy-denied private, loopback, link-local, metadata, or control-plane destination should not be retried through alternate encodings or DNS aliases. ## Service is not ready Inspect both the service resource and its operation. Check engine support, sidecar capacity/quota, owned volume encryption/state, image digest, sandbox availability, and service failure fields. The connection URL appears only inside the sandbox's volatile delivery file; its absence from API/UI output is expected. After service creation, restart executions/processes that need its environment. After resume or restore, wait for service `ready` before migrations or application startup. ## Snapshot, restore, or volume failure * Snapshot creation: inspect member inventory, quiesce result, storage capacity, checksums, and terminal status. * Restore: target must be sleeping; verify image/backend/supervisor compatibility and keep the safety snapshot enabled. * Corruption: never bypass checksum failure. Preserve the snapshot metadata and select a different ready snapshot. * Volume attach/detach: use an absolute mount path; detach while the sandbox sleeps. * Volume grow: request a capacity strictly larger than the current value; shrink is unsupported. * Delete: snapshot must not be legally held/in use; volume must detach; destructive CLI commands require `--yes`. ## CLI exit codes | Code | Meaning | | ----: | ------------------------------------------------------------------ | | `0` | Success | | `1` | General or unmapped remote failure | | `2` | CLI usage/validation error | | `3` | Authentication or permission failure | | `4` | Resource not found | | `5` | Conflict or failed precondition | | `6` | Network, server, cancellation timeout, or request deadline failure | | `7` | Partial result | | `130` | User cancellation signal | ## Escalation bundle Provide only: request ID, operation ID, sandbox/resource IDs, UTC time range, CLI/SDK version, API error code, redacted command shape, desired/observed/readiness state, generation values, last event sequence, and relevant safe failure messages. State whether retry used the same idempotency key. Review recovery and data-loss rules. Review operation and error contracts.