Database troubleshooting

​

Goal

Identify the failing database stage and take a safe action without bypassing qualification, validation, or rollback evidence.

​

Prerequisites

• An existing database

​

Workflow

​

Classify the failure first

• Stuck in provisioning or failed: a creation/runtime problem — check the database status and logs.
• Running but unreachable: a connectivity or credentials problem, not a database-down problem.
• Backup capture/restore failed: inspect recovery storage and the durable backup operation.
• Upload/fetch/export/restore/validate failed: inspect the transfer stage and current attempt evidence.
• Promoting/rolling back failed: preserve stable-endpoint and agent journal evidence; do not manually rewrite proxy/firewall state.
• Download not found: distinguish token expiry/consumption from missing gateway routing.
• Deleting with an error: inspect the delete operation and its last error; deletion is not complete while the resource remains visible.

​

Connectivity

• Re-open the credentials from the database surface rather than reusing a stale connection string.
• Confirm whether the app should use direct or pooled access; pointing at the wrong endpoint can look like an outage.
• Verify the required TLS mode, CA, private/public policy, and allowlist.
• Check the metrics and logs to see whether connections are being refused or the database is saturated.

​

Tunnel access is unavailable or stuck

• Check Public access in Credentials. A disabled StackShift tunnel choice includes an eligibility reason, such as an unsupported engine or an agent waiting for its tunnel update or heartbeat.
• A pending or applying policy has no usable public endpoint yet. Wait for Active before copying connection details.
• If the policy fails, use its displayed failure phase and code. A node or edge acknowledgement, TLS probe, or route lease failure needs investigation; do not switch to direct access as a workaround.
• For Restricted by IP, confirm the client’s actual public source IP belongs to an allowed CIDR and the policy has not expired. Verify the displayed port and TLS hostname db-tunnel.stackshift.cloud with the downloaded CA.
• To switch transport, revoke the old policy and wait for Revoked before creating a new one. A stale failed direct policy may also need revocation before a fresh request.

​

Deletion needs attention

Choose Delete Database from the database menu, or send DELETE to the project or standalone database resource URL. The accepted response starts asynchronous retirement. Wait until the resource leaves the active list or reports deleted. If the UI shows Deletion needs attention, the delete operation has a last error and retirement is incomplete. Capture the database ID, operation state, target node, and safe error for support. Do not treat the request acknowledgement as proof that data and network artifacts were removed.

​

Import / Export is missing or returns not found

• Confirm the source-controlled Database Transfer V2 capability is enabled in the deployed release.
• Confirm the database is running PostgreSQL/MySQL on qualified hosted V2 containerd with stable lineage and transfer-capable agent evidence.
• Use the project route for a project database and the standalone route only for an owner-only standalone database.
• Redis, connected-node databases, hidden candidates, cross-tenant IDs, and unqualified runtimes intentionally remain unavailable.

​

Upload and URL import

• Multipart incomplete: reselect the same file and reconcile the identical session; upload only missing parts before starting import.
• Idempotency conflict: the key was reused with different canonical filename, size, checksum, parts, or source; use the original request or a new key.
• Checksum mismatch: hash stored bytes; for gzip, hash the .gz file itself.
• URL blocked: use verified HTTPS and ensure every redirect/DNS answer/peer stays outside private, metadata, control-plane, and deployment networks.
• URL stalled/oversized: use a reliable object origin or resumable upload; streamed limits apply regardless of Content-Length.

​

Generic export or restore pipeline failure

A high-level database transfer export pipeline failed message does not identify the failed subprocess. Inspect operation and worker/agent evidence for the first failed dump client, compressor, stored/expanded byte limit, object upload, multipart completion, checksum, decompression, SQL guard, or SQL client. A later successful step cannot hide an earlier failed status.

​

Validation, promotion, and rollback

• Validation failed: read individual checksum, SQL-client, health, integrity-tool, counts, and manifest evidence; do not promote.
• Promotion unavailable: candidate must be ready_to_promote, current validation must pass, and typed name must exactly match case.
• Promotion incomplete: allow the fenced journal to attest completion or automatically roll back; do not manually change proxy/firewall state.
• Manual rollback unavailable: the transfer must remain active with rollback available, the promoted generation must still be current, and time must be before rollback_deadline.

​

Download returns not found

Download tokens are opaque, single-use, and valid for 10 minutes. A browser scanner, preview, or interrupted first request can consume a token. Request a fresh link instead of replaying it. Range downloads are not supported.

• A valid download path starts with /api/database-transfers/download/, not the project or standalone transfer base.
• If a newly issued token fails immediately, operators must verify the isolated download gateway and current edge-proxy route.
• A token consumed before an object-store read failure is intentionally not restored; renew after storage health is fixed.

​

What to include when escalating

• Database, transfer, backup, and operation IDs; engine/version; runtime and agent digest/version.
• State, stage, attempt count, safe code/message, timestamps, and whether a new operation also fails.
• Whether the active contents or stable endpoint changed and the current rollback state/deadline.
• Never include credentials, object-store secrets, URL query strings, candidate credentials/IDs, or encryption identities.

​

Expected result

​

Common failures

​

Related guides