Failure reasons
bret reports why something stopped in the Ready condition of the object, in its
status.message, and as a Warning Event with the same reason. Read them with:
kubectl -n <namespace> describe restore <name>kubectl -n <namespace> describe restoretest <name>Reasons that only mark progress (RestoreStarted, CheckPassed, RunSkipped and others) are
on Metrics and events.
How a reason ends
Section titled “How a reason ends”Ends the Restore as Failed |
Waits and retries on its own | Blocks the RestoreTest |
|---|---|---|
SourceMissing, SourceMismatch, BackupUnsupported, RestoredVolumeMissing, RepositoryMissing, CheckRefMissing, SizeUnknown, InsufficientCapacity, RestoreFailed, ProviderUnsupported (Velero check) |
CheckNotFound, CheckUnsupported, ProviderUnsupported (provider disabled), RestoreNameInUse |
BackupNotFound, BackupSelectorAmbiguous, InvalidCron, ProviderUnsupported, SourceMissing |
A Failed attempt of a RestoreTest is retried until retries.count runs out. Fix the cause
first, or every retry fails the same way. RestoreCanceled ends the attempt as Canceled and
is not retried.
BackupNotFound
Section titled “BackupNotFound”- Shows on:
RestoreTestReady=False. With VolSync andrestoreAsOf, also on aRestore, which endsFailed. - Means: a source has no finished backup that matches its
backupSelector. The message names the sources. With VolSync, restic found no snapshot at or beforerestoreAsOf. - Fix: check the selector and the backups. What counts as finished differs per tool: see “Know which backup bret picks” on Velero, VolSync and Kopiur.
BackupSelectorAmbiguous
Section titled “BackupSelectorAmbiguous”- Shows on:
RestoreTestReady=False. VolSync only. - Means: the
labelsof a source match more than oneReplicationSource. The message names the matches. - Fix: use
backupSelector.name, or labels that only oneReplicationSourcecarries. See VolSync limitations.
BackupUnsupported
Section titled “BackupUnsupported”- Shows on:
Restore, endsFailedbefore Velero starts. Velero only. - Means: the
BackuphassnapshotVolumes: false, or a source PVC is in it only as file system backup. Velero writes file system backup data only while it restores the pod, and bret restores no pods, so the PVC would come back empty. - Fix: back the PVC up with CSI snapshots, best with
snapshotMoveData. See Volume data must be a snapshot.
CheckNotFound
Section titled “CheckNotFound”- Shows on:
RestoreReady=False. It retries until the check exists. - Means: the
RestoreCheckofrestoreCheckRefis not in the namespace of theRestore. - Fix: create the check in that namespace, or fix the name.
CheckRefMissing
Section titled “CheckRefMissing”- Shows on:
Restore, endsFailed. Velero only. - Means: a Secret or ConfigMap that the check names in
imagePullSecrets,envorenvFromis not in theRestorenamespace, so bret cannot copy it into the scratch namespace. - Fix: create it in the
Restorenamespace, or mark an env referenceoptional: true.
CheckUnsupported
Section titled “CheckUnsupported”- Shows on:
RestoreReady=False. It retries, so a fixed check is picked up. - Means: the check is invalid:
restoredVolumesis empty, asourcePVCis used twice or is not a DNS label, asourcePVCclashes with a scratch volume claim, the check pod cannot be built, or thebret.linogics.io/mover-uidannotation is not a JSON map. The same error shows asReady=False, reasonInvalid, on theRestoreCheck. - Fix: read the message and fix the check.
kubectl get restorecheckshows the result.
InsufficientCapacity
Section titled “InsufficientCapacity”- Shows on:
Restore, endsFailed. - Means:
size, or theautoExpandresult (for example amaxbelow the source), is smaller than the size of the source. - Fix: raise
size, or drop it and letautoExpandsize the volume. See Scratch volume size.
InvalidCron
Section titled “InvalidCron”- Shows on:
RestoreTestReady=False. - Means:
triggers.crondoes not parse. The message quotes it. - Fix: use a standard 5-field cron expression.
ProviderUnsupported
Section titled “ProviderUnsupported”- Shows on:
RestoreorRestoreTestReady=False. - Means: one of two things.
- The provider of
spec.provideris disabled in the chart (providers.<name>.enabled). The object waits. - Velero: the check uses
serviceAccountName,hostPath, or a configMap, secret, projected or persistentVolumeClaim volume. Those do not work in the scratch namespace. TheRestoreendsFailed.
- The provider of
- Fix: enable the provider, or pass Secrets to a Velero check with
envorenvFrom. See Velero limitations.
RepositoryMissing
Section titled “RepositoryMissing”- Shows on:
Restore, endsFailed. VolSync only. - Means: the
ReplicationSourcehas nospec.restic.repository, or that Secret is not in theRestorenamespace. - Fix: keep the repository Secret next to the
Restore. See VolSync limitations.
RestoreCanceled
Section titled “RestoreCanceled”- Shows on:
Restore, endsCanceled. Event reason. - Means:
restoreWindowran out before the restore and the check finished. - Fix: raise
restoreWindowfor large volumes or slow image pulls. See restoreWindow.
RestoreFailed
Section titled “RestoreFailed”- Shows on:
Restore, endsFailed. Event reason. - Means: the backup tool reported a failed restore (Velero
Failed,PartiallyFailedorFailedValidation, KopiurFailed), or the check pod exited non-zero. The message says which. - Fix: for a failed check, read the check pod logs. Keep the pod with
keepOnFailure: true. For a tool failure, read the native object named instatus.providerRefs.
RestoreNameInUse
Section titled “RestoreNameInUse”- Shows on:
RestoreTestReady=False. It retries on its own. - Means: the name of the next attempt is taken by a
Restorethat is being deleted, or that anotherRestoreTestowns. - Fix: usually none. The run starts once that
Restoreis gone.
RestoredVolumeMissing
Section titled “RestoredVolumeMissing”- Shows on:
Restore, endsFailed. Velero only. - Means: a
sourcePVCwas not restored, or the VeleroRestorecompleted without restoring any item. Thespec.veleroselector did not match the PVC, or theBackuplacks it. - Fix: label the PVC so the selector matches, and take a new backup. See Velero limitations.
SizeUnknown
Section titled “SizeUnknown”- Shows on:
Restore, endsFailed. - Means:
sizeis not set and the source size is unknown, so bret cannot size the scratch volume. - Fix: set
sizeon the source. See Scratch volume size.
SourceMismatch
Section titled “SourceMismatch”- Shows on:
Restore, endsFailed. Kopiur and VolSync. - Means: the backup holds another PVC than the
sourcePVCof its source. A KopiurSnapshotPolicywith severalpvc:entries backs up only the first one. - Fix: point the source at the backup of that PVC. With Kopiur, use one policy per PVC.
SourceMissing
Section titled “SourceMissing”- Shows on:
Restore, endsFailed. On aRestoreTest,Ready=Falsewhen it has no sources. - Means: the
sourcePVCvalues of the check (restoredVolumes) and of theRestoreorRestoreTest(sources) differ, or the object has no sources at all. - Fix: add the missing entry on either side. See Which PVC gets restored.
Nothing happens after a RestoreTest is created
Section titled “Nothing happens after a RestoreTest is created”- No condition and no Events: the namespace is not in
watchNamespaces, so the operator ignores the object. Orspec.triggersis not set, or no trigger is due yet. Wait, or start a manual run. See Schedule restore tests. BackupsPendingisTruewith reasonWaitingForBackups: withafterBackup, some sources have a new backup and others do not yet. The run starts when every source has one.- A source mismatch is not a
Readyreason of theRestoreTest. The run endsFailedandstatus.lastRun.reasoncarries the message of theRestore.
Errors inside check pods
Section titled “Errors inside check pods”Postgres: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750)
Section titled “Postgres: Permissions should be u=rwx (0700) or u=rwx,g=rx (0750)”- Cause: the restored data directory has the wrong mode. A pod
fsGroupchanged it to2770. - Fix: remove
fsGroupfrom the database container. Or run apgdata-permsinit container that runschmod 0700 "$PGDATA"as the database UID. The Postgres checks do this.
Postgres: another server might be running; trying to start server anyway
Section titled “Postgres: another server might be running; trying to start server anyway”- Cause: a crash-consistent copy keeps
postmaster.pid. Postgres checks whether that PID is alive, finds a different process, and starts anyway. - Fix: none needed. The message is a warning. The check still starts the server.
Postgres: pg_amcheck: error: no relations to check
Section titled “Postgres: pg_amcheck: error: no relations to check”- Cause:
amcheckis not installed in any database in the scratch copy.pg_amcheckfinds nothing to check and exits 1. - Fix: run
create extension if not exists amcheckin each database beforepg_amcheck. The scratch copy is the right place for this. Do not change the source.
Errors of the backup tools themselves are on their pages: Velero, VolSync, Kopiur.
