Skip to content

Schedule restore tests

A RestoreTest says what to restore, when, how often to retry, and what to keep. This page explains each field with an example. The complete field list is in API reference.

A RestoreTest name can have at most 50 characters. The names of its Restore objects add up to 13 characters and must fit a 63-character label value.

The WordPress example for each tool. The tests differ only in provider, the backupSelector and the scratch volume fields. Details per tool: Backup providers.

examples/kopiur/restoretest.yaml

# RestoreTest: restore the newest Kopiur Snapshot of each PVC and run the check.
# Each backupSelector matches the policy label Kopiur sets on every Snapshot.
apiVersion: bret.linogics.io/v1alpha1
kind: RestoreTest
metadata:
name: wordpress
spec:
provider: kopiur
sources:
- sourcePVC: html
backupSelector:
labels:
kopiur.home-operations.com/config: html
storageClassName: csi-hostpath-sc
- sourcePVC: mysql
backupSelector:
labels:
kopiur.home-operations.com/config: mysql
storageClassName: csi-hostpath-sc
restoreCheckRef:
name: wordpress
triggers:
cron: "17 3 * * *" # 03:17 UTC every day.
afterBackup:
delay: 30m # Also 30 minutes after every source has a new backup.
manual: true # On demand with the bret.linogics.io/run-now annotation.
restoreWindow: 45m
retries:
count: 1
backoff: 10m

One of velero, volsync, kopiur. The tool that made the backups. The operator uses the same tool to restore them. Setup for each tool: providers/.

A test for a provider that is disabled in the chart gets the condition reason ProviderUnsupported and starts no run.

One entry per PVC that the app has. Each run restores all of them together. The sourcePVC values must be exactly the sourcePVC values of the restoredVolumes in the RestoreCheck, or the Restore ends Failed with SourceMissing. See Volumes and scratch space.

Each source has a sourcePVC, a backupSelector and the scratch volume fields (below).

A one-volume app has one entry:

sources:
- sourcePVC: data
backupSelector:
labels:
app.kubernetes.io/name: orders-db

Picks which backups one source uses. Set either:

  • name: one backup by name, or
  • labels: all backups with these labels. The test uses the newest one.

Set at least one. The API rejects an empty selector.

What name and labels select depends on the provider:

Provider Backup object name labels
kopiur Snapshot in the test namespace, phase Succeeded one Snapshot Snapshots with these labels
volsync ReplicationSource in the test namespace, with lastSyncTime the ReplicationSource must match exactly one ReplicationSource
velero Backup in the Velero namespace, phase Completed, covering the test namespace one Backup Backups with these labels, e.g. velero.io/schedule-name

Each run restores the newest backup of every source. For Velero every source must have the same backupSelector, because one Backup holds all PVCs. For VolSync the run is pinned to the picked sync time with restoreAsOf. The details are under “How bret picks backups” on each tool page in providers/.

If a source has no matching backup, the test sets the reason BackupNotFound. The message names the sources without a backup.

name of a RestoreCheck in the same namespace as the RestoreTest. The RestoreCheck is the pod template that checks the restored data. See Write a check.

Each source is restored into its own scratch volume. The fields are in the source itself, next to backupSelector: size, storageClassName and autoExpand. All are optional. Without size and autoExpand, the source size and the default headroom decide. The size rules are in Volumes and scratch space.

Use any combination of three triggers. A run starts when any one of them fires.

Field Example Behavior
cron "17 3 * * *" 5-field cron in UTC. The API checks the pattern. Each tick starts a run unless overlapPolicy says otherwise.
afterBackup.delay 30m Every source needs a new matching backup. When the last of them completes at time T, a run starts at T + delay. The newest handled backup of each source is recorded in status.lastHandledBackups. A backup counts as new only if it is newer than that (completion time, then name). Older backups never trigger. Backups that completed before the RestoreTest was created never trigger either. While some sources have a new backup and others do not, the BackupsPending condition is True with reason WaitingForBackups. Ready does not change.
manual true Enables the run-now annotation. Any changed value on bret.linogics.io/run-now starts one run. Setting the same value again while it is still set does nothing.

A manual run needs no CLI:

Terminal window
kubectl -n wordpress annotate restoretest wordpress bret.linogics.io/run-now="$(date +%s)" --overwrite

The flow from a trigger to a run result is in Restore test lifecycle.

Pick the times on purpose. A test that starts while the backup is still writing can fail for no good reason. Leave some margin after the backup window, as the delay above does.

Deadline for one attempt. The default is 45m. It covers the native restore, image pulls, and the check pod. When it passes, the operator marks the attempt Canceled. It deletes the native object and the scratch volume, unless keepOnFailure keeps them.

Size it for the largest backup that you test, plus image pulls. A database of 200 GB may need hours. Check how long a restore takes in your environment before you set this value.

Field Default Meaning
count 0 Extra attempts after the first. The maximum is 10.
backoff 5m Wait between attempts.

With count: 2, the operator makes up to three attempts. It cleans up each failed attempt before it starts the next one. A canceled attempt writes a RestoreCanceled Event on its Restore.

Retries help with transient failures, such as an image pull that times out, or a busy storage backend. They do not fix a broken check. A check that fails three times has a real problem.

Default false. When true and the last attempt failed, the operator keeps:

  • the check pod, so kubectl logs and kubectl exec still work,
  • the native restore object,
  • the scratch volume, so the restored data stays readable.

The kept objects get the label bret.linogics.io/retained: "true". The RestoreTest lists them in status.retainedResources. The operator does not remove them automatically. v1 has no TTL (retainedFor is not implemented). Inspect them, then delete the Restore. Its finalizer deletes the check pod, the provider objects, the Velero scratch namespace and the PVCs.

When the run settles, the operator removes earlier failed attempts. Only the last attempt is kept.

Turn it on for a check that you still debug. Leave it off in a long-running schedule. Otherwise, failed runs pile up volumes.

What happens when a trigger fires while a run is still going.

Value Behavior
Skip (default) Write an Event and drop the trigger.
Queue Keep one pending run. It starts when the running one ends. Extra triggers are dropped.
Replace Delete the running Restore, then start a new run. The operator does not wait for the old one’s cleanup. Kubernetes garbage collection removes the objects it owned.

Manual runs follow the same policy.

Skip is the safe default. Use Queue when runs are short and you never want to miss one. Use Replace only when the newest backup must win.

Field Meaning
status.lastRun.time When the last run ended.
status.lastRun.result Succeeded or Failed.
status.lastRun.reason Free text. It copies the message of the Restore that ended the run.
status.nextRun Next cron tick.
status.lastHandledBackups Per source: name and time of the newest backup that afterBackup ran for.
status.retainedResources Names of kept objects when keepOnFailure kept a failed run.
status.conditions Standard conditions. The Ready reason is the run result (Succeeded or Failed), Running, or a blocking reason: BackupNotFound (names the sources without a backup). When the sources and the check volumes differ, the run ends Failed and status.lastRun.reason carries the SourceMissing message of the Restore. The BackupsPending condition is True (reason WaitingForBackups) while afterBackup waits for sources that have no new backup yet.

Read it with:

Terminal window
kubectl -n wordpress get restoretest wordpress -o jsonpath='{.status.lastRun}{"\n"}'

The recipes show sources. The rest of the spec is as above.

Test only the newest backups of one app, nightly

Section titled “Test only the newest backups of one app, nightly”
spec:
provider: kopiur
sources:
- sourcePVC: html
backupSelector:
labels:
app.kubernetes.io/name: nextcloud-html
- sourcePVC: pgdata
backupSelector:
labels:
app.kubernetes.io/name: nextcloud-pgdata
restoreCheckRef:
name: nextcloud
triggers:
cron: "30 4 * * *"
restoreWindow: 60m
retries:
count: 1
backoff: 15m
spec:
provider: kopiur
sources:
- sourcePVC: html
backupSelector:
labels:
app.kubernetes.io/name: nextcloud-html
- sourcePVC: pgdata
backupSelector:
labels:
app.kubernetes.io/name: nextcloud-pgdata
restoreCheckRef:
name: nextcloud
triggers:
afterBackup:
delay: 20m
keepOnFailure: true
spec:
provider: kopiur
sources:
- sourcePVC: html
backupSelector:
name: html-1
- sourcePVC: pgdata
backupSelector:
name: pgdata-1
restoreCheckRef:
name: nextcloud
triggers:
manual: true
spec:
provider: kopiur
sources:
- sourcePVC: data
backupSelector:
labels:
app.kubernetes.io/name: orders-db
restoreCheckRef:
name: minimal
triggers:
manual: true

A RestoreTest lives in the namespace of the application that it tests. Its pods and volumes go there too. The operator watches all namespaces by default. The chart value watchNamespaces (a list) limits the operator to those namespaces. Empty means cluster-wide. See How bret works.