Schedule restore tests
A RestoreTest says what to restore, when, how often to retry, and what to keep. This page
explains each field with an example. The complete field list is in
API reference.
A RestoreTest name can have at most 50 characters. The names of its Restore objects add up to 13 characters and must fit a 63-character label value.
A complete example
Section titled “A complete example”The WordPress example for each tool. The tests differ only in provider, the backupSelector
and the scratch volume fields. Details per tool: Backup providers.
examples/kopiur/restoretest.yaml
# RestoreTest: restore the newest Kopiur Snapshot of each PVC and run the check.# Each backupSelector matches the policy label Kopiur sets on every Snapshot.apiVersion: bret.linogics.io/v1alpha1kind: RestoreTestmetadata: name: wordpressspec: provider: kopiur sources: - sourcePVC: html backupSelector: labels: kopiur.home-operations.com/config: html storageClassName: csi-hostpath-sc - sourcePVC: mysql backupSelector: labels: kopiur.home-operations.com/config: mysql storageClassName: csi-hostpath-sc restoreCheckRef: name: wordpress triggers: cron: "17 3 * * *" # 03:17 UTC every day. afterBackup: delay: 30m # Also 30 minutes after every source has a new backup. manual: true # On demand with the bret.linogics.io/run-now annotation. restoreWindow: 45m retries: count: 1 backoff: 10mexamples/volsync/restoretest.yaml
# RestoreTest: restore each PVC from its ReplicationSource and run the check.# The backup is the last sync (status.lastSyncTime). bret pins the restore to that time,# so a sync during the run does not change what is tested.apiVersion: bret.linogics.io/v1alpha1kind: RestoreTestmetadata: name: wordpressspec: provider: volsync sources: - sourcePVC: html backupSelector: name: html # The ReplicationSource. labels must match exactly one. storageClassName: csi-hostpath-sc - sourcePVC: mysql backupSelector: name: mysql storageClassName: csi-hostpath-sc restoreCheckRef: name: wordpress triggers: cron: "17 3 * * *" afterBackup: delay: 30m manual: true restoreWindow: 45m retries: count: 1 backoff: 10mexamples/velero/restoretest.yaml
# RestoreTest: restore the newest Completed Velero Backup that covers this namespace.# Every source has the same backupSelector, because one Backup holds both PVCs.apiVersion: bret.linogics.io/v1alpha1kind: RestoreTestmetadata: name: wordpressspec: provider: velero sources: - sourcePVC: html backupSelector: labels: velero.io/schedule-name: wordpress - sourcePVC: mysql backupSelector: labels: velero.io/schedule-name: wordpress velero: labelSelector: matchLabels: app.kubernetes.io/name: wordpress restoreCheckRef: name: wordpress triggers: cron: "17 3 * * *" afterBackup: delay: 30m manual: true restoreWindow: 45m retries: count: 1 backoff: 10mFields
Section titled “Fields”provider
Section titled “provider”One of velero, volsync, kopiur. The tool that made the backups. The operator uses the same
tool to restore them. Setup for each tool: providers/.
A test for a provider that is disabled in the chart gets the condition reason
ProviderUnsupported and starts no run.
sources
Section titled “sources”One entry per PVC that the app has. Each run restores all of them together. The sourcePVC values
must be exactly the sourcePVC values of the restoredVolumes in the RestoreCheck, or the Restore
ends Failed with SourceMissing. See Volumes and scratch space.
Each source has a sourcePVC, a backupSelector and the scratch volume fields (below).
A one-volume app has one entry:
sources: - sourcePVC: data backupSelector: labels: app.kubernetes.io/name: orders-dbbackupSelector
Section titled “backupSelector”Picks which backups one source uses. Set either:
name: one backup by name, orlabels: all backups with these labels. The test uses the newest one.
Set at least one. The API rejects an empty selector.
What name and labels select depends on the provider:
| Provider | Backup object | name |
labels |
|---|---|---|---|
| kopiur | Snapshot in the test namespace, phase Succeeded |
one Snapshot | Snapshots with these labels |
| volsync | ReplicationSource in the test namespace, with lastSyncTime |
the ReplicationSource | must match exactly one ReplicationSource |
| velero | Backup in the Velero namespace, phase Completed, covering the test namespace |
one Backup | Backups with these labels, e.g. velero.io/schedule-name |
Each run restores the newest backup of every source. For Velero every source must have the same
backupSelector, because one Backup holds all PVCs. For VolSync the run is pinned to the picked
sync time with restoreAsOf. The details are under “How bret picks backups” on each tool page in
providers/.
If a source has no matching backup, the test sets the reason BackupNotFound. The message names
the sources without a backup.
restoreCheckRef
Section titled “restoreCheckRef”name of a RestoreCheck in the same namespace as the RestoreTest. The RestoreCheck is the
pod template that checks the restored data. See Write a check.
Scratch volume per source
Section titled “Scratch volume per source”Each source is restored into its own scratch volume. The fields are in the source itself, next to
backupSelector: size, storageClassName and autoExpand. All are optional. Without size
and autoExpand, the source size and the default headroom decide. The size rules are in
Volumes and scratch space.
triggers
Section titled “triggers”Use any combination of three triggers. A run starts when any one of them fires.
| Field | Example | Behavior |
|---|---|---|
cron |
"17 3 * * *" |
5-field cron in UTC. The API checks the pattern. Each tick starts a run unless overlapPolicy says otherwise. |
afterBackup.delay |
30m |
Every source needs a new matching backup. When the last of them completes at time T, a run starts at T + delay. The newest handled backup of each source is recorded in status.lastHandledBackups. A backup counts as new only if it is newer than that (completion time, then name). Older backups never trigger. Backups that completed before the RestoreTest was created never trigger either. While some sources have a new backup and others do not, the BackupsPending condition is True with reason WaitingForBackups. Ready does not change. |
manual |
true |
Enables the run-now annotation. Any changed value on bret.linogics.io/run-now starts one run. Setting the same value again while it is still set does nothing. |
A manual run needs no CLI:
kubectl -n wordpress annotate restoretest wordpress bret.linogics.io/run-now="$(date +%s)" --overwriteThe flow from a trigger to a run result is in Restore test lifecycle.
Pick the times on purpose. A test that starts while the backup is still writing can fail for no
good reason. Leave some margin after the backup window, as the delay above does.
restoreWindow
Section titled “restoreWindow”Deadline for one attempt. The default is 45m. It covers the native restore, image pulls, and the check
pod. When it passes, the operator marks the attempt Canceled. It deletes the native object and
the scratch volume, unless keepOnFailure keeps them.
Size it for the largest backup that you test, plus image pulls. A database of 200 GB may need hours. Check how long a restore takes in your environment before you set this value.
retries
Section titled “retries”| Field | Default | Meaning |
|---|---|---|
count |
0 |
Extra attempts after the first. The maximum is 10. |
backoff |
5m |
Wait between attempts. |
With count: 2, the operator makes up to three attempts. It cleans up each failed attempt
before it starts the next one. A canceled attempt writes a RestoreCanceled Event on its Restore.
Retries help with transient failures, such as an image pull that times out, or a busy storage backend. They do not fix a broken check. A check that fails three times has a real problem.
keepOnFailure
Section titled “keepOnFailure”Default false. When true and the last attempt failed, the operator keeps:
- the check pod, so
kubectl logsandkubectl execstill work, - the native restore object,
- the scratch volume, so the restored data stays readable.
The kept objects get the label bret.linogics.io/retained: "true". The RestoreTest lists them
in status.retainedResources. The operator does not remove them automatically. v1 has no TTL
(retainedFor is not implemented). Inspect them, then delete the Restore. Its finalizer deletes
the check pod, the provider objects, the Velero scratch namespace and the PVCs.
When the run settles, the operator removes earlier failed attempts. Only the last attempt is kept.
Turn it on for a check that you still debug. Leave it off in a long-running schedule. Otherwise, failed runs pile up volumes.
overlapPolicy
Section titled “overlapPolicy”What happens when a trigger fires while a run is still going.
| Value | Behavior |
|---|---|
Skip (default) |
Write an Event and drop the trigger. |
Queue |
Keep one pending run. It starts when the running one ends. Extra triggers are dropped. |
Replace |
Delete the running Restore, then start a new run. The operator does not wait for the old one’s cleanup. Kubernetes garbage collection removes the objects it owned. |
Manual runs follow the same policy.
Skip is the safe default. Use Queue when runs are short and you never want to miss one.
Use Replace only when the newest backup must win.
Status
Section titled “Status”| Field | Meaning |
|---|---|
status.lastRun.time |
When the last run ended. |
status.lastRun.result |
Succeeded or Failed. |
status.lastRun.reason |
Free text. It copies the message of the Restore that ended the run. |
status.nextRun |
Next cron tick. |
status.lastHandledBackups |
Per source: name and time of the newest backup that afterBackup ran for. |
status.retainedResources |
Names of kept objects when keepOnFailure kept a failed run. |
status.conditions |
Standard conditions. The Ready reason is the run result (Succeeded or Failed), Running, or a blocking reason: BackupNotFound (names the sources without a backup). When the sources and the check volumes differ, the run ends Failed and status.lastRun.reason carries the SourceMissing message of the Restore. The BackupsPending condition is True (reason WaitingForBackups) while afterBackup waits for sources that have no new backup yet. |
Read it with:
kubectl -n wordpress get restoretest wordpress -o jsonpath='{.status.lastRun}{"\n"}'Recipes
Section titled “Recipes”The recipes show sources. The rest of the spec is as above.
Test only the newest backups of one app, nightly
Section titled “Test only the newest backups of one app, nightly”spec: provider: kopiur sources: - sourcePVC: html backupSelector: labels: app.kubernetes.io/name: nextcloud-html - sourcePVC: pgdata backupSelector: labels: app.kubernetes.io/name: nextcloud-pgdata restoreCheckRef: name: nextcloud triggers: cron: "30 4 * * *" restoreWindow: 60m retries: count: 1 backoff: 15mTest every new backup, and debug failures
Section titled “Test every new backup, and debug failures”spec: provider: kopiur sources: - sourcePVC: html backupSelector: labels: app.kubernetes.io/name: nextcloud-html - sourcePVC: pgdata backupSelector: labels: app.kubernetes.io/name: nextcloud-pgdata restoreCheckRef: name: nextcloud triggers: afterBackup: delay: 20m keepOnFailure: trueRun on demand only, with named backups
Section titled “Run on demand only, with named backups”spec: provider: kopiur sources: - sourcePVC: html backupSelector: name: html-1 - sourcePVC: pgdata backupSelector: name: pgdata-1 restoreCheckRef: name: nextcloud triggers: manual: trueOne volume
Section titled “One volume”spec: provider: kopiur sources: - sourcePVC: data backupSelector: labels: app.kubernetes.io/name: orders-db restoreCheckRef: name: minimal triggers: manual: trueNamespaces and scope
Section titled “Namespaces and scope”A RestoreTest lives in the namespace of the application that it tests. Its pods and volumes go there
too. The operator watches all namespaces by default. The chart value watchNamespaces (a list)
limits the operator to those namespaces. Empty means cluster-wide. See
How bret works.
