Restore workspace from backup #1572

Allda · 2026-01-14T12:18:51Z

What does this PR do?

Add init container for workspace restoration

A new init container is added to the workspace deployment in case user choose to restore the workspace from backup.

By setting the workspace attribute "controller.devfile.io/restore-workspace" the controller sets a new init container instead of cloning data from git repository.

By default an automated path to restore image is used based on cluster settings. However user is capable overwrite that value using another attribute "controller.devfile.io/restore-source-image".

The restore container runs a wokspace-recovery.sh script that pull an image using oras an extract files to a /project directory.

What issues does this PR fix or reference?

#1525

Is it tested? How?

No automated tests are available in the first phase. I will add tests once I get the first approval that the concept is ok.

How to test:

Configure a cluster to enable backups
Create a new workspace and make changes in any of its files and save it.
Stop the workspace `kubectl patch devworkspace restore-workspace-2 --type=merge -p '{"spec": {"started": false}}'
Wait till it is stopped, and backup is executed for the workspace (verify the backup image exists in the registry)
Delete a workspace from cluster kubectl delete devworkspace restore-workspace-2
Add an attribute to the workspace CRD as shown below (controller.devfile.io/restore-workspace)
Create a workspace
Wait till it boots up and verify the changed file is present

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  labels:
    controller.devfile.io/creator: ""
  name: restore-workspace-2
spec:
  started: true
  routingClass: 'basic'
  template:
    attributes:
      controller.devfile.io/storage-type: common
      controller.devfile.io/restore-workspace: 'true'

PR Checklist

E2E tests pass (when PR is ready, comment /test v8-devworkspace-operator-e2e, v8-che-happy-path to trigger)
- v8-devworkspace-operator-e2e: DevWorkspace e2e test
- v8-che-happy-path: Happy path for verification integration with Che

What's missing:

integration with registry authentication ✅
integration with built-in OCP registry ✅

pkg/library/env/workspaceenv.go

pkg/library/restore/restore.go

controllers/workspace/devworkspace_controller.go

pkg/library/restore/restore.go

pkg/library/storage/storage.go

pkg/library/restore/restore.go

pkg/constants/attributes.go

controllers/workspace/devworkspace_controller_test.go

pkg/library/restore/restore.go

pkg/constants/attributes.go

controllers/workspace/devworkspace_controller_test.go

apis/controller/v1alpha1/devworkspaceoperatorconfig_types.go

rohanKanojia · 2026-01-19T12:00:09Z

@Allda : I'm facing a strange issue while testing this functionality on CRC cluster . I've tried on both amd64 and arm64 variants but face same issue. I used samples/plain-workspace.yaml for testing.

Everything goes fine till step 4, but when I create the restore backup manifest I can see devworkspace resource is created but there is no corresponding pod for it:

oc create -f restore-dw.yaml
devworkspace.workspace.devfile.io/plain-devworkspace created
oc get dw
NAME                 DEVWORKSPACE ID             PHASE     INFO
plain-devworkspace   workspace612b8ddca9ff45d5   Running   Workspace is running
oc get pods
No resources found in rokumar-dev namespace.

I had just modified the name from the restore manifest you shared:

kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  labels:
    controller.devfile.io/creator: ""
  name: plain-devworkspace
spec:
  started: true
  routingClass: 'basic'
  template:
    attributes:
      controller.devfile.io/storage-type: common
      controller.devfile.io/restore-workspace: 'true'

Could you please check if I'm missing something?

Allda · 2026-01-21T12:36:34Z

@Allda : I'm facing a strange issue while testing this functionality on CRC cluster . I've tried on both amd64 and arm64 variants but face same issue. I used samples/plain-workspace.yaml for testing.

Everything goes fine till step 4, but when I create the restore backup manifest I can see devworkspace resource is created but there is no corresponding pod for it:
oc create -f restore-dw.yaml
devworkspace.workspace.devfile.io/plain-devworkspace created
oc get dw
NAME                 DEVWORKSPACE ID             PHASE     INFO
plain-devworkspace   workspace612b8ddca9ff45d5   Running   Workspace is running
oc get pods
No resources found in rokumar-dev namespace.
I had just modified the name from the restore manifest you shared:
kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  labels:
    controller.devfile.io/creator: ""
  name: plain-devworkspace
spec:
  started: true
  routingClass: 'basic'
  template:
    attributes:
      controller.devfile.io/storage-type: common
      controller.devfile.io/restore-workspace: 'true'
Could you please check if I'm missing something?

I am not sure why it doesn't work on your system. I tried the workspace you mentioned and the backup and other pods were created successfully.

~ k get deployments -n araszka
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
workspacefba99cfc93514828   1/1     1            1           40s

~ k get pods -n araszka                                                               
NAME                                         READY   STATUS             RESTARTS      AGE
fuse-builder-pod                             0/1     ImagePullBackOff   2 (20d ago)   63d
workspacefba99cfc93514828-84b865df77-t5gwz   1/1     Running            0             51s

Are there any logs you can share or see if the workspace has any pods at the very start?

rohanKanojia · 2026-01-21T12:40:18Z

@Allda : Okay, I'll try it again and report back

dkwon17 · 2026-01-21T23:20:54Z

I was able to successfully restore a real Dev Spaces project:

project-backup/workspace-recovery.sh

dkwon17 · 2026-01-22T22:51:04Z

pkg/library/restore/restore.go

+
+	if !hasContainerComponents(workspaceTemplate) {
+		// Avoid adding restore init container when DevWorkspace does not define any containers
+		return nil, nil


@rohanKanojia for your comment: #1572 (comment)

I believe you're running into this case?

Yeah I'm trying to literally create the same DevWorkspace manifest as provided in the description.

It started working when I provided full DevWorkspace manifest with restore attribute

openshift-ci · 2026-02-09T15:32:20Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Allda, ibuziuk
Once this PR has been reviewed and has the lgtm label, please assign dkwon17 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci · 2026-02-09T21:39:26Z

New changes are detected. LGTM label has been removed.

A new init container is added to the workspace deployment in case user choose to restore the workspace from backup. By setting workspace attribute "controller.devfile.io/restore-workspace" the controller sets a new init container instead of cloning data from git repository. By default an automated path to restore image is used based on cluster settings. However user is capable overwrite that value using another attribute "controller.devfile.io/restore-source-image". The restore container runs a wokspace-recovery.sh script that pull an image using oras an extract files to a /project directory. Signed-off-by: Ales Raszka <araszka@redhat.com>

A new tests that verifies the workspace is created from a backup. It checks if a deployment is ready and if it contains a new restore init container with proper configuration. There are 2 tests - one focused on common pvc and other that have per-workspace storage. Signed-off-by: Ales Raszka <araszka@redhat.com>

The condition whether an workspace should be restored from workspace was in the restore module itself. This make a reading a code more difficult. Now the condition is checked in the controller itself and restore container is only added when enabled. This commit also fixes few minor changes based on the code review comments: - Licence header - Attribute validation - Add a test for disabled workspace recovery - Typos Signed-off-by: Ales Raszka <araszka@redhat.com>

A new config is added to control the restore container. Default values are set for the new init container. It can be changed by user in the config. The config uses same logic as the project clone container config. Signed-off-by: Ales Raszka <araszka@redhat.com>

Signed-off-by: Ales Raszka <araszka@redhat.com>

The function handleRegistryAuthSecret is also needed for restore action. The backup controller is not a right place for it as it can't be reused. Moving it to separate module to be reusable. Signed-off-by: Ales Raszka <araszka@redhat.com>

The workspace restore process now support access to registries that require authentication. The default operator config is used to detect a right secret name and the secret is injected to the container as a volume. The second part of the commit adds support for OCP build-in registry. With this registry a user doesn't have to provide any secrets because a workspace service account have image puller role associated using role bindings. Signed-off-by: Ales Raszka <araszka@redhat.com>

The image stream created by the backup mechanism can't be owned by the workspace otherwise a deletion of the workspace also deletes the image stream. Removing the reference keep these 2 objects unrelated. Signed-off-by: Ales Raszka <araszka@redhat.com>

In case a workspace directory is not empty the restore process should just log the event and skip the restoration. This logic is added to not override data in case a workspace is resumed. Usual flow: - create a workspace and enable restoration - a restore container restores data from backup - a user updates files in workspace - a user stops a workspace - a user resumes a workspace - a restore container skips the restore process due to not empty dir Signed-off-by: Ales Raszka <araszka@redhat.com>

A new sample image is created in the CI and is used in the integration tests. Signed-off-by: Ales Raszka <araszka@redhat.com>

Signed-off-by: Ales Raszka <araszka@redhat.com>

In case the backup config is not present the value might be null and fails. The new condition handles it. Signed-off-by: Ales Raszka <araszka@redhat.com>

The delay between checking the empty dir and copying a backup content might cause an issue. This fix moves the check right before the content is being copied which minimize the delay. Signed-off-by: Ales Raszka <araszka@redhat.com>

The previous solution always deleted a secret if exist and re-create it. This can lead to potential issues. The new code uses SyncObjectWithCluster that is used across a whole codebase and minimize the risk of issues. Signed-off-by: Ales Raszka <araszka@redhat.com>

Signed-off-by: Ales Raszka <araszka@redhat.com>

Signed-off-by: David Kwon <dakwon@redhat.com>

dkwon17 · 2026-02-09T21:41:31Z

Added new commit to fix the rolebinding issue: 460cd25

and rebased.

I will merge this PR shortly

openshift-ci · 2026-02-09T22:54:46Z

@Allda: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/v14-che-happy-path	`460cd25`	link	true	`/test v14-che-happy-path`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Allda requested review from akurinnoy, dkwon17, ibuziuk and rohanKanojia as code owners January 14, 2026 12:18

tolusha reviewed Jan 15, 2026

View reviewed changes

pkg/library/env/workspaceenv.go Outdated Show resolved Hide resolved

tolusha reviewed Jan 15, 2026

View reviewed changes

pkg/library/restore/restore.go Outdated Show resolved Hide resolved

tolusha reviewed Jan 15, 2026

View reviewed changes

controllers/workspace/devworkspace_controller.go Outdated Show resolved Hide resolved

dkwon17 reviewed Jan 16, 2026

View reviewed changes

pkg/library/restore/restore.go Outdated Show resolved Hide resolved

dkwon17 reviewed Jan 16, 2026

View reviewed changes

pkg/library/storage/storage.go Outdated Show resolved Hide resolved

dkwon17 reviewed Jan 16, 2026

View reviewed changes

pkg/library/restore/restore.go Outdated Show resolved Hide resolved

dkwon17 reviewed Jan 16, 2026

View reviewed changes

pkg/constants/attributes.go Outdated Show resolved Hide resolved

openshift-merge-robot added the needs-rebase label Jan 16, 2026

Allda force-pushed the 23570-restore-workspace branch from 1b95d94 to 5480c1c Compare January 16, 2026 11:04

openshift-merge-robot removed the needs-rebase label Jan 16, 2026

Allda commented Jan 16, 2026

View reviewed changes

controllers/workspace/devworkspace_controller_test.go Show resolved Hide resolved