Race condition between backup and database deletion causes stuck resources

[tartkri-bot]

Summary

When a PostgreSQL database is deleted while an initial backup is still running, the operator removes pgbackrest-secrets before the backup completes, causing the backup to fail and the database to get stuck in Deleting state.

Steps to Reproduce

1. Provision a PostgreSQL database with backups enabled (scheduled + PITR)
2. Wait for database to reach Ready state
3. Immediately delete the database (while backup is still In Progress)
4. Database gets stuck in deletion

Actual Behavior

1. Database deletion is initiated
2. Operator removes <db-name>-pgbackrest-secrets Secret
3. Running backup pod fails with:

   MountVolume.SetUp failed for volume "pgbackrest-config": 
   secret "<db-name>-pgbackrest-secrets" not found
   

4. Backup remains stuck Running, database stuck Deleting

Resource State When Stuck

$ kubectl get pg  # cluster gone
No resources found

$ kubectl get perconapgbackups
NAME                           STATUS    AGE
postgresql-wza-backup-lcmn     Running   5m13s

$ kubectl get perconapgcluster
NAME             STATUS     AGE
postgresql-wza   Deleting   10m

Root Cause

Deletion flow removes Secrets before checking/waiting on in-progress backups. The backup pod needs the Secret to mount its config but it is already deleted.

Workaround

Manually recreate the pgbackrest-secrets Secret (clone from another DB using same S3 bucket). This allows backup to finish and deletion to proceed.

Proposed Fix

Ensure proper deletion ordering: wait for or cancel in-progress backups before deleting pgbackrest-secrets.

Environment

• Operator: percona-postgresql-operator (via everest-operator)
• Storage: S3
• Discovered during: automated health check workflows with rapid create/delete cycles

[spron-in]

is going to solve: openeverest/openeverest#1914

[pooknull]

The issue with deleted secret is caused by the everest-operator. It creates this secret with an owner reference pointing to the cluster:
https://github.com/openeverest/openeverest-operator/blob/df4fbae671e541666b907e87a6136d67d1f71bb6/internal/controller/everest/providers/pg/applier.go#L1233-L1236

As a result, when the cluster is deleted, kubernetes also deletes the secret.

The issue where the backup remains stuck in the Running state is on the percona-postgresql-operator side and will be fixed soon.

[pooknull]

I have created https://perconadev.atlassian.net/browse/K8SPG-992 for this issue

[spron-in]

@pooknull

The issue with deleted secret is caused by the everest-operator. It creates this secret with an owner reference pointing to the cluster

But it is kinda how it should be. Deleting the cluster should also lead to the deletion of the secret. Not having an owner reference leads to plethora of other issues.

[egegunes]

@spron-in actually, no. we don't delete secrets on cluster deletion by default. deletion of secrets is tightly coupled with deletion of PVCs for us and PVC deletion is controlled by a special finalizer.