fix: clean up Temporal server-side versioning data on TWD deletion

[anujagrawal380]

• Add a finalizer to TemporalWorkerDeployment to run Temporal server-side cleanup before K8s deletion
• Add a finalizer to TemporalConnection to prevent it from being deleted while any TWD still references it
• On TWD deletion, set current version to unversioned, clear ramping version, and delete registered versions

Problem

When a TemporalWorkerDeployment CRD is deleted (e.g., switching back to plain Deployments), the Temporal server retains the build ID routing configuration. The matching service continues routing new tasks to the deleted build ID's physical queue, while unversioned workers poll a different physical queue. Tasks sit in Scheduled state indefinitely with no errors.

A secondary race condition exists: Helm deletes both the TemporalConnection and TWD in the same upgrade. Without the connection, the controller cannot talk to Temporal to clean up. This is solved by adding a finalizer to the TemporalConnection that blocks its deletion until all referencing TWDs are gone.

Changes

internal/controller/worker_controller.go:

TWD finalizer (temporal.io/worker-deployment-cleanup):

• Added to all TWD resources during normal reconciliation
• On deletion, triggers handleDeletion() which:
  1. Sets the current version to unversioned (BuildID: "") -- the critical step that unblocks task dispatch
  2. Clears any ramping version
  3. Deletes all registered versions with SkipDrainage: true
  4. Attempts to delete the deployment record itself
  5. Removes the connection finalizer if no other TWDs reference it
  6. Removes its own finalizer, allowing K8s to complete deletion

TemporalConnection finalizer (temporal.io/connection-in-use):

• Added to the TemporalConnection during normal TWD reconciliation via ensureConnectionFinalizer()
• Prevents the connection from being deleted while any TWD still references it
• Removed by removeConnectionFinalizerIfUnused() during TWD deletion, after checking no other TWDs in the same namespace reference the connection
• Guarantees the connection is always available during TWD cleanup -- no race condition with Helm deleting both resources simultaneously

RBAC updates:

• Added update;patch verbs for temporalconnections (was get;list;watch)
• Added update verb for temporalconnections/finalizers

Deletion flow

Helm upgrade (TWD disabled)
  |
  v
Helm deletes TWD CRD + TemporalConnection CRD simultaneously
  |
  +--> TemporalConnection: has finalizer, K8s sets DeletionTimestamp but blocks deletion
  |
  +--> TWD: has finalizer, K8s sets DeletionTimestamp, triggers Reconcile
         |
         v
       handleDeletion() runs:
         1. Fetches TemporalConnection (guaranteed to exist via finalizer)
         2. Connects to Temporal server
         3. Sets current version to unversioned
         4. Deletes versions
         5. Removes connection finalizer (no other TWDs reference it)
         6. Removes TWD finalizer
         |
         v
       TWD deleted by K8s
         |
         v
       TemporalConnection: no more finalizers, deleted by K8s

Issue #55
Issue #166

[CLAassistant]

All committers have signed the CLA.

[anujagrawal380]

PTAL @carlydf

[jaypipes]

@anujagrawal380 awesome contribution, thank you so much for this PR! I couple really minor comments below, but overall excellent work.

[anujagrawal380]

@anujagrawal380 awesome contribution, thank you so much for this PR! I couple really minor comments below, but overall excellent work.

Thanks, resolved both the comments!

[jaypipes]

rock on :) nice work on this @anujagrawal380!

[carlydf]

we need integration tests for this before merging to main / including it in a release

[anujagrawal380]

we need integration tests for this before merging to main / including it in a release

@carlydf Added the integration tests. PTAL