feat: Implement lease-based concurrency backend for PipelineRun queuing#2687
feat: Implement lease-based concurrency backend for PipelineRun queuing#2687chmouel wants to merge 2 commits intotektoncd:mainfrom
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new Kubernetes-backed concurrency queue backend called "lease" to Pipelines-as-Code, alongside the existing in-process "memory" backend. This new coordination mechanism uses Kubernetes Lease objects and PipelineRun annotations to manage concurrent execution more resiliently across watcher restarts. The changes include the implementation of a LeaseManager, a background recovery loop in the controller, and enhanced debugging visibility through new PipelineRun annotations. Feedback was provided to optimize performance in the LeaseManager by using server-side label selectors when listing PipelineRuns.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2687 +/- ##
==========================================
+ Coverage 58.85% 59.85% +0.99%
==========================================
Files 204 207 +3
Lines 20149 21201 +1052
==========================================
+ Hits 11859 12690 +831
- Misses 7525 7703 +178
- Partials 765 808 +43 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
chmouel
force-pushed
the
concurrency-lease-backend
branch
8 times, most recently
from
af6d7ca to
57be58c
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces a new lease-backed concurrency backend for Pipelines-as-Code, providing a more resilient alternative to the default in-memory queue by using Kubernetes Leases and PipelineRun claims. The implementation includes a background recovery loop to handle stalled promotions and adds comprehensive debugging annotations and events. Feedback focuses on performance optimizations, specifically recommending the use of label selectors instead of labels.Everything() in the recovery loop and transitioning from direct API list calls to informer-based listers in the reconciliation path to minimize API server load.
Introduced an opt-in concurrency backend that used Kubernetes Lease objects and PipelineRun annotations for queue coordination. This addressed potential state drift and race conditions during watcher restarts or API delays by storing the queue state in the cluster instead of only in memory. Added a global configuration setting to choose between the legacy memory backend and the new lease-based logic. Updated the queue manager interface to support context-aware operations and ensured that stale claims were automatically reclaimed via TTL-based expiration. The system preserved existing lease objects during the release process instead of completely deleting them. This reused the same records across multiple acquisition cycles, which reduced excessive communication overhead with the cluster. AI-assisted-by: Cursor (Codex) Signed-off-by: Chmouel Boudjnah <chmouel@redhat.com>
Refactored the event checking logic to specifically target and report warning events instead of filtering out individual event reasons. This ensures that unexpected errors are correctly identified during test validation.
chmouel
force-pushed
the
concurrency-lease-backend
branch
from
57be58c to
04733a6
Compare
|
closing for now, will come back to it when time allows it |
2 participants
📝 Description of the Change
Implements a lease-based concurrency backend that replaces the previous in-memory queue mechanism for coordinating PipelineRun execution. This enables cross-replica queue management in distributed deployments.
Key Features
🧪 Testing Strategy
🤖 AI Assistance
✅ Submitter Checklist
fix:,feat:) matches the "Type of Change" I selected above.make testandmake lintlocally to check for and fix any issues.