Skip to content

fix(worker-ops): recover failed workers before scheduler gating #69

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
liobrasil wants to merge 4 commits into from
Closed

fix(worker-ops): recover failed workers before scheduler gating #69

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
14fcda4
fix(worker-ops): recover failed workers before scheduler gating
liobrasil Mar 11, 2026
4ca30be
docs(worker-ops): clarify failed worker recovery comment
liobrasil Mar 12, 2026
cf92817
Adjust worker recovery effort budgeting
liobrasil Mar 12, 2026
2666212
Increase close worker effort to 5000
liobrasil Mar 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions FLOW_YIELD_VAULTS_EVM_BRIDGE_DESIGN.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -504,11 +504,11 @@ Execution effort values are configurable via the `executionEffortConstants` dict

| Key | Default | Description |
|-----|---------|-------------|
| `schedulerBaseEffort` | 700 | Base effort for SchedulerHandler execution |
| `schedulerPerRequestEffort` | 1000 | Additional effort per request preprocessed |
| `workerCreateYieldVaultRequestEffort` | 5000 | Effort for CREATE_YIELDVAULT requests |
| `workerDepositRequestEffort` | 2000 | Effort for DEPOSIT_TO_YIELDVAULT requests |
| `workerWithdrawRequestEffort` | 2000 | Effort for WITHDRAW_FROM_YIELDVAULT requests |
| `schedulerBaseEffort` | 950 | Base effort for SchedulerHandler execution |
| `schedulerPerRequestEffort` | 250 | Additional effort per request preprocessed |
| `workerCreateYieldVaultRequestEffort` | 6500 | Effort for CREATE_YIELDVAULT requests |
| `workerDepositRequestEffort` | 1500 | Effort for DEPOSIT_TO_YIELDVAULT requests |
| `workerWithdrawRequestEffort` | 3000 | Effort for WITHDRAW_FROM_YIELDVAULT requests |
| `workerCloseYieldVaultRequestEffort` | 5000 | Effort for CLOSE_YIELDVAULT requests |

Priority is dynamically determined based on execution effort:
Expand Down
37 changes: 22 additions & 15 deletions cadence/contracts/FlowYieldVaultsEVMWorkerOps.cdc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -477,6 +477,10 @@ access(all) contract FlowYieldVaultsEVMWorkerOps {
let manager = FlowYieldVaultsEVMWorkerOps._getManagerFromStorage()!
let worker = self.workerCap.borrow()!

// Always clear failed/stale worker entries before capacity and backlog checks.
// This keeps failed requests recoverable even when no new EVM requests are pending.
self._checkForFailedWorkerRequests(manager: manager, worker: worker)

var message = ""
var nextRunCapacity: UInt8 = 0
var pendingCount: Int? = nil
Expand Down Expand Up @@ -556,12 +560,10 @@ access(all) contract FlowYieldVaultsEVMWorkerOps {

/// @notice Main scheduler logic
/// @dev Flow:
/// 1. Check for failed worker requests
/// - If a failure is identified, mark the request as failed and remove it from scheduledRequests
/// 2. If fetchCount > 0, fetch pending requests from EVM
/// 3. Preprocess requests to drop invalid requests
/// 4. Start processing requests (PENDING -> PROCESSING)
/// 5. Schedule WorkerHandlers and assign request ids to them
/// 1. If fetchCount > 0, fetch pending requests from EVM
/// 2. Preprocess requests to drop invalid requests
/// 3. Start processing requests (PENDING -> PROCESSING)
/// 4. Schedule WorkerHandlers and assign request ids to them
/// @param manager The scheduler manager
/// @param worker The worker resource
/// @param fetchCount Number of pending requests to fetch in this run
Expand All @@ -571,9 +573,6 @@ access(all) contract FlowYieldVaultsEVMWorkerOps {
worker: &FlowYieldVaultsEVM.Worker,
fetchCount: Int,
): String? {
// Check for failed worker requests
self._checkForFailedWorkerRequests(manager: manager, worker: worker)

// Fetch pending requests from EVM
if fetchCount > 0 {
if let pendingRequests = worker.getPendingRequestsFromEVM(
Expand Down Expand Up @@ -749,7 +748,15 @@ access(all) contract FlowYieldVaultsEVMWorkerOps {
let perRequestEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.SCHEDULER_PER_REQUEST_EFFORT
]!
let executionEffort = baseEffort + UInt64(forNumberOfRequests) * perRequestEffort
// Budget for the larger of:
// - requests expected to be processed next run
// - tracked worker entries that may need recovery before any new work is scheduled
let trackedRecoveryWorkload = UInt64(FlowYieldVaultsEVMWorkerOps.scheduledRequests.length)
let requestedProcessingWorkload = UInt64(forNumberOfRequests)
let schedulerWorkload = trackedRecoveryWorkload > requestedProcessingWorkload
? trackedRecoveryWorkload
: requestedProcessingWorkload
let executionEffort = baseEffort + schedulerWorkload * perRequestEffort

let transactionId = self._scheduleTransaction(
manager: manager,
Expand Down Expand Up @@ -941,11 +948,11 @@ access(all) contract FlowYieldVaultsEVMWorkerOps {
self.WORKER_CLOSE_YIELDVAULT_REQUEST_EFFORT = "workerCloseYieldVaultRequestEffort"

self.executionEffortConstants = {
self.SCHEDULER_BASE_EFFORT: 700,
self.SCHEDULER_PER_REQUEST_EFFORT: 1000,
self.WORKER_CREATE_YIELDVAULT_REQUEST_EFFORT: 5000,
self.WORKER_WITHDRAW_REQUEST_EFFORT: 2000,
self.WORKER_DEPOSIT_REQUEST_EFFORT: 2000,
self.SCHEDULER_BASE_EFFORT: 950,
self.SCHEDULER_PER_REQUEST_EFFORT: 250,
self.WORKER_CREATE_YIELDVAULT_REQUEST_EFFORT: 6500,
self.WORKER_WITHDRAW_REQUEST_EFFORT: 3000,
self.WORKER_DEPOSIT_REQUEST_EFFORT: 1500,
self.WORKER_CLOSE_YIELDVAULT_REQUEST_EFFORT: 5000
}

Expand Down
27 changes: 25 additions & 2 deletions cadence/transactions/scheduler/init_and_schedule.cdc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,12 +97,33 @@ transaction {
}

execute {
let workerCreateEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.WORKER_CREATE_YIELDVAULT_REQUEST_EFFORT
]!
let workerWithdrawEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.WORKER_WITHDRAW_REQUEST_EFFORT
]!
let workerDepositEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.WORKER_DEPOSIT_REQUEST_EFFORT
]!
let workerCloseEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.WORKER_CLOSE_YIELDVAULT_REQUEST_EFFORT
]!

// Make sure WorkerHandler is registered in the manager
if self.manager.getHandlerTypeIdentifiers()[self.workerHandlerTypeIdentifier] == nil {
// Schedule dummy (data=nil) WorkerHandler transaction to register the WorkerHandler in the manager
let workerHandlerPriority = FlowTransactionScheduler.Priority.Medium
let workerHandlerExecutionEffort = 5000 as UInt64
var workerHandlerExecutionEffort = workerCreateEffort
if workerWithdrawEffort > workerHandlerExecutionEffort {
workerHandlerExecutionEffort = workerWithdrawEffort
}
if workerDepositEffort > workerHandlerExecutionEffort {
workerHandlerExecutionEffort = workerDepositEffort
}
if workerCloseEffort > workerHandlerExecutionEffort {
workerHandlerExecutionEffort = workerCloseEffort
}
let transactionId = _scheduleTransaction(
manager: self.manager,
handlerCap: self.workerHandlerCap,
Expand Down Expand Up @@ -130,7 +151,9 @@ transaction {
// Schedule scheduler
if !schedulerRunning {
let schedulerPriority = FlowTransactionScheduler.Priority.Medium
let schedulerExecutionEffort = 700 as UInt64
let schedulerExecutionEffort = FlowYieldVaultsEVMWorkerOps.executionEffortConstants[
FlowYieldVaultsEVMWorkerOps.SCHEDULER_BASE_EFFORT
]!
// First scheduler run will be scheduled without any requests to preprocess
// If there are pending requests, they will be preprocessed in the next scheduler execution
let schedulerTransactionId = _scheduleTransaction(
Expand Down
Loading