Skip to content

feat: Implement work item filters#168

Open
YunchuWang wants to merge 14 commits intomainfrom
feature/work-item-filters
Open

feat: Implement work item filters#168
YunchuWang wants to merge 14 commits intomainfrom
feature/work-item-filters

Conversation

@YunchuWang
Copy link
Member

Summary

Implements the work item filter feature in the JavaScript/TypeScript SDK to match the .NET SDK's DurableTaskWorkerWorkItemFilters (introduced in v1.21.0, PR #616).

Work item filters allow workers to tell the sidecar which orchestrations, activities, and entities they are configured to handle. The sidecar then only dispatches matching work items to that worker, enabling more efficient routing in multi-worker deployments.

Behavior

Configuration Behavior
Default (workItemFilters not set) Filters are auto-generated from the worker's registered orchestrations, activities, and entities
Explicit filters (workItemFilters: {...}) User-provided filters override auto-generation
Disabled (workItemFilters: null) No filters sent — worker receives all work items

Versioning Integration

When VersionMatchStrategy.Strict is configured, the auto-generated filters include the worker's version string. Other strategies (None, CurrentOrOlder) produce empty version arrays (match all versions).

Entity filters never include versions (matching .NET SDK behavior).

Changes

Proto Layer

  • Updated orchestrator_service.proto with WorkItemFilters, OrchestrationFilter, ActivityFilter, EntityFilter message types
  • Added workItemFilters field (field 11) to GetWorkItemsRequest
  • Regenerated gRPC stubs from updated proto definitions

Core SDK (@microsoft/durabletask-js)

  • New file: work-item-filters.ts — TypeScript interfaces (WorkItemFilters, OrchestrationWorkItemFilter, ActivityWorkItemFilter, EntityWorkItemFilter), generateWorkItemFiltersFromRegistry() for auto-generation, and toGrpcWorkItemFilters() for protobuf conversion
  • Registry: Added getOrchestratorNames(), getActivityNames(), getEntityNames() methods
  • Worker: Added workItemFilters option to TaskHubGrpcWorkerOptions; filters are built and attached to GetWorkItemsRequest in internalRunWorker()
  • Exports: All new types exported from package index

Azure Managed Package (@microsoft/durabletask-js-azuremanaged)

  • Added useWorkItemFilters() builder method to DurableTaskAzureManagedWorkerBuilder
  • Passes filters through to TaskHubGrpcWorker constructor

Testing

36 new unit tests covering:

  • toGrpcWorkItemFilters — empty filters, single/multiple orchestration/activity/entity filters, version handling, mixed filters
  • generateWorkItemFiltersFromRegistry — empty registry, auto-detection of names, versioning strategy integration (Strict/None/CurrentOrOlder), named registrations, entity name lowercasing, version array isolation
  • TaskHubGrpcWorker — accepts explicit filters, null filters, undefined (auto-generate), combined with versioning
  • Registry name getter methods
  • End-to-end: registry → filter generation → gRPC protobuf conversion

All 959 existing tests + 36 new tests pass (60 test suites, 0 failures).

Reference

Add work item filters to the GetWorkItemsRequest, matching the .NET SDK's
DurableTaskWorkerWorkItemFilters feature. This allows workers to tell the
sidecar which orchestrations, activities, and entities they can handle,
enabling more efficient work item routing.

Changes:
- Update orchestrator_service.proto with WorkItemFilters, OrchestrationFilter,
  ActivityFilter, and EntityFilter message types (field 11 on GetWorkItemsRequest)
- Regenerate gRPC stubs from updated proto definitions
- Add WorkItemFilters TypeScript interfaces and conversion functions
- Add Registry.getOrchestratorNames/getActivityNames/getEntityNames methods
- Auto-generate filters from registry in TaskHubGrpcWorker.internalRunWorker
- Support explicit filters via workItemFilters option on TaskHubGrpcWorkerOptions
- Support null to explicitly disable filtering (receive all work items)
- Add useWorkItemFilters() builder method to DurableTaskAzureManagedWorkerBuilder
- Export new types from package index
- Add 36 unit tests covering conversion, generation, versioning integration,
  and end-to-end registry-to-gRPC flows
Copilot AI review requested due to automatic review settings March 12, 2026 21:54
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds work item filtering support to the DurableTask JS worker so a worker can ask the sidecar to only stream orchestrations/activities/entities it can handle (optionally including version constraints), and exposes helpers to generate/convert these filters.

Changes:

  • Introduces WorkItemFilters types plus helpers to generate filters from the Registry and convert them into protobuf messages.
  • Extends TaskHubGrpcWorker (and the Azure Managed worker builder) with a workItemFilters option (undefined = auto-generate, null = disable filtering).
  • Updates the protocol (GetWorkItemsRequest) and generated protobuf JS/TS stubs to include WorkItemFilters, and adds comprehensive unit tests for filter generation/conversion and registry name getters.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
packages/durabletask-js/test/work-item-filters.spec.ts New unit tests for filter conversion/generation, worker option wiring, and registry name getters.
packages/durabletask-js/src/worker/work-item-filters.ts New SDK-level filter model + registry→filters generator + filters→protobuf converter.
packages/durabletask-js/src/worker/task-hub-grpc-worker.ts Sends workItemFilters on GetWorkItemsRequest (auto-generated by default; null disables).
packages/durabletask-js/src/worker/registry.ts Adds name getter APIs used for filter auto-generation.
packages/durabletask-js/src/proto/orchestrator_service_pb.js Regenerated protobuf JS: adds WorkItemFilters and related messages + request field.
packages/durabletask-js/src/proto/orchestrator_service_pb.d.ts Regenerated protobuf TS types for WorkItemFilters and new request field.
packages/durabletask-js/src/index.ts Exports the new filter APIs from the public package surface.
packages/durabletask-js-azuremanaged/src/worker-builder.ts Adds useWorkItemFilters() builder method and forwards filters to core worker.
internal/protocol/protos/orchestrator_service.proto Adds WorkItemFilters messages and workItemFilters field to GetWorkItemsRequest.

You can also share your feedback on Copilot code review. Take the survey.

- Change orchestrator test helpers to direct async generator functions
  instead of factory functions returning generators (Copilot review)
- Extract _buildGetWorkItemsRequest() from internalRunWorker() for
  testability and add 4 unit tests verifying filter request building:
  auto-generate, null (disabled), explicit filters, and Strict versioning
  (Copilot review)
- Add 4 e2e tests in test/e2e-azuremanaged/work-item-filters.spec.ts covering:
  auto-generated filters, explicit filters, and disabled filters (null)
- Add examples/work-item-filters/ demonstrating all three filter modes
- All tests verified passing against DTS emulator (localhost:8080)
@YunchuWang
Copy link
Member Author

E2E Test & Example Results (DTS Emulator)

All tests and examples ran successfully against the DTS emulator (localhost:8080).

E2E Test Results (test/e2e-azuremanaged/work-item-filters.spec.ts)

 PASS  test/e2e-azuremanaged/work-item-filters.spec.ts (43.68 s)
  Work Item Filters E2E Tests
    Auto-generated filters (default behavior)
      √ should process orchestrations with auto-generated filters from registry (8726 ms)
      √ should process orchestrations with activities using auto-generated filters (9966 ms)
    Explicit filters
      √ should process orchestrations with explicit work item filters (9995 ms)
    Disabled filters (null)
      √ should process all orchestrations when filters are explicitly disabled (10012 ms)

Test Suites: 1 passed, 1 total
Tests:       4 passed, 4 total
Snapshots:   0 total
Time:        43.99 s

Example Output (examples/work-item-filters/index.ts)

Connecting to DTS emulator at localhost:8080, taskHub: default

=== Scenario 1: Auto-Generated Filters (Default) ===
Worker started with auto-generated filters for: greetingOrchestrator, greet
Scheduled orchestration: 7bbda1e1-9fa2-4b23-8b99-f9173232e6dd
Result: "Hello, Auto-Filters!"

=== Scenario 2: Explicit Filters ===
Worker started with explicit filters for: mathOrchestrator, add
Scheduled orchestration: 2ef05470-179a-4981-b2b5-168218dcaf17
Result: 42

=== Scenario 3: Disabled Filters (null) ===
Worker started with filters DISABLED (receives all work items)
Scheduled orchestration: b5282d9e-14b2-448b-bc17-fdde3f5bbee4
Result: "Hello, No-Filters!"

=== All scenarios completed successfully! ===

The explicit filter test was functionally identical to the auto-generated
filter test since it provided the same names that auto-generation would
produce.
@YunchuWang YunchuWang changed the title feat: Implement work item filters for GetWorkItems RPC (parity with .NET SDK) feat: Implement work item filters for GetWorkItems RPC Mar 12, 2026
@YunchuWang YunchuWang changed the title feat: Implement work item filters for GetWorkItems RPC feat: Implement work item filters Mar 12, 2026
- Add test for auto-generated filters with Strict versioning strategy
- Add test for explicit versioned filters via useWorkItemFilters()
- Both tests verified passing against DTS emulator and real DTS scheduler
- Unskip 'should not dispatch orchestration that is not in the filter' test
  (server-side filter enforcement now works on DTS scheduler)
- Add 'should not dispatch orchestration when name matches but version does
  not' test verifying versioned filter enforcement
- All 6 e2e tests verified passing against DTS scheduler (East Asia)
- Example verified passing against DTS scheduler
@YunchuWang
Copy link
Member Author

Updated E2E Test Results (DTS Scheduler — East Asia)

All tests and example verified against real DTS scheduler (wbtestfilter in East Asia) with server-side work item filter enforcement.

E2E Test Results (test/e2e-azuremanaged/work-item-filters.spec.ts)

PASS test/e2e-azuremanaged/work-item-filters.spec.ts (46.804 s)
  Work Item Filters E2E Tests
    Auto-generated filters (default behavior)
      √ should process orchestrations with auto-generated filters from registry
      √ should process orchestrations with activities using auto-generated filters
    Disabled filters (null)
      √ should process all orchestrations when filters are explicitly disabled
    Filters with versions
      √ should process orchestration with explicit versioned filters
    Filtered-out work items
      √ should not dispatch orchestration that is not in the filter
      √ should not dispatch orchestration when name matches but version does not

Test Suites: 1 passed, 1 total
Tests:       6 passed, 6 total

Key findings:

  • Server-side filter enforcement is working on the East Asia DTS scheduler — unmatched orchestrations correctly stay in PENDING status
  • Version-based filtering also works — an orchestration with matching name but wrong version stays PENDING
  • The DTS emulator does NOT yet enforce filters (dispatches all work items regardless)

Example Output (examples/work-item-filters/index.ts)

=== Scenario 1: Auto-Generated Filters (Default) ===
Worker started with auto-generated filters for: greetingOrchestrator, greet
Result: "Hello, Auto-Filters!"

=== Scenario 2: Explicit Filters ===
Worker started with explicit filters for: mathOrchestrator, add
Result: 42

=== Scenario 3: Disabled Filters (null) ===
Worker started with filters DISABLED (receives all work items)
Result: "Hello, No-Filters!"

=== All scenarios completed successfully! ===

- Register both orchA and orchB but only include orchA in the explicit filter
- Schedule orchB and verify it stays PENDING (not dispatched to worker)
- Verified passing against DTS scheduler (East Asia)
Remove two tests that overlap with existing orchestration e2e tests
(auto-generated filters are now the default behavior for all workers).
Remaining 5 tests focus on filter-specific scenarios: disabled filters,
explicit versioned filters, and filter enforcement (name/version mismatch,
registered-but-excluded).
Skip the 'Filtered-out work items' test group since the DTS emulator (used
in CI) does not yet enforce server-side work item filters. These tests pass
against a real DTS scheduler but fail against the emulator which dispatches
all work items regardless of filters.
Explicitly lowercase entity names in toGrpcWorkItemFilters() to match .NET
SDK behavior where entity names are normalized to lowercase in the backend.
This ensures explicit user-provided entity filters are also lowercased,
not just auto-generated ones from the registry.
The disabled filters (null) scenario produces the same observable result
as auto-generated filters. Keep only auto-generated and explicit filter
scenarios which demonstrate distinct behaviors.
Update SOURCE_COMMIT to track upstream commit 1caadbd which already
includes the WorkItemFilters proto definitions. Proto content is identical
to our manual addition — confirms alignment with the shared proto repo.
The auto-generated work item filters include the worker's version when
VersionMatchStrategy.Strict is configured. This causes the sidecar to
filter out version-mismatched orchestrations server-side, preventing the
SDK's local FailureStrategy logic from running. Disable filters in
createWorkerWithVersioning() so version mismatches are handled locally.
…t test

The slow activities may not be dispatched/executed before the orchestrator
catches the whenAll failure and completes. Whether they run is purely
timing-dependent, so asserting their counter is inherently flaky.
The test's purpose is to verify the orchestration stays COMPLETED after
fail-fast, which the remaining assertions already cover.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants