feat(cwl): integration of CWL job submission and execution into DiracX by ryuwd · Pull Request #877 · DIRACGrid/diracx

ryuwd · 2026-04-02T09:30:59Z

End-to-end CWL job submission and execution for DiracX — from CLI to worker node and back.

Follows the plan in #858.

Goes with DIRACGrid/DIRAC#8506

CLI (`diracx-cli`)

dirac job submit cwl <workflow> [inputs...] — submit CWL jobs with local file sandbox upload, LFN references, and parametric --range expansion
dirac job submit cmd -- <command> — quick submission with auto-generated CWL (captures stdout/stderr to log files)
dirac job search — search jobs with conditions and rich table output
dirac job sandbox list|peek|get <job_id> — explore and retrieve output sandbox files
Submission pipeline: CWL/YAML parsing, input validation, sandbox scanning/grouping/upload, confirmation prompt, range expansion
CWL executor (dirac-cwl-run): custom cwltool executor with DIRAC-aware FsAccess and PathMapper for LFN resolution via replica maps

Worker Node (`diracx-api`)

JobWrapper: full CWL job lifecycle — pre-process (sandbox download, LFN resolution, replica map building), async subprocess execution with live stderr streaming, post-process (output sandbox upload, output data registration)
ApplicationStatus reporting: cwltool lifecycle transitions (e.g. [job echo-tool] completed success, [workflow ] starting step greet) streamed as ApplicationStatus with rate-limited commits
Sandbox handling: SB:<se>|<s3_path>#<filename> URI scheme — #fragment identifies the file inside the tar archive; sandbox reference preserved with SB: prefix throughout the system
JobReport: accumulates status updates, flushes via HTTP with rate limiting

Server (`diracx-logic`, `diracx-routers`, `diracx-core`)

CWL-to-JDL translation: cwl_to_jdl() extracts dirac:Job hints from CWL, maps to JDL fields (CPUTime, Sites, Tags, I/O sandboxes, InputData, OutputData)
Auto stdout/stderr collection: CWL stdout:/stderr: fields automatically added to OutputSandbox in JDL
InputSandbox #fragment stripping: JDL InputSandbox contains bare SB: refs (no fragment) for server ownership checks; full URI with #filename preserved in CWL inputs for worker extraction
Range expansion: server-side parametric job expansion from --range spec
Models: JobHint, IOSource, OutputDataEntry, ReplicaMap, pre/post-process command framework

Client (`diracx-client`)

Generated client extensions for CWL submission, workflow retrieval, and sandbox operations

Key design decisions

CWL-native: no JDL on the client side — CWL is the job description format, JDL is an internal detail
cwltool passthrough for status: ApplicationStatus shows verbatim cwltool lifecycle lines rather than a custom translation layer
SB: URI scheme: SB:<se>|<s3_path>#<relative_path> — logical reference (not PFN), server resolves to presigned URL
Replica map: JSON file passed to cwltool executor, maps LFN/SB paths to local files — decouples CWL execution from DIRAC data management

Test coverage

Unit tests: JobWrapper commands, output parsing, CWL hint extraction, sandbox path parsing, replica map injection, submission pipeline, input parsing, executor path mapping
Integration tests: full JobWrapper lifecycle with mocked services, real CWL execution, stderr streaming, ApplicationStatus filtering

Status

Under certification testing on diracx-cert.app.cern.ch. Actively fixing issues found during grid execution.

cc @aldbr

read-the-docs-community · 2026-04-02T11:35:25Z

Documentation build overview

📚 diracx | 🛠️ Build #32203433 | 📁 Comparing eeb15b1 against latest (63dc086)

🔍 Preview build

No files changed.

Introduce the `POST /api/jobs/` endpoint for submitting CWL workflows with per-job input parameters, and `GET /api/workflows/{workflow_id}` for retrieving stored workflows. CWL definitions are stored once in a new `Workflows` table (content-addressed by SHA-256), with per-job parameters and workflow references added as new columns on `Jobs`. During the transition period, JDL is still produced via `cwl_to_jdl()` for compatibility with existing DIRAC infrastructure.

Pydantic now validates schema_version at model construction instead of a runtime check in the logic layer. Users can omit schema_version to get the default "1.0", but invalid versions are rejected immediately.

Parse YAML to dict and serialize as sorted JSON before SHA-256 hashing, so whitespace, comments, and key ordering differences produce the same workflow ID. The original YAML is still stored as-is for readability.

Replace UploadFile multipart with a CWLJobSubmission pydantic model to fix autorest client generation. Add job_wrapper_template.py to diracx-logic for worker-side CWL execution via importlib.resources.

chore: regenerate client for gubbins

Move JobWrapper, JobReport, commands, and submission models into diracx-api and diracx-core so DIRAC does not need dirac-cwl installed.

fix: again

…eferences

…mines layout

Rename _resolve_lfn to _resolve_path and update all methods (_abs, glob, open, exists, isfile, isdir, size) to handle both LFN: and SB: prefixed paths. SB: keys are stored with prefix in the replica map while LFN keys are stored without prefix.

Parse SB: prefixed paths in input sandbox downloads, cache per PFN, and inject sandbox entries into the replica map for CWL executor resolution.

…mmandLineTool 28 tests covering LFN extraction, replica map filtering/merging, path resolution in DiracPathMapper, and tool factory routing in DiracCommandLineTool. Loads executor modules by file path to bypass __init__.py's mypyc compat hook.

…plica map wiring

…lica map - Create test_executor_integration.py with three subprocess tests: test_basic_execution_with_replica_map (LFN resolved via replica map), test_execution_without_replica_map (plain local file), and test_sb_reference_in_replica_map (SB: key resolved via replica map) - Fix DiracPathMapper to convert file:// PFNs to local paths for MapperEnt.target so cwltool passes filesystem paths (not file:// URLs) to subcommands - Extend DiracPathMapper.visit to handle SB: URI scheme alongside LFN: - Add DiracPathMapper.mapper() override so SB: keys with '#' fragments are found directly in _pathmap without cwltool's fragment-stripping logic

Skip non-SB: paths in input_sandbox and non-LFN: paths in input_data with a warning instead of silently handling them. This JobWrapper only supports CWL jobs where sandboxes are always SB: prefixed and input data is always LFN: prefixed.

Add integration test exercising the complete run_job() lifecycle: hint parsing, sandbox download, LFN download, replica map building, SB injection, real dirac-cwl-run execution, and output parsing. Also fix dirac-cwl-run logging to use stderr so that only the cwltool JSON output is written to stdout, allowing job_wrapper.py to reliably parse it with json.loads(). test(api): add JobWrapper integration test for full CWL execution chain Add integration test exercising the complete run_job() lifecycle: hint parsing, sandbox download, LFN download, replica map building, SB injection, real dirac-cwl-run execution, and output parsing. Add conftest.py to pre-load real cwl_utils/ruamel.yaml before other test files mock them, ensuring the integration test can access real types. Also fix dirac-cwl-run logging to use stderr so that only the cwltool JSON output is written to stdout, allowing job_wrapper.py to reliably parse it with json.loads().

All dependencies (cwltool, cwl_utils, ruamel.yaml, DIRACCommon, diracx.client.aio, diracx.api.job_report, diracx.api.jobs) are now installed in the test environment, so replace the sys.modules mock scaffolding and importlib file-loading with direct imports. Delete conftest.py which existed solely to save real module references before mocking occurred.

The executor is a CLI tool (dirac-cwl-run entry point), not an API component. Moves source and tests to diracx-cli, updates cwltool dependency location, and rewrites test_no_cwltool_import to verify the mypyc compatibility patch is installed before cwltool loads. chore: update lockfile chore: update .gitignore

fix: cwl-utils

Shows scalar key=value pairs from the input dict in the job name, e.g. "hello-world (seed=42)" so parametric jobs are distinguishable in search results and monitoring.

DIRAC's wrapper script passes only the jobID (not a json-file), so accept 1 or 2 args and always read the jobID from the last argument.

Report meaningful ApplicationStatus using the CWL task label instead of leaving it as "Unknown". Shows e.g. "parametric-echo completed" or "parametric-echo failed (exit 1)" in job search results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace subprocess.run with Popen to read stderr line-by-line, re-emit each line to the wrapper's stderr, and store it as ApplicationStatus with immediate commit for real-time visibility. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…Status Read stdout in a background thread to prevent pipe deadlock when both stdout and stderr use subprocess.PIPE. Add explicit application_status to the failure path so the final status is not just the last cwltool line. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Replace subprocess.Popen + threading with asyncio.create_subprocess_exec for proper async cooperation. Stderr is streamed line-by-line without blocking the event loop, stdout collected concurrently via asyncio task. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add stdout/stderr capture fields to generate_cwl() so cwltool writes the tool's output to stdout.log and stderr.log in the output sandbox, preventing job output from being silently discarded.

Avoids naming collision with DIRAC's conventional std.out filename. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Batch ApplicationStatus commits to at most once per 2 seconds instead of per stderr line, reducing HTTP traffic. All lines are still stored via set_job_status — only the commit frequency changes. Commit failures are logged as warnings instead of killing the job. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Only report step/job/workflow lifecycle lines (start, completed, failed, skipped) as ApplicationStatus. Noise lines (validation, file staging, timing) still go to stderr for Watchdog peek but are not sent to the server. Log level prefixes (INFO/WARNING) are stripped — ApplicationStatus shows clean cwltool output like "[job X] completed success". Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Require INFO/WARNING/ERROR prefix before cwltool lifecycle patterns to prevent false positives from user application output. Use group(1) to extract the bracket-onward portion, stripping the level prefix. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The server checks sandbox ownership using the PFN without the #fragment. The fragment is only needed by the worker to extract the correct file from the tar archive.

Also rename fake_pfn → fake_sb_ref in submission integration tests and fix assertion to expect #filename fragment.

Adds list, peek, and get subcommands: - list: show files and sizes in the output sandbox - peek: display file contents with syntax highlighting - get: download sandbox files to a local directory

The generated client regex has a typo ((:? vs (?:) that rejects the SB: prefix. The server accepts the bare /S3/... path.

Path() mangles the SB: reference. Also filter None values from get_job_sandbox response in CLI.

ryuwd · 2026-04-10T13:50:43Z

diracx-api/src/diracx/api/job_report.py

+            raise RuntimeError(f"Could not set job statuses: {ret}")
+
+    async def commit(self):
+        """Send all the accumulated information."""


This should be debounced or use some rate limiter in the JobReport class itself

This would replace the rate limiting logic in the JobWrapper which would be better here

diracx-api/src/diracx/api/job_wrapper.py

for more information, see https://pre-commit.ci

ryuwd · 2026-04-10T13:56:18Z

diracx-cli/src/diracx/cli/executor/fs_access.py

+    This class extends StdFsAccess to handle LFN: and SB: prefixed paths by
+    looking them up in the replica map and using the physical file path instead.
+
+    Key difference: LFN keys are stored WITHOUT prefix, SB keys are stored WITH prefix.


Should be harmonised

ryuwd · 2026-04-10T14:02:04Z

diracx-core/src/diracx/core/config/schema.py

    Services: ServicesConfig = ServicesConfig()
    """Configuration for various DIRAC services."""
-    SoftwareDistModule: str = "LocalSoftwareDist"
+    SoftwareDistModule: str = ""


Was causing errors in the Pilot

to be checked with @chaen

ryuwd · 2026-04-10T14:03:32Z

diracx-core/src/diracx/core/models/commands/store_output_data.py

+                # TODO: Compute Adler32 checksum before upload
+                # TODO: Extract POOL/ROOT GUID if applicable
+                # TODO: Prefer local SEs (getSEsForSite) before remote ones
+                # TODO: Implement retry with exponential backoff on transient failures
+                # TODO: On complete failure, create a failover Request (RMS)
+                #       for async recovery instead of raising immediately
+                # TODO: Report upload progress via job status updates


This command is still untested in cert. StoreOutputData still needs fuller implementation, discussion, and testing.

ryuwd added 4 commits April 7, 2026 13:22

fix: enforce schema_version as Literal["1.0"] with default in JobHint

5d8ed95

Pydantic now validates schema_version at model construction instead of a runtime check in the logic layer. Users can omit schema_version to get the default "1.0", but invalid versions are rejected immediately.

fix: normalize CWL before hashing to prevent duplicate workflow rows

001e8ff

Parse YAML to dict and serialize as sorted JSON before SHA-256 hashing, so whitespace, comments, and key ordering differences produce the same workflow ID. The original YAML is still stored as-is for readability.

feat: use JSON body for CWL submission and add job wrapper template

13f8c14

Replace UploadFile multipart with a CWLJobSubmission pydantic model to fix autorest client generation. Add job_wrapper_template.py to diracx-logic for worker-side CWL execution via importlib.resources.

ryuwd force-pushed the feat/cwl-job-submission branch from 11d928f to b1362d6 Compare April 7, 2026 11:26

ryuwd added 24 commits April 8, 2026 15:53

chore: regenerate diracx client

93f3215

chore: regenerate client for gubbins

refactor: move job wrapper and commands from dirac-cwl into diracx

13bba67

Move JobWrapper, JobReport, commands, and submission models into diracx-api and diracx-core so DIRAC does not need dirac-cwl installed.

fix: job wrapper uses jobs.search

0de1f46

fix: again

feat: ReplicaMap integration and added dirac cwl executor

f649f63

refactor: cleaned executor significantly

69bb5ae

feat(core): extend replica map key validation to accept SB: sandbox r…

ddadf76

…eferences

refactor(core): remove path field from IOSource — tar structure deter…

e4416cf

…mines layout

feat(api): add SB: path parsing and sandbox replica map injection

bdad870

Parse SB: prefixed paths in input sandbox downloads, cache per PFN, and inject sandbox entries into the replica map for CWL executor resolution.

feat(api): add cwltool as explicit dependency of diracx-api

b0ea2d5

test(api): extend FsAccess tests with LFN resolution coverage

2c46728

test(api): add unit tests for JobWrapper commands, output parsing, re…

f1ce58d

…plica map wiring

fix: depend on cwl-utils

25c45be

fix: cwl-utils

chore: update lockfile

d6c3341

feat: add CWL input parsing module (YAML, JSON, CLI args, range)

1db8f7b

feat: add sandbox scanning, grouping, and path rewriting

bd85cab

test: add missing SB passthrough test for rewrite_sandbox_refs

fc6b398

feat: add submission confirmation prompt

6c7f8cd

ryuwd and others added 21 commits April 9, 2026 18:26

feat: append input parameters to job name for parametric jobs

8c19b4d

Shows scalar key=value pairs from the input dict in the job name, e.g. "hello-world (seed=42)" so parametric jobs are distinguishable in search results and monitoring.

fix: accept jobID as last argument in job wrapper

483ceb7

DIRAC's wrapper script passes only the jobID (not a json-file), so accept 1 or 2 args and always read the jobID from the last argument.

feat(cli): capture stdout/stderr to log files in generated CWL

76acbf8

Add stdout/stderr capture fields to generate_cwl() so cwltool writes the tool's output to stdout.log and stderr.log in the output sandbox, preventing job output from being silently discarded.

fix: use stdout.log instead of std.out in integration test

6f13421

Avoids naming collision with DIRAC's conventional std.out filename. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(cli): sandbox submit

36fbae1

fix: strip #fragment from InputSandbox JDL field

582ac77

The server checks sandbox ownership using the PFN without the #fragment. The fragment is only needed by the worker to extract the correct file from the tar archive.

fix: no temp files

85eea01

fix: some things need uploading regardless

0beb974

refactor: code around job wrapper sandbox handling

1f35e11

fix: resolve relative path before as_uri() in sandbox replica map

b3cc06a

Also rename fake_pfn → fake_sb_ref in submission integration tests and fix assertion to expect #filename fragment.

feat(cli): add dirac job sandbox commands for output sandbox exploration

de7349e

Adds list, peek, and get subcommands: - list: show files and sizes in the output sandbox - peek: display file contents with syntax highlighting - get: download sandbox files to a local directory

fix(cli): use SandboxType enum instead of string for get_job_sandbox

82fdbfb

fix(cli): strip SB:SE| prefix before get_sandbox_file client call

caf9f89

The generated client regex has a typo ((:? vs (?:) that rejects the SB: prefix. The server accepts the bare /S3/... path.

fix: output sandbox assignment — don't wrap SB: ref in Path()

e1ecb19

Path() mangles the SB: reference. Also filter None values from get_job_sandbox response in CLI.

ryuwd changed the title ~~feat(cwl): add CWL workflow submission endpoint and DB storage model~~ feat(cwl): integration of CWL job submission and execution into DiracX Apr 10, 2026

ryuwd mentioned this pull request Apr 10, 2026

feat(cwl): CWL job wrapper DIRACGrid/DIRAC#8506

Draft

ryuwd commented Apr 10, 2026

View reviewed changes

diracx-api/src/diracx/api/job_wrapper.py Outdated Show resolved Hide resolved

ryuwd and others added 2 commits April 10, 2026 15:52

fix: remove unnecessary doc

933c6dc

[pre-commit.ci] auto fixes from pre-commit.com hooks

eeb15b1

for more information, see https://pre-commit.ci

ryuwd commented Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(cwl): integration of CWL job submission and execution into DiracX#877

feat(cwl): integration of CWL job submission and execution into DiracX#877
ryuwd wants to merge 67 commits intoDIRACGrid:mainfrom
ryuwd:feat/cwl-job-submission

ryuwd commented Apr 2, 2026 •

edited

Loading

Uh oh!

read-the-docs-community bot commented Apr 2, 2026 •

edited

Loading

Uh oh!

ryuwd Apr 10, 2026

Uh oh!

ryuwd Apr 10, 2026

Uh oh!

Uh oh!

ryuwd Apr 10, 2026

Uh oh!

ryuwd Apr 10, 2026

Uh oh!

ryuwd Apr 10, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

ryuwd commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI (diracx-cli)

Worker Node (diracx-api)

Server (diracx-logic, diracx-routers, diracx-core)

Client (diracx-client)

Key design decisions

Test coverage

Status

Uh oh!

read-the-docs-community bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation build overview

Uh oh!

ryuwd Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

ryuwd Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ryuwd Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

ryuwd Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

ryuwd Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

ryuwd commented Apr 2, 2026 •

edited

Loading

CLI (`diracx-cli`)

Worker Node (`diracx-api`)

Server (`diracx-logic`, `diracx-routers`, `diracx-core`)

Client (`diracx-client`)

read-the-docs-community bot commented Apr 2, 2026 •

edited

Loading

ryuwd Apr 10, 2026 •

edited

Loading