[Feature]: Job grouping for HPCs with no external connectivity

[AcquaDiGiorgio]

User Story

As a representative of the LHCb DIRAC community at Barcelona, we need some kind of system that would let us group single core jobs, so that when it reaches our worker nodes they can be executed parallelly using as many cores as possible.

Feature Description

Currently we are underutilizing our resources, as it is only executing single core jobs in worker nodes with 112 cores. This is harming performance greatly, as we could theoretically reach an 112x improvement.

This issue is due to the fact that these worker nodes do not have external connectivity so we are unable to accept pilot jobs. On top of that, the main simulation program being executed is Gauss, which only uses a singular core for its simulations.

This problem has existed for a while, but we need a solution now more than ever, and with the improvements done to the PushJobAgent, it is the perfect moment to have this.

The idea would be te create some kind of intermediary CE that sits between the PushJobAgent and an AREX CE that receives single core jobs.
Those single core jobs should be grouped or bundled in a singular multiprocessor job that gets sent to the AREX as just 1 job.
Finally the worker node should divide those single core jobs and execute them all at once, utilizing all of the nodes if possible.

Must be easy to configure for administrators and also transparent for the user.

Definition of Done

• Create a system capable of grouping multiple jobs as a singular one that then gets send to the worker node, split and executed.
• Ensure that the system is robust against an AREX CE.
• It has to work when the jobs get submitted through a PushJobAgent.

Alternatives Considered

Modifying the Matcher so it matches multiple jobs at a time instead of one by one could also work.

Related Issues

No response

Additional Context

No response

[aldbr]

Thanks @AcquaDiGiorgio for the detailed work on this. The problem running single-core jobs on 112-core nodes with no external connectivity is real. Let's discuss about the best way of solving this issue.

Understanding the architectural constraint

I can see why a simple "just submit multi-core jobs" answer doesn't work here, and why the PR ended up with a DB + Service architecture. The root cause is how the PushJobAgent currently operates:

match one job
preprocess (with user proxy)
create JobWrapper
ce.submitJob()
...
repeat

Each job is preprocessed and submitted individually and immediately. There's no accumulation point where N jobs can be gathered before a single multi-core submission. The BundleCE + BundleDB + BundlerService essentially creates a stateful accumulator that sits between the PushJobAgent and the real CE, holding jobs until enough arrive to fill a node.

I understand the reasoning, but I believe this introduces disproportionate complexity for what it solves, and more importantly, I think this problem should be tackled at the VO level rather than in DIRAC's core.

Why this should be an LHCb-level solution, not a DIRAC-level one

This is a very specific use case: one site (Barcelona), one VO (LHCb), one application (Gauss), one constraint (no external connectivity on 112-core nodes). Building 4 new DIRAC components (~1,900 lines), a new MySQL database, and a new DIRAC service into the core framework for this is a significant maintenance commitment that affects the entire project.

A "simpler" implementation could consist in preparing LHCbDIRAC jobs with 1 Gauss task per slot, and a mechanism to group the outputs at the end of the execution. They would be submitted as is to the WMS.
The LHCbDIRAC transformation system could potentially submit multi-core "jobs" that internally spawn N Gauss instances. This is clearly not an easy change, but it's potentially cleaner than a WMS architecture change.

Concerns with the current PR approach

Even setting aside the "where should this live" question, the implementation has significant technical concerns:

The straggler problem

The bash wrapper runs all jobs as & background processes and calls wait. The batch allocation lives until the slowest job finishes. With 112 jobs, if 111 finish in 30 minutes and one takes 12 hours, ~1,276 core-hours are wasted.

Operational complexity

• Proxy locality: the service and PushJobAgent must run on the same machine because proxy file paths are stored in the DB.
• State management: bundle lifecycle (Waiting - Running - Done/Failed) with flags (Cleaned, Purged) across a DB, service, and agent is effectively a mini-WMS within the WMS.
• Failure surface: if the BundlerService crashes mid-bundle, or the DB becomes inconsistent, recovery is non-trivial.
• The BundleManagerAgent exists solely to handle stalled bundles that never fill up.

Recommendation

I would suggest solving this at the LHCbDIRAC production level:

• improving the structure of the LHCb jobs to leverage the new PushJobAgent first (because we still cannot rely on it)
• if Gaussino is still not ready by this time, then considering the possibility of submitting multi-core meta-jobs that internally run N Gauss instances.