[Feature]: Disable rescheduling in DiracX

[aldbr]

User Story

As an operator and as a developer, it can be very complex to follow rescheduled jobs properly and we sometimes see some discrepancies. We decided not to implement the rescheduling feature within DiracX as it would be much easier to manage.

Feature Description

Nevertheless, before making this decision, we already started to implement a reschedule method in diracx-routers. This one should be removed from the code.

Then, one should add an a CS option in DIRAC to enable/disable rescheduling.
In each function where the rescheduling is called, one should check the value and return S_ERROR if disabled.

Definition of Done

• Option to disable rescheduling in DIRAC
• no mention of rescheduling within DiracX
• Unit tests in DIRAC
• Documentation

Alternatives Considered

No response

Related Issues

No response

Additional Context

No response

[aldbr]

Should we add a flag at all?

• We could say from a given version there is no more rescheduling in DIRAC.
• Probably some unknown so adding a flag is probably safer (stress on Transformation System?)

Should we disable rescheduling? Normal user jobs can suffer from system deficiency, not user fault

Can probably be discussed during a Dops meeting to see with other VOs (need their inputs)

[chrisburr]

@atsareg For the user job side of things I think we need a description of what the actual requirement is. Within LHCb user jobs rely on retries in a similar idea to the transformation system (i.e. we resubmit rather than rescheduling). I don't see why this wouldn't work for other VOs, if it's a problem of bookkeeping jobs there could be a job parameter to trace where jobs came from. (I'm not fully convinced it's worth it so that needs some discussion about why.)

[andresailer]

we resubmit rather than rescheduling

For user jobs, is this "Resubmit" also just a button in the webapp that "clones" the selected job and creates a new one with all the same parameters?

[chrisburr]

For user jobs, is this "Resubmit" also just a button in the webapp that "clones" the selected job and creates a new one with all the same parameters?

Yes exactly. From a DIRAC perspective I think it would be nicer to:

• Have a "Resubmitted" state in the state machine which cannot be resubmitted again
• Add a job parameter with the ID of the new job that was created

This would then let you be able to follow a chain of resubmitted jobs from the original ID while avoiding the issues that come with rescheduling.

[fstagni]

Summary points from discussion happened at https://indico.cern.ch/event/1640518/

• In general, the removal of rescheduling is not opposed, but many observation and requirements seem to (at a minimum) slow down on the proposal.
• Jobs can be rescheduled by users (via dirac-wms-job-reschedule, or the WebApp button), or automatically in case of errors, e.g. when the JobWrapper is not capable of resolving the input data. This difference is felt important.
  • Users do reschedule jobs by hand, but this could create problems later on. @atsareg proposes to remove the possibility to reschedule jobs by hand.
• Users, but also external tools (Ganga, or the Belle2 "scout jobs", and maybe others) are relying on the initial list of jobIDs coming out of the initial jobs submissions. The channel is closed after submission, so, while not impossible, the list of such jobIDs could not be easily updated later on in case of re-submission.
  • A "Resubmitted" state does not feel very natural, nor a chain of jobs coming out from the initial job.
• Keeping full access to all errors of rescheduled jobs is felt important.

Proposals inspired by the discussion:

• Do not fully eliminate the concept of job rescheduling, but limit it. For example:
• Do not reschedule Transformation Jobs (set jobs as FAILED instead), let the TS re-submit them.
• Do not allow users to reschedule their own jobs.
• Replace dirac-wms-job-reschedule and the WebApp button with job resubmit action.

[zhangxiaomei]

Since it is job resubmit instead of job rescheduling, is it possible to redefine sites for these failed jobs? It could be useful in the case the jobs with memory problem could survive in some specific sites with large memory nodes.

[fstagni]

Since it is job resubmit instead of job rescheduling, is it possible to redefine sites for these failed jobs? It could be useful in the case the jobs with memory problem could survive in some specific sites with large memory nodes.

That could also be implemented, if felt necessary.