[Change]: better queing for jobs

[thsm-kb]

Browsertrix Host

Self-Hosted

What change would you like to see?

I would like to create jobs in a state between "run now" and "scheduled". It could be called "capacity-aware". I would like to be able to create them so that they run, if the free capacity is above a certain treshold set in settings.
If I have a capacity of 50 jobs and have set the treshold to keep 10 free for scheduled and "run now" jobs, then the other 40 could be used to run "capacity-aware" jobs. This way I could queue thousands of "capacity-aware" jobs, without conflicting with "run now" or "scheduled".

Feel free to ask if anything is unclear.
This would be a significant improvement for anyone using Browsertrix in larger scale.

Additional details

No response

[ikreymer]

Hm, it's an interesting idea.. do the 'capacity aware' jobs restart immediately automatically after they're done, or do they just run once?

You can sort of do this now with two orgs + concurrent crawl limits. With the fix #2945, you should be able to queue lots of jobs fairly efficiently, I think. You can:

• Set one org to have a quota of 40 jobs at a time. Set all jobs in this org to run on a daily schedule (or custom) schedule.
• Set another org to have a quota of 10 jobs at a time. These can be run as needed.

I think with this setup, the 40 jobs will always be running, and you can queue as many as you want (crawl_queue_limit_scale can be used to set a limit to how many jobs are allowed to be queued as a multiplier on concurrency, otherwise no limit)
And you can start upto 10 jobs in the other org at the same time as well.

I guess you probably want these to be within the same org, right? Then it's probably similar to having different priorities for jobs. But the two orgs will give you the ability to have two sets of queues already, if you want to try that..

[ikreymer]

If you want the 'capacity aware' jobs to run continuously, eg. start a new crawl as soon as previous one is finished, I think may just need to add sufficiently large startingDeadlineSeconds to the CronJob and a small schedule. That way, it'll restart as soon as the previous one is finished, no matter how long it takes.. I think that should work with the above.

[thsm-kb]

Yes I would like it in the same org if possible. Part of the goal is to run at close to 100% utilization. Part of the goal is to never be out of ressources for scheduled and important adhoc jobs.
My idea was 'capacity aware'-jobs should just run once. I can always submit them to the queue again later. But the queue would be large in my situation - broadcrawl large.