Large submission ZIPs (>1GB) consistently fail silently or hang in "Running" status during evaluation

[QHua-Lin]

Hi CodaBench team,

We’re currently running a challenge on the platform (NTIRE 2026 Efficient LLIE, https://www.codabench.org/competitions/13382) and have observed a recurring issue during the final test phase:

• Submissions with result ZIP files around 500–600 MB typically complete within ~15–20 minutes, produce valid scores, and show full logs.

• However, submissions with ZIP sizes exceeding ~1 GB almost always either:

  Remain stuck in "Running" status indefinitely, or

  Eventually return a score of 0.0 with no evaluation logs—suggesting failure during early stages (e.g., unpacking or I/O), before the scoring script even starts.

Our scoring script itself is lightweight and finishes quickly when given valid inputs. The large ZIPs contain only PNG images (no code or extra files), but due to the nature of the task (image enhancement on a large test set), file size can grow significantly depending on bit depth and channel format.

We’re unsure whether this is due to:

• A hard limit on ZIP size or extracted directory size,

• Timeout during extraction/unpacking,

• Disk quota limits in the worker environment,

• Or another resource constraint.

Could you clarify if there are known limits on submission size or extracted content volume? And if so, what are the recommended thresholds to avoid silent failures?

Thank you very much for your support—and for maintaining this valuable platform!

[Didayolo]

Hello,

Thank you for reporting your problem. Indeed we currently have instabilities where the submissions get stuck.
You can track the problem in the following issues:

• Continuously stuck in the "Running" state #2177
• Submission failed: "scoring_hostname-worker9" #2206

The following PR will be deployed soon and should solve the problem:

• Worker status to FAILED instead of SCORING or FINISHED in case of failure #2030
• Compute worker - Improve status update and logs #2223

[LorVC]

Hello, this also happened to me yesterday, but for even smaller files, The system goes into timeout.

[Didayolo]

Hi @LorVC and @QHua-Lin,

In order to progress on this problem quickly, we set up a new queue Codabench-Test with the last version of the compute worker (#2223).

The queue is public, so you can use it on your own competitions (or ask the organizers to do so):

Please report if it helps or if you are facing any problem with it. It should improve the process and the logs, but is still under development.

[LorVC]

hello, on updated instances i get child task failed or non zero return code

[Didayolo]

hello, on updated instances i get child task failed or non zero return code

Can you please report the behavior with more details? Competition URL, submission ID, and logs returned. Thank you.

---

EDIT: I think I could find the logs in question:

[LorVC]

Hello, sorry I deleted the submissions, however if they are still on the server, they must be the latest ones.
The competition is at https://www.codabench.org/competitions/10050/#/participate-tab

The only I haven't deleted is one with id #607155.

I believe there is a problem with the size of the submission files, I'm trying combining files from different submissions with the faulty one, and sometimes it works, so it shouldn't be a formatting problem