Wrong leaderboard ordering and `N/a` scores discussion

[ihsaan-ullah]

1. Wrong leaderboard ordering

Sometimes the ranking on the leaderboard does not follow the order from the primary column. It happens several times and we can hardly reproduce it. Once a competition starts to have a scrambled ordered, it stays this way, but the problem does not appear again when re-uploading the same bundle and submissions.

Issues:

• n/a scores are ranked higher than numerical values #1284
• Leaderboard not well ordered #1171 <-- P1
• Another example below in comments

2. N/a scores

We have several issues that report N/a scores on the leaderboard. N/a is a replacement of score shown for a task not found. The problem is that the scores and tasks are separate and are sent to the frontend where they are matched and then scores are shown for a task or several tasks. This opens room for inconsistencies.

I propose to make the leaderboard on the backend and just send it to be displayed without any extraction or computation. This way we will never have n/a scores.

• a) N/a when submission rule is "Force Last"

• n/a scores on leaderboard if Submission Rule is Force Last #1055

• Display bug (n/a) when force submission to the leaderboard #1275

• b) N/a for multi-task when submission rule is "Force Last"

• Multi-task not working with Force Last submission rule #2211

• Migration with "Force Last" setting gives n/a scores #1174

• Score n/a for one of the subtasks when submitting for both subtasks at once. #1570

• c) N/a score after auto-migration (Fix auto-migration to next phase #2239)

• Issues when migrating to a new phase with fact sheet answers/different tasks that outputs identical data #1472

• Migration with "Force Last" setting gives n/a scores #1174

3. When multiple tasks, how should we order the entries?

Right now it seems that the ordering is based on a sum of scores across tasks.

4. Request: separate leaderboards for tasks

• Feature Request: Support for Separate Leaderboards per Task #1392

[Didayolo]

How much work does this change represent?

[ihsaan-ullah]

I guess quite amount of work. It may also disclose some hidden bugs. But it seems important because this problem is hanging there from a long time.

This is just the start of the discussion. We may want to discuss in detail how things are done now in the code and what we want to do change there and discuss the benefits.

One thing I insist on is to move the logic to the backend. Frontend should not have logic (unless really needed). This simplifies the front-end and less buggy and makes the backend strong. Additionally, backend should be written in a clear way so that in this effort we don't make it buggy.

[ihsaan-ullah]

Some more problems I discovered:

• When you have more than one task and leaderboard rule is Force last or Force last multiple, you see n/a because one of the child submission is on the leaderboard and the other one shows n/a. We should NOT show the child submission on the leaderboard. Only the parent submission should be shown with scores from child submissions. It looks like when the rule is applied, it is applied to submissions regardless of if they are parent or child

• It looks like for all the tasks you must have the same columns in the leaderboard

• Computation columns are populated once (when submission is processed). If you change e.g. avg computation to sum, that will work for future submissions but not for that past (already processed) ones.

• Submission ID is different in submission panel and leaderboard. This happens when you have multiple tasks