Training 0.6B model on decontaminated data (Nemotron-CC)

[vitiugin]

Compare performance of 0.6B 100BT models trained on Nemotron-CC and decontaminated Nemotron-CC

[vitiugin]

Model trained on decontaminated Nemotron-CC in the queue. For ~100B tokens (~30M documents) filtered out ~30K documents.

[vitiugin]

Results of Nemotron-CC and DCLM doesn't show drop in performance after decontamination of both datasets:

[vitiugin]

In addition, I got results for both datasets evaluated on dclm-core-22 (with oellm-cli evaluation settings).

Evaluations include: hellaswag (10 shot), arc_easy, openbookqa, arc_challenge, hellaswag (0-shot), piqa

Another set of evaluations: copa, commonsense_qa, agieval_lsat_ar ,wsc273, boolq, lambada_openai, winogrande