I have a question: why do we sample 256 samples of length seqlen when evaluating the C4 dataset? This specifically refers to the get_c4 and get_c4_new functions in datautils.py.
The paper mentions that 128 samples are drawn from C4 as the calibration dataset, but I haven’t found explanations in the paper for why 256 samples are used during C4 dataset PPL evaluation.
Should we evaluate on the full C4 dataset instead? Could anyone familiar with this help explain? Thanks a lot!
I have a question: why do we sample 256 samples of length seqlen when evaluating the C4 dataset? This specifically refers to the get_c4 and get_c4_new functions in datautils.py.
The paper mentions that 128 samples are drawn from C4 as the calibration dataset, but I haven’t found explanations in the paper for why 256 samples are used during C4 dataset PPL evaluation.
Should we evaluate on the full C4 dataset instead? Could anyone familiar with this help explain? Thanks a lot!