In Tequila, training from scratch or continuing training?

[tuzhijun]

May I ask whether the results reported in the Tequila paper were obtained by training from scratch on 10B tokens, or by continuing training on 10B tokens based on pre-trained Llama3-1B and Llama3-3B parameters?

[yghstill]

@tuzhijun Based on pre-trained Llama3-1B and Llama3-3B parameters.

[tuzhijun]

May I ask whether the training data you used is from https://huggingface.co/datasets/sumuks/Ultra-FineWeb-10B, or did you randomly sample 10B tokens from Ultra-FineWeb yourself? In addition, are there any plans to open-source this 10B training dataset and the trained Tequila model?

[Little0o0]

@tuzhijun The training data is from Ultra-FineWeb (https://huggingface.co/datasets/openbmb/Ultra-FineWeb). We will open source the datasets soon. Currently, we have no plans to open source the trained models.

[tuzhijun]

May I ask when this training data plan will be released? We hope to strictly follow the experimental setup described in the paper to reproduce the results. This work is very meaningful and should serve as a baseline for all the subsequent low-bit LLM training efforts.