incremental training?

[dcsan]

is there any way to re-train without building everything from scratch?

In addition to our labels, I am using a large DB of pre-trained word vectors, and just adding a few new labels seems to require processing all of the data again?

[cpuhrsch]

Hello @dcsan,

Thank you for your post. I'm adding a related issue to this post. Please consider commenting on it instead and closing this to deduplicate issues.

#272

Thanks,
Christian

[dcsan]

is #272 the same topic?

Saving model after epoch(s): Looking for modification directions #272

[airlsyn]

@dcsan Hope this PR #423 be helpful

[SergeiAlonichau]

This PR implements proper incremental training in fast text: #1327 or here https://github.com/SergeiAlonichau/fastText .

It allows for:

1. test after each epoch
2. checkpointing
3. training on large data which does not fit into memory (largest I tested was 1.6TB)
4. finetuning already trained models

The trained model is indistinguishable from a model that was created by an original tool and can be used for inference by the old code.

[rtkclouds]

This PR implements proper incremental training in fast text: #1327 or here https://github.com/SergeiAlonichau/fastText .

It allows for:

1. test after each epoch
2. checkpointing
3. training on large data which does not fit into memory (largest I tested was 1.6TB)
4. finetuning already trained models

The trained model is indistinguishable from a model that was created by an original tool and can be used for inference by the old code.

I did a test by creating a small hack, previously tokenizing the text with the llama 2 tokenizer and apparently it can create language models faster than the conventional way used, I'm going to use your update to train 2 trillion tokens, the same dataset from llama2