how to prepare training dataset ?

[zyjcsf]

Hi,
Thank you very much for your sharing. Could you please provide an example of model training? For instance, how to prepare the training data?

[yairl]

@yoadsn

[yoadsn]

@zyjcsf Generally speaking the process will follow the following steps:

Prepare a Whisper compatible training data, matching the expectations of this training script#21

here is an example of a dataset the fits the expectations (It's a hebrew dataset, but other languages could be trained).

Please read the dataset card for details of how this dataset is structured - key points:

• We expect audio samples up to 30s
• Each samples should have text describing the spoken audio signal
• If the sample "has timestamps" - we expect timestamp tokens to properly segment the text, and the "has timestamps" flag to be on
• If the same "has previous text" - we expect this text to be provided and the "has prev" flag to be on (if with timestamps, then the prev text should have them as well).

We prepare such datasets from different sources on this repo.

Preprocess the dataset

You could do this while training, but if you want to reuse the preprocessed dataset - I suggest to create it once up front and use it.
A command like this would work:

python train-whisper.py \
  --output_model_name T \
  --model_name /hf/openai/whisper-large-v3 \
  --train_datasets "/whisper_datasets/ds_whisper_training:train" \
  --eval_dataset "/whisper_datasets/ds_whisper_training:test" \
  --save_processed "/whisper_preprocessed/ds_processed" \
  --ds_processor_proc_num 110 \
  --include_timestamps_prob 0.5 \
  --include_prev_text_prob 0.5 \
  --skip_push_to_hub

Notes:

• output model name does not matter here.
• We do need a proper model name since this will dictate the tokenizer and audio feature extraction used.
• train/eval datasets are the dataset I described above matching the expectations
• include timestamps/prev text - ask the script to sample from the dataset and remove some feature to match the distribution you aim for. 0.5 for both, if you have enough such data is advised. especially for training the turbo size.
• You could skip pushing to hub if you store all locally, you can push to hub to pull from other machines if you want to.

Run Training

You use the same script, indicating which processed ds to use, and provide hyper params for training.
here is a command to start from:

train-whisper.py \
    --output_model_name /models/ft_large_v3_exp_1 \
    --use_preprocessed /whisper_preprocessed/ds_processed \
    /whisper_preprocessed/ds2_processed \
    /whisper_preprocessed/ds3_processed \
    --use_preprocessed_probs 0.15 0.8 0.05 \
    --model_name /hf/openai/whisper-large-v3 \
    --per_device_train_batch_size 2 \
    --gradient_accumulation_steps 2 \
    --per_device_eval_batch_size 16 \
    --learning_rate 1e-5 \
    --warmup_steps 800 \
    --lr_scheduler_type linear \
    --mixed_precision bf16 \
    --attn_implementation sdpa \
    --skip_push_to_hub \
    --num_train_epochs 5 \
    --logging_steps 20 \
    --eval_steps 400 \
    --save_steps 400 \
    --save_only_model \
    --max_checkpoints_to_keep 50 \
    --max_eval_set_size 2000 \
    --run_name exp_1

Notes:

• You can provide multiple datasets, but then you also need to provide the sampling ratio between them.
• you start from a pretrained model usually - but in theory could modify the script to start from scratch
• batch sizes, learning rate, warmup steps - those depend on you environment, data, etc.
• use bf16 and sdpa if your hardware supports it - it will train faster and with less memory

Please know, we don't have the ability to document everything and we really hope you can spend time reading the script code, otherwise - you might miss options to do more, or get the results you look for.

Good Luck.