[AAC] - train BEATs encoder

[songkyu-park]

I'm trying to train by changing the slam-aac encoder to beats.
(I've already completed training and evaluation with eats)

1. Change ++model_config.encoder_name=beats
2. Specify the model and path
3. Download BEATs to the fairseq subfolder

Is this all I need to do? (Diverge when trying to train)

@cwx-worst-one

[cwx-worst-one]

If the model loads correctly, there shouldn't be other major issues. Did the training diverge right from the beginning? If the divergence occurred partway through due to instability, you could try disabling FP16 by setting train_config.use_fp16 = false.

[songkyu-park]

@cwx-worst-one

When I start by making a fine-tune code like below, it diverges from the moment training starts. Is there anything wrong?

run_dir=/workspace/SLAM-LLM
cd $run_dir
code_dir=examples/slam_aac

#encoder_fairseq_dir=/workspace/SLAM-LLM/fairseq/BEATs
encoder_fairseq_dir=/workspace/SLAM-LLM/src/slam_llm/models/BEATs

audio_encoder_path=/workspace/1.models/BEATs_iter3_plus_AS2M.pt
llm_path=/workspace/1.models/vicuna-7b-v1.5

seed=10086
btz=4
lr=8e-6
encoder_projector_ds_rate=5

train_jsonl_path=/workspace/2.manifest/clotho/augmented_clotho.jsonl
val_jsonl_path=/workspace/2.manifest/clotho/validation.jsonl

exp_name=BEATs_clotho_fine
output_dir=/workspace/3.fine_tuning/clotho_BEATs

ckpt_path=/workspace/1.models/pretrained/check/model.pt # path to load the pre-trained model
peft_ckpt="''"

hydra_args="
hydra.run.dir=$output_dir
++model_config.llm_name=vicuna-7b-v1.5
++model_config.llm_path=$llm_path
++model_config.llm_dim=4096
++model_config.encoder_name=beats
++model_config.encoder_ds_rate=2
++model_config.encoder_projector_ds_rate=$encoder_projector_ds_rate
++model_config.encoder_path=$audio_encoder_path
++model_config.encoder_dim=768
++model_config.encoder_projector=linear
++model_config.encoder_fairseq_dir=$encoder_fairseq_dir
++dataset_config.encoder_projector_ds_rate=${encoder_projector_ds_rate}
++dataset_config.dataset=audio_dataset
++dataset_config.train_data_path=$train_jsonl_path
++dataset_config.val_data_path=$val_jsonl_path
++dataset_config.input_type=mel
++dataset_config.fbank_mean=-4.268
++dataset_config.fbank_std=4.569
++dataset_config.model_name=beats
++dataset_config.fixed_length=true
++dataset_config.target_length=1024
++train_config.model_name=aac
++train_config.freeze_encoder=true
++train_config.freeze_llm=false
++train_config.batching_strategy=custom
++train_config.warmup_steps=1000
++train_config.total_steps=100000
++train_config.lr=$lr
++train_config.validation_interval=500
++train_config.batch_size_training=$btz
++train_config.val_batch_size=$btz
++train_config.num_workers_dataloader=4
++train_config.use_fp16=false
++train_config.output_dir=$output_dir
++train_config.seed=${seed}
++train_config.use_peft=true
++train_config.peft_config.peft_method=lora
++train_config.specaug=false
++log_config.log_file="${output_dir}/train.log"
++log_config.wandb_dir=${output_dir}
++log_config.wandb_entity_name=wxc12
++log_config.wandb_project_name=slam-llm
++log_config.wandb_exp_name=$exp_name
++log_config.use_wandb=false
++metric=acc
++ckpt_path=$ckpt_path
++peft_ckpt=$peft_ckpt \

[cwx-worst-one]

Oh! I think I found the issue — BEATs and EAT use different normalization settings during training, so they require different normalization parameters.

You should update dataset_config.fbank_mean and dataset_config.fbank_std accordingly, as shown here:

SLAM-LLM/src/slam_llm/models/BEATs/BEATs.py

Lines 119 to 124 in 440ece6

	def preprocess(
	cls,
	source: torch.Tensor,
	fbank_mean: float = 15.41663,
	fbank_std: float = 6.55582,
	) -> torch.Tensor:

Try updating these values and retraining the model — that may help.

[songkyu-park]

@cwx-worst-one
after changing the parameters, the loss remains as nan during training. In the collator section of audio_dataset.py, I modified the code as follows:

audio_mel_mask = torch.ones(len(samples), audio_mel_max_length) # changed from zeros to ones
for line, sample in enumerate(samples):
audio_mel_mask[line, :sample['audio_mel'].shape[0]] = 0 # changed from 1 to 0

After making this change, it seems like training is working. Can you confirm if this modification is correct?

[cwx-worst-one]

Oh, I think you're right — I just checked the official BEATs code, and the model does use a zero padding mask during the forward pass.

It’s likely that we missed this during code integration. Thanks a lot for pointing it out!