AttributeError: 'AcceleratedOptimizer' object has no attribute '_modules'. Did you mean: '__module__'?

[endoedgar]

I was running some training and had to move to another machine, but got an weird issue that never happened before:

5tts-1  | accelerate launch --mixed_precision=bf16 /workspace/F5-TTS/src/f5_tts/train/finetune_cli.py --exp_name F5TTS_v1_Base --learning_rate 7.5e-05 --batch_size_per_gpu 8000 --batch_size_type frame --max_samples 64 --grad_accumulation_steps 1 --max_grad_norm 1 --epochs 684 --num_warmup_updates 500 --save_per_updates 1500 --keep_last_n_checkpoints 3 --last_per_updates 3000 --dataset_name pt-br --tokenizer char --log_samples
f5tts-1  | 
f5tts-1  | The following values were not passed to `accelerate launch` and had defaults used instead:
f5tts-1  |      `--num_processes` was set to a value of `1`
f5tts-1  |      `--num_machines` was set to a value of `1`
f5tts-1  |      `--dynamo_backend` was set to a value of `'no'`
f5tts-1  | To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
f5tts-1  | 
f5tts-1  | vocab :  2545
f5tts-1  | 
f5tts-1  | vocoder :  vocos
f5tts-1  | Using logger: None
f5tts-1  | Loading dataset ...
f5tts-1  | Download Vocos from huggingface charactr/vocos-mel-24khz
f5tts-1  | 
f5tts-1  | Sorting with sampler... if slow, check whether dataset is provided with duration:   0%|          | 0/10000 [00:00<?, ?it/s]
f5tts-1  | Sorting with sampler... if slow, check whether dataset is provided with duration: 100%|██████████| 10000/10000 [00:00<00:00, 4090407.65it/s]
f5tts-1  | 
f5tts-1  | Creating dynamic batches with 8000 audio frames per gpu:   0%|          | 0/10000 [00:00<?, ?it/s]
f5tts-1  | Creating dynamic batches with 8000 audio frames per gpu: 100%|██████████| 10000/10000 [00:00<00:00, 5593897.04it/s]
f5tts-1  | Traceback (most recent call last):
f5tts-1  |   File "/workspace/F5-TTS/src/f5_tts/train/finetune_cli.py", line 214, in <module>
f5tts-1  |     main()
f5tts-1  |   File "/workspace/F5-TTS/src/f5_tts/train/finetune_cli.py", line 207, in main
f5tts-1  |     trainer.train(
f5tts-1  |   File "/workspace/F5-TTS/src/f5_tts/model/trainer.py", line 327, in train
f5tts-1  |     start_update = self.load_checkpoint()
f5tts-1  |                    ^^^^^^^^^^^^^^^^^^^^^^
f5tts-1  |   File "/workspace/F5-TTS/src/f5_tts/model/trainer.py", line 245, in load_checkpoint
f5tts-1  |     self.accelerator.unwrap_model(self.optimizer).load_state_dict(checkpoint["optimizer_state_dict"])
f5tts-1  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/accelerator.py", line 2866, in unwrap_model
f5tts-1  |     return extract_model_from_parallel(model, keep_fp32_wrapper, keep_torch_compile)
f5tts-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/utils/other.py", line 176, in extract_model_from_parallel
f5tts-1  |     has_compiled = has_compiled_regions(model)
f5tts-1  |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/utils/other.py", line 70, in has_compiled_regions
f5tts-1  |     if module._modules:
f5tts-1  |        ^^^^^^^^^^^^^^^
f5tts-1  | AttributeError: 'AcceleratedOptimizer' object has no attribute '_modules'. Did you mean: '__module__'?
f5tts-1  | Traceback (most recent call last):
f5tts-1  |   File "/opt/conda/bin/accelerate", line 8, in <module>
f5tts-1  |     sys.exit(main())
f5tts-1  |              ^^^^^^
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/commands/accelerate_cli.py", line 50, in main
f5tts-1  |     args.func(args)
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/commands/launch.py", line 1198, in launch_command
f5tts-1  |     simple_launcher(args)
f5tts-1  |   File "/opt/conda/lib/python3.11/site-packages/accelerate/commands/launch.py", line 785, in simple_launcher
f5tts-1  |     raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
f5tts-1  | subprocess.CalledProcessError: Command '['/opt/conda/bin/python', '/workspace/F5-TTS/src/f5_tts/train/finetune_cli.py', '--exp_name', 'F5TTS_v1_Base', '--learning_rate', '7.5e-05', '--batch_size_per_gpu', '8000', '--batch_size_type', 'frame', '--max_samples', '64', '--grad_accumulation_steps', '1', '--max_grad_norm', '1', '--epochs', '684', '--num_warmup_updates', '500', '--save_per_updates', '1500', '--keep_last_n_checkpoints', '3', '--last_per_updates', '3000', '--dataset_name', 'pt-br', '--tokenizer', 'char', '--log_samples']' returned non-zero exit status 1.
Gracefully stopping... (press Ctrl+C again to force)

If you see this issue and don't want to waste your time like me, it is because Accelerate version 1.7.0 has been released a few hours ago and caused some breaking changes with this code.

To make it work again, make sure you change the

"accelerate>=0.33.0",
to
"accelerate<=1.6.0",

in your dependencies property of the pyproject.toml file.

[SodaPrettyCold]

My hero👍

[SWivid]

Hi @endoedgar thanks for report,
Seems broke with huggingface/accelerate@7b5774a

Have you verified that the accelerate config is set use_dynamo to NO, or it will always raise an error no matter dynamo or not.

Likely fixed with huggingface/accelerate@e6e7175
We will push a new package here if their accelerate v1.8.0 is ready.

[endoedgar]

@SWivid I tried with the default settings, so it was set to NO. AFAIK at least on my machine with a NVIDIA 3090, I get completely different errors other settings for use_dynamo. And it might be because I'm missing the params to setup them correctly for those specific scenarios. Should I use one in specific? ( 'eager', 'aot_eager', 'inductor', 'aot_ts_nvfuser', 'nvprims_nvfuser', 'cudagraphs', 'ofi', 'fx2trt', 'onnxrt', 'tensorrt', 'aot_torchxla_trace_once', 'torhchxla_trace_once', 'ipex', 'tvm')

[SWivid]

Just use_dynamo NO is correct.
Thought it was for a mismatched default value when Accelerate team update v1.7.0 with new compilation function.