Welcome v5

[LysandreJik]

In this issue we share our plan for the upcoming version 5 of transformers. We've talked about version 5 for years and it's finally around the corner! We'll release a blog post announcing the focus of this release shortly, and wanted to share what we believe the process will look like over the coming weeks.

• Soon, a new branch named v4 will be created on the repository. It is from this branch that all v4-related updates will take place. Going forward, main will act as the version 5 branch.
• For the next few weeks, every PR except breaking changes or significant refactors will be merged in both main and v4.
• In a few weeks, we release what will likely be one of the last minor v4 releases (v4.57.0)
• A few weeks later, we will release v5. We will aim to limit, as much as possible, the breaking changes within that release; but expect a migration guide as well as some specific breaking changes enabling much more versatile, performant, and cleaner code going forward.
• Over the next few months, we'll continue patching the v4 branch and will release patch updates.

The v5 on pypi will be preceded by RC releases that we will share in this issue. Please subscribe to this issue to be updated, and let us know if you have thoughts about the outlined process above.

[LysandreJik]

Please see the following issue for the tokenization plan for version 5: #40938

[daskol]

Thanks for sharing the v5 plan and opening this thread. I have a few general questions that may be useful for end users and downstream maintainers:

• Is there a centralized, running list of proposed or accepted breaking changes for v5?
• Is there an index of RFCs or enhancement proposals, along with their discussion/decision status?
• What is the deprecation policy for v5 (and remaining v4 support), and what are the time windows for v5 RC testing and v4 deprecating?
• Will breaking changes in transformers require coordinated updates in other Hugging Face packages (e.g., huggingface-hub, tokenizers, peft, trl, etc.)?

Thanks again for all the work on the v5 transition.

[LysandreJik]

Hey @daskol, thanks for chiming in!

There is no centralized list at the moment, but we recommend you follow the following label in case you want to see the current changes being done: for_v5?.

We're in the process of opening some draft PRs for some of these, and I'll link the most relevant topics here over the next few hours.

The time window for v5 testing will vary according to the issues that users face when working with the rcs; we aim for an initial RC release in a few weeks' time, and we'll invite everyone to try it out and to share any issues they might face. The migration guide we'll be releasing alongside it should hopefully help make the transition seamless.

The changes might require some minor version updates in the downstream packages and community packages (Unsloth, Axolotl, vllm, ...). We're coordinating with every tool to ensure that everything works as expected.

Stay tuned as we'll share some of the ongoing refactors/simplifications in this issue over the next few hours.

[LysandreJik]

🚨 We're removing the TensorFlow and Jax parts of the library in v5: #40760.

These will continue to exist in v4 (latest release as of now being v4.57.0) but their maintenance will be low as usage is already extremely minimal.

This will help us focus fully on torch going forward and will greatly reduce the maintenance cost of models. We are working with tools from the Jax ecosystem still (such as MaxText) in order to see how we can remain compatible with their tool while keeping torch as the only backend for now.

[SunMarc]

🚨 Quantization breaking change - load_in_4bit and load_in_8bit` removed

We will remove load_in_4bit and load_in_8bit args in from_pretrained method. These args were introduced when we first started integrating bitsandbytes quantization into transformers but since then, we have added lots of new methods (e.g AWQ, FP8 ...) and the API changed to reflect that. Users needs to pass quantization_config argument instead

For example, to load a 4-bit bitsandbytes model, you will need to do the following

from transformers import AutoModelForCausalLM, BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(load_in_4bit=True)

model_4bit = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3B",
    device_map="auto",
    quantization_config=quantization_config
)

Link to the PR #41287

[zucchini-nlp]

Processing breaking changes

• In processing classes each attribute will be serialized under processor_config.json as a nested dict, instead of serializing attributes in their own config files. Loading will be supported for all old format processors (🚨 [v5] Toggle the serialization format in processors #41474)
• XXXFeatureExtractors classes are completely removed in favor of XXXImageProcessor class for all vision models (🚨 [v5] Delete feature extractors used for vision #41174)
• Minor change: XXXFastImageProcessorKwargs is removed in favor of XXXImageProcessorKwargs which will be shared between fast and slow processors (🚨 [unbloating] unify TypedDict usage in processing #40931)

Configuration breaking changes

• Methods to init a nested config such as from_xxx_config are deleted. Configs can be init from the __init__ method in the same way (🚨 [v5] Rendundant code in nested configs #41314)

Modeling breaking changes

• Some RotaryEmbeddings layers will start returning a dict of tuples, in case the model uses several RoPE configurations (Gemma2, ModernBert). Each value will be a tuple of "cos, sin" per RoPE type.
• Config attribute for RotaryEmbeddings layer will be unified and accessed via config.rope_parameters. Config attr for rope_theta might not be accessible anymore for some models, and instead will be in config.rope_parameters['rope_theta']. BC will be supported for a while as much as possible, and in the near future we'll gradually move to the new RoPE format (🚨 [v5] Refactor RoPE for layer types #39847)

[vasqu]

Feature breaking changes

We decided to remove some features for the upcoming v5 as they are currently only supported in a few old models and no longer integrated in current model additions. It's recommended to stick to v4.x in case you need them. Following features are affected:

• No more head masking, see 🚨 [v5] Remove headmasking #41076. This feature allowed to turn off certain heads during the attention calculation and only worked for eager.
• No more relative positional biases in Bert-like models, see 🚨 [v5] Remove relative position embeddings (for bert like models) #41170. This feature was introduced to allow relative position scores within attention calculations (similar to T5). However, this feature is barely used in official models and a lot of complexity instead. It also only worked with eager.
• No more head pruning, see 🚨 [v5] Prune prune_heads #41417 by @gante. As the name suggests, it allowed to prune heads within your attention layers.

[SunMarc]

🚨 TrainingArguments & Trainer breaking changes

Removing arguments without deprecation cycle in TrainingArguments due to low usage

• mp_parameters -> legacy param that was later on added to sagemaker trainer
• _n_gpu -> not intended for users to set, we will initialize it correctly instead of putting it in the TrainingArguments
• overwrite_output_dir - > replaced by resume_from_checkpoint and it was only used in examples script, no impact on Trainer.
• logging_dir -> only used for tensorboard, set TENSORBOARD_LOGGING_DIR env var instead
• jit_mode_eval -> use use_torch_compile instead as torchscript is not recommended anymore
• tpu_num_cores-> It is actually better to remove it as it is not recommended to set the number of cores. By default, all tpu cores are used . Set TPU_NUM_CORES env var instead
• past_index -> it was only used for a very small number of models that have special architecture like transformersxl + it was not documented at all how to train those model
• ray_scope -> only for a minor arg for ray integration. Set RAY_SCOPE var env instead
• warmup_ratio -> use warmup_step instead. We combined both args together by allowing passing float values in warmup_step.

Removing deprecated arguments in TrainingArguments

• fsdp_min_num_params and fsdp_transformer_layer_cls_to_wrap -> use fsdp_config
• tpu_metrics_debug -> debug
• push_to_hub_token -> hub_token
• push_to_hub_model_id and push_to_hub_organization -> hub_model_id
• include_inputs_for_metrics -> include_for_metrics
• per_gpu_train_batch_size -> per_device_train_batch_size
• per_gpu_eval_batch_size -> per_device_eval_batch_size
• use_mps_device -> mps will be used by default if detected
• fp16_backend and half_precision_backend -> we will only rely on torch.amp as everything has been upstream to torch
• no_cuda -> use_cpu
• include_tokens_per_second -> include_num_input_tokens_seen
• use_legacy_prediction_loop -> we only use evaluation_loop function from now on

Removing deprecated arguments in Trainer

• tokenizer in initialization -> processor
• model_path in train() -> resume_from_checkpoint

Removed features for Trainer

• sigpot integration for hp search was removed as the library was archived + the api stopped working
• drop support for sagemaker API <1.10
• bump accelerate minimum version to 1.1.0

New defaults for Trainer

• use_cache in the model config will be set to False. You can still change the cache value through TrainingArguments usel_cache argument if needed.

[gante]

CLI breaking changes

• The deprecated transformers-cli ... command was deprecated, transformers ... is now the only CLI entry point (🚨 [v5] remove deprecated entry point #40997)
• See other CLI breaking changes in Welcome v5 #40822 (comment)

generate breaking changes

• Old, deprecated output type aliases were removed (e.g. GreedySearchEncoderDecoderOutput). We now only have 4 output classes built from the following matrix: decoder-only vs encoder-decoder, uses beams vs doesn't use beams (🚨 [v5] remove generate output retrocompatibility aliases #40998)
• Removed deprecated classes regarding decoding methods that were moved to the Hub due to low usage (constraints and beam scores) (🚨 [v5] remove deprecated generate classes (constraints and beam scorers) #41223)
• If generate doesn't receive any KV Cache argument, the default cache class used is now defined by the model (as opposed to always being DynamicCache) (🚨 [v5] generate delegates default cache initialization to the model #41505)