🚨 Give mixed-rank LoRAs without alpha keys their intended scale

[apolinario]

get_peft_kwargs takes lora_alpha from the first entry of the rank dict and never revisits it, and with no alpha keys alpha_pattern stays empty. For an adapter with mixed ranks and no alpha keys that means every module whose rank differs from the first key's rank gets an arbitrary scale: PEFT falls back to the global lora_alpha (peft/tuners/lora/model.py, alpha_pattern.get(key, config.lora_alpha)), so the direction of the error depends on state dict key order.

Real case: larryvrh/MiniMax-H3-Turbo-Lora ships 518 tensors, ranks 64 (attention/FFN) and 16 (AdaLN), zero alpha keys, and states W_eff = W + lora_B @ lora_A, alpha = rank. The AdaLN key sorts first, so lora_alpha = 16, r = 64 and all rank 64 modules load at 0.25x. Its pre converted mirror (InstantX/MiniMax-H3-Turbo-Lora-Diffusers) documents a manual network_alphas workaround for exactly this.

The fix mirrors the ranks into the alphas when the checkpoint brings no alpha information, which is the diffusers/PEFT convention (alpha == rank, scale 1.0) and what load_lora_adapter already does for SAI control LoRAs a few lines below. It is gated on the absence of alpha data: mixed rank adapters with a declared alpha keep it (a uniform declared alpha must not fall back to per module ranks), uniform rank adapters are a no-op, and the metadata serialization path is unaffected because it bypasses this function entirely.

🚨 Behavior change for existing mixed rank, no alpha LoRAs: they previously loaded at an order dependent wrong scale and now load at the intended scale 1.0.

Kohya conversion precedent for the invariant: lora_conversion_utils.py folds alpha / rank into the weights with default_alpha = rank, i.e. it also assumes scale 1.0 when alpha is absent.

Tests cover the mixed rank no alpha case (including key order independence), uniform rank, uniform declared alpha, and per module alphas.

[apolinario]

(unsure if we need this test)

[BenjaminBossan]

Thanks for the fix.

From the PEFT point of view, this looks correct. One edge case I'm not sure about is if rank_dict contains exactly one unique value: I assume in that case, we would never reach this function and it's okay. If we can, however, reach this function, then it's uncovered.

I don't know enough about the Diffusers ecosystem to judge whether this change could break existing checkpoints, but it's reasonable to assume that no alpha implies the intended scale is 1.0.

As for the test, it probably doesn't add a lot, I'd say is best if a maintainer takes a look.

[sayakpaul]

If this is the case then do we need the changes made in the H3 LoRA loading PR?

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.