diff --git a/docs/source/en/api/pipelines/minimax_h3.md b/docs/source/en/api/pipelines/minimax_h3.md index cccf2d712b64..4d815156b2a5 100644 --- a/docs/source/en/api/pipelines/minimax_h3.md +++ b/docs/source/en/api/pipelines/minimax_h3.md @@ -27,40 +27,47 @@ MiniMax-H3 is integrated as [Modular Diffusers](../../modular_diffusers/overview MiniMax-H3 was released as two checkpoint partitions that share every component except the transformer, so the diffusers conversion puts both in **one repository**: -| Subfolder | Blocks | Tasks | -|---|---|---| -| `transformer/` | [`MiniMaxH3Blocks`] | `t2va` (text only) and `fl2va` (first and/or last keyframe) | -| `transformer_ref/` | [`MiniMaxH3Ref2VABlocks`] | `ref2va` (an ordered mix of image, video and audio references) | +| Subfolder | Workflows | +|---|---| +| `transformer/` | `t2va` (text only), `fl2va` (first and/or last keyframe) | +| `transformer_ref/` | `ref2va` (an ordered mix of image, video and audio references) | Everything but the transformer, i.e. the video VAE, the audio VAE, the Qwen3-VL conditioner, its tokenizer and processor, and the two schedulers, is shared and stored once. -The repository carries one `modular_model_index.json`, which names every component of both halves with its own loading spec. Each blockset declares only the components it runs, and `load_components` fetches exactly those subfolders: loading the `t2va` / `fl2va` half never touches `transformer_ref/`, and loading the `ref2va` half never touches `transformer/`. Nothing else in the repository is fetched either, which is what lets one repository carry the two partitions, and the original checkpoint folders next to the converted ones. +The conditioner is a `Qwen3VLForConditionalGeneration`, and MiniMax-H3 reads the *unnormalized* hidden state after its 50th decoder layer rather than the last one, so the full released checkpoint is used with its language-model head unused. -`modular_model_index.json` names the `t2va` / `fl2va` half as its own class, so [`~ModularPipeline.from_pretrained`] resolves that half. The `ref2va` half reads the very same file through its own blocks: +All three tasks are workflows of the one [`MiniMaxH3Blocks`], and the repository carries one `modular_model_index.json` naming every component with its own loading spec. To serve a single task, pass the workflow to `from_pretrained`: it keeps only that workflow's blocks, so the pipeline's signature (`pipe.doc`) documents exactly that task's inputs, only that task's components are declared, and `load_components` fetches exactly their subfolders — a `t2va` / `fl2va` pipeline never touches `transformer_ref/`, a `ref2va` one never touches `transformer/`. ```py import torch from diffusers import ModularPipeline -from diffusers.modular_pipelines import MiniMaxH3Ref2VABlocks -# `t2va` / `fl2va`: loads `transformer/`, and never `transformer_ref/`. -pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", workflow="ref2va") +pipe.load_components(dtype=torch.bfloat16) +``` -# `ref2va`: loads `transformer_ref/`, and never `transformer/`, out of the same repository. -pipe = MiniMaxH3Ref2VABlocks().init_pipeline("MiniMaxAI/MiniMax-H3") +> [!TIP] +> `pipe.doc` prints what the pipeline in front of you takes and returns — every input with its default, the components it expects and the outputs it produces. Pruned to one workflow it describes exactly that task, which is the quickest way to see what a request needs before making one. -pipe.load_components(dtype=torch.bfloat16) +To keep every workflow available on one pipeline instead, leave the `workflow` argument out: the pipeline then picks the workflow per call from the inputs it is passed. + +```py +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") ``` -Each blockset also carries the workflows it serves — `t2va`, `fl2va` and `fl2va_last_frame` for [`MiniMaxH3Blocks`], `ref2va` for [`MiniMaxH3Ref2VABlocks`] — which name the inputs each task requires. +The *loading* can still go one workflow at a time. This one call fetches `transformer/` and every shared component, which serves both `t2va` and `fl2va`: -The conditioner is a `Qwen3VLForConditionalGeneration`, and MiniMax-H3 reads the *unnormalized* hidden state after its 50th decoder layer rather than the last one, so the full released checkpoint is used with its language-model head unused. +```py +pipe.load_components(workflow="t2va", dtype=torch.bfloat16) +``` + +A plain `load_components()` with no `workflow=` pulls **both** 61.7GB transformer partitions, which is what lets one pipeline serve all three workflows without another loading call. Pair it with a [`ComponentsManager`] and auto offloading: the weights live in host RAM and the manager moves onto the accelerator just what each step needs, so when the `ref2va` denoiser wants the device the strategy offloads whatever frees enough room. See [Memory](#memory) for the recipes. ## Two schedulers -Video and audio latents step down two different schedules inside a single transformer call per step, which is why both blocksets expect two [`MiniMaxH3Scheduler`] instances: `scheduler` for the video latents (`shift=12.0` in the released checkpoints) and `audio_scheduler` for the audio latents (`shift=3.0`). +Video and audio latents step down two different schedules inside a single transformer call per step, which is why the blocks expect two [`MiniMaxH3Scheduler`] instances: `scheduler` for the video latents (`shift=12.0` in the released checkpoints) and `audio_scheduler` for the audio latents (`shift=3.0`). -The checkpoint is guidance-distilled: guidance is baked into the weights, so there is no guider, no `negative_prompt` and no `guidance_scale`, and every step runs exactly one forward pass. +Both transformer partitions are guidance-distilled, so this holds for every workflow: guidance is baked into the weights, there is no guider, no `negative_prompt` and no `guidance_scale`, and every step runs exactly one forward pass. ## Generation constraints @@ -81,7 +88,7 @@ from diffusers import ComponentsManager, ModularPipeline manager = ComponentsManager() pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager) -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(workflow="t2va", dtype=torch.bfloat16) manager.enable_auto_cpu_offload(device="cuda", memory_reserve_margin="12GB") pipe.transformer.set_attention_backend("_flash_3_hub") # Hopper, roughly 3x faster; kernels fetched from the Hub ``` @@ -117,7 +124,7 @@ pipe.update_components( ), ), ) -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(workflow="t2va", dtype=torch.bfloat16) # version=2 int8 tensors are pinnable, which streamed offload needs, and freezing removes the one autograd # path the quantized tensors cannot serve. @@ -147,7 +154,7 @@ pipe.update_components( "MiniMaxAI/MiniMax-H3", subfolder="text_encoder", dtype=torch.bfloat16, device_map={"": "cuda:1"} ), ) -pipe.load_components(dtype=torch.bfloat16) +pipe.load_components(workflow="t2va", dtype=torch.bfloat16) pipe.transformer.to("cuda:0") pipe.vae.to("cuda:0") pipe.audio_vae.to("cuda:0") @@ -161,31 +168,37 @@ Two 80 GB cards run full bfloat16 this way with nothing streaming; two 48 GB car ```py import torch -from diffusers import ModularPipeline +from diffusers import ComponentsManager, ModularPipeline from diffusers.utils import load_image from diffusers.utils.export_utils import encode_video -pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3") -pipe.load_components(dtype=torch.bfloat16) -pipe.to("cuda") +# 61.7GB of transformer and 62.1GB of conditioner do not sit on one accelerator, so the components are +# registered in a manager that moves each one on and off as the blocks reach it. See [Memory](#memory). +manager = ComponentsManager() +manager.enable_auto_cpu_offload(device="cuda") + +pipe = ModularPipeline.from_pretrained("MiniMaxAI/MiniMax-H3", components_manager=manager) +pipe.load_components(workflow="fl2va", dtype=torch.bfloat16) prompt = "A red fox trotting through a snowy pine forest, snow crunching underfoot" +# `output=` returns exactly the named outputs instead of the whole pipeline state. +outputs = ["videos", "audio", "sampling_rate"] # Text to video + audio. -state = pipe(prompt=prompt, generator=torch.Generator().manual_seed(42)) +results = pipe(prompt=prompt, num_frames=124, generator=torch.Generator().manual_seed(42), output=outputs) # First frame (and optionally last frame) to video + audio. The canvas follows the first keyframe. image = load_image( "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/astronaut.jpg" ) -state = pipe(prompt=prompt, image=image, generator=torch.Generator().manual_seed(42)) +results = pipe(prompt=prompt, image=image, num_frames=124, generator=torch.Generator().manual_seed(42), output=outputs) encode_video( - state.get("videos")[0], + results["videos"][0], fps=24, output_path="minimax_h3_fl2va.mp4", - audio=state.get("audio")[0], - audio_sample_rate=state.get("sampling_rate"), + audio=results["audio"][0], + audio_sample_rate=results["sampling_rate"], ) ``` @@ -193,61 +206,117 @@ Video and audio are generated jointly and come out of the call as separate outpu ## Omni-references -[`MiniMaxH3Ref2VABlocks`] conditions on an ordered list of references: up to 9 images, 3 videos and 3 audio clips, 12 in total. The order is semantic. It labels the references in the prompt presentation (`""`, `"