Run MiniMax H3 Locally with Diffusers

MiniMax H3 jointly generates video and audio in one pass. ComfyUI makes it easy to iterate on a node graph, but for a reproducible local pipeline you want the Python API directly. This guide drives H3 with Modular Diffusers — no workflow JSON, no nodes — and, because H3 is huge, spends most of its time answering the question that actually gates you: how much GPU memory do I need, and how do I split it?

What H3 loads

H3’s local integration is built around a large denoising transformer plus Qwen3-VL text/image conditioning, separate video and audio VAEs, and their schedulers. The two dominant BF16 components are roughly:

Before the VAEs, schedulers, and activations. That sum — about 124 GB for those two alone — is why “load the whole pipeline to one card in bfloat16” does not work on any single GPU you can buy, and why memory management is the real subject of this page.

H3 uses Modular Diffusers

The current H3 integration is Modular Diffusers only. Use:

from diffusers import ModularPipeline

not a generic DiffusionPipeline. You select exactly one workflow when you load:

t2va    text → video + audio
fl2va   first/last frame → video + audio
ref2va  ordered image/video/audio references → video + audio

Option 1 — One high-memory GPU (80–96 GB)

If you have a single large card, the simplest setup lets Diffusers move the big components between GPU and host memory for you. Current Diffusers documentation explicitly describes an 80 GB single-GPU configuration; a 96 GB card gives more headroom, but it is not a strict minimum — full BF16 residency of both components would need well over 96 GB (see the table above), so offload is doing real work either way.

import torch

from diffusers import ComponentsManager, ModularPipeline
from diffusers.utils import load_image
from diffusers.utils.export_utils import encode_video

MODEL = "MiniMaxAI/MiniMax-H3"
IMAGE = "reference.png"          # first-frame keyframe
OUTPUT = "h3_fl2va.mp4"

PROMPT = """
A cinematic scene with natural motion and synchronized environmental audio.
"""

HEIGHT = 544
WIDTH = 960
NUM_FRAMES = 124                 # ~5 s at 24 fps
STEPS = 30

manager = ComponentsManager()

pipe = ModularPipeline.from_pretrained(
    MODEL,
    workflow="fl2va",
    components_manager=manager,
)

pipe.load_components(dtype=torch.bfloat16)

# Automatic CPU offload for a high-memory single-GPU configuration.
manager.enable_auto_cpu_offload(
    device="cuda",
    memory_reserve_margin="12GB",
)

image = load_image(IMAGE)

result = pipe(
    prompt=PROMPT,
    image=image,                  # first frame; add last_image= for first+last
    height=HEIGHT,
    width=WIDTH,
    num_frames=NUM_FRAMES,
    num_inference_steps=STEPS,
    generator=torch.Generator(device="cpu").manual_seed(42),
    output=["videos", "audio", "sampling_rate"],
)

# H3 returns video and audio separately — mux them here.
encode_video(
    result["videos"][0],
    fps=24,
    output_path=OUTPUT,
    audio=result["audio"][0],
    audio_sample_rate=result["sampling_rate"],
)

print(f"Saved: {OUTPUT}")

Two things to notice. First, there is no negative_prompt or guidance_scale — H3 is guidance-distilled, so the guidance behaviour is baked into the model and those controls do not exist on this pipeline. Second, the output gives you videos, audio, and the sampling_rate separately; encode_video is what fuses them into one file.

Why 960×544? It is divisible by 32 (H3’s grid), is substantially cheaper than the larger trained canvas, is directly supported by the upstream memory guidance, and is the size our working setup used. Higher resolutions increase both memory and runtime; treat this as the memory/performance-oriented starting point, not a cap.

This is the easiest route to understand, but with CPU offload the large components do not stay resident on the GPU for the whole run — expect transfers on the critical path.

Option 2 — Split across two GPUs (tested)

If you have two mid-sized cards (for example, two 48 GB), the better strategy is to divide conditioning and generation between them so each large component can stay resident on its own card. Our tested split was:

GPU 0                              GPU 1
─────────────────────────────      ─────────────────────────────
NVFP4 FL2VA transformer            INT8 Qwen3-VL conditioner
video VAE + audio VAE              before_encode
denoising + decoding blocks        text_encoder

The split works because FL2VA conditioning runs first: the prompt plus image pass through before_encode and Qwen3-VL to produce prompt embeddings, those embeddings are moved to GPU 0, and only then does the heavy denoising transformer generate the video/audio latents. GPU 1 never has to run the big transformer.

import gc
import torch

from diffusers import (
    MiniMaxH3Transformer3DModel,
    ModularPipeline,
    SequentialPipelineBlocks,
)
from diffusers.utils import load_image
from diffusers.utils.export_utils import encode_video
from transformers import Qwen3VLForConditionalGeneration
from transformers import TorchAoConfig as TransformersTorchAoConfig
from torchao.quantization import Int8WeightOnlyConfig

# ------------------------------------------------------------------
# Configuration — edit these local paths to match your checkout.
# ------------------------------------------------------------------

# Your downloaded H3 checkpoint (holds text_encoder/, VAEs, scheduler).
MODEL = "/home/user/models/MiniMax-H3"

# Locally converted NVFP4 transformer for the FL2VA/T2VA partition.
NVFP4_TRANSFORMER = "/home/user/models/MiniMax-H3-FL2VA-NVFP4/transformer"

IMAGE = "reference.png"
OUTPUT = "minimax_h3_fl2va.mp4"

DEVICE_MAIN = torch.device("cuda:0")   # generation side
DEVICE_TEXT = torch.device("cuda:1")   # conditioning side

HEIGHT = 544
WIDTH = 960
NUM_FRAMES = 124
STEPS = 30

PROMPT = """
A cinematic scene with natural subject motion, coherent camera movement,
and synchronized environmental audio.
"""

# ------------------------------------------------------------------
# 1. Load the already-converted NVFP4 FL2VA transformer on GPU 0
# ------------------------------------------------------------------

print("Loading NVFP4 FL2VA transformer...")

transformer = MiniMaxH3Transformer3DModel.from_pretrained(
    NVFP4_TRANSFORMER,
    dtype=torch.bfloat16,
    use_safetensors=False,
    local_files_only=True,
)

transformer.requires_grad_(False)
transformer.to(DEVICE_MAIN)

gc.collect()
torch.cuda.empty_cache()

print(
    "CUDA0 after transformer:",
    f"{torch.cuda.memory_allocated(0) / 1024**3:.1f} GiB",
)

# ------------------------------------------------------------------
# 2. Load Qwen3-VL as INT8 on GPU 1
# ------------------------------------------------------------------

text_quant = TransformersTorchAoConfig(
    Int8WeightOnlyConfig(version=2),
    modules_to_not_convert=[
        "model.visual",
        "model.language_model.embed_tokens",
        "model.language_model.norm",
        "lm_head",
    ],
)

print("Loading INT8 Qwen3-VL conditioner...")

text_encoder = Qwen3VLForConditionalGeneration.from_pretrained(
    MODEL,
    subfolder="text_encoder",
    dtype=torch.bfloat16,
    local_files_only=True,
    quantization_config=text_quant,
    low_cpu_mem_usage=True,
)

text_encoder.requires_grad_(False)
text_encoder.to(DEVICE_TEXT)

gc.collect()
torch.cuda.empty_cache()

print(
    "CUDA1 after text encoder:",
    f"{torch.cuda.memory_allocated(1) / 1024**3:.1f} GiB",
)

# ------------------------------------------------------------------
# 3. Resolve only the FL2VA workflow
# ------------------------------------------------------------------

base = ModularPipeline.from_pretrained(
    MODEL,
    workflow="fl2va",
    local_files_only=True,
)

blocks = base.blocks

# ------------------------------------------------------------------
# 4. Split conditioning onto GPU 1
#    GPU 1: before_encode + text_encoder
#    GPU 0: the remaining FL2VA blocks
# ------------------------------------------------------------------

condition_blocks = SequentialPipelineBlocks.from_blocks_dict(
    {
        "before_encode": blocks.sub_blocks.pop("before_encode"),
        "text_encoder": blocks.sub_blocks.pop("text_encoder"),
    }
)

conditioner = condition_blocks.init_pipeline(MODEL)
rest = blocks.init_pipeline(MODEL)

# ------------------------------------------------------------------
# 5. Reuse the already-loaded large components
# ------------------------------------------------------------------

conditioner.update_components(text_encoder=text_encoder)
rest.update_components(transformer=transformer)

# ------------------------------------------------------------------
# 6. Load the smaller support components
# ------------------------------------------------------------------

conditioner.load_components(dtype=torch.bfloat16, local_files_only=True)
rest.load_components(dtype=torch.bfloat16, local_files_only=True)

# Keep the VAEs with the generation side.
rest.vae.to(DEVICE_MAIN)
rest.audio_vae.to(DEVICE_MAIN)

gc.collect()
torch.cuda.empty_cache()

print("CUDA0 ready:", f"{torch.cuda.memory_allocated(0) / 1024**3:.1f} GiB")
print("CUDA1 ready:", f"{torch.cuda.memory_allocated(1) / 1024**3:.1f} GiB")

# ------------------------------------------------------------------
# 7. Run FL2VA conditioning once on GPU 1, then move state to GPU 0
# ------------------------------------------------------------------

image = load_image(IMAGE)

print("Running FL2VA conditioning on CUDA1...")

state = conditioner(
    prompt=PROMPT,
    image=image,
    height=HEIGHT,
    width=WIDTH,
)

state.set("prompt_embeds", state.prompt_embeds.to(DEVICE_MAIN))

# ------------------------------------------------------------------
# 8. Generate on GPU 0
# ------------------------------------------------------------------

result = rest(
    state=state,
    num_frames=NUM_FRAMES,
    num_inference_steps=STEPS,
    generator=torch.Generator(device="cpu").manual_seed(42),
    output=["videos", "audio", "sampling_rate"],
)

# ------------------------------------------------------------------
# 9. Save video + generated audio
# ------------------------------------------------------------------

encode_video(
    result["videos"][0],
    fps=24,
    output_path=OUTPUT,
    audio=result["audio"][0],
    audio_sample_rate=result["sampling_rate"],
)

print(f"Saved: {OUTPUT}")

This example is intentionally stripped down to the pieces that demonstrate the memory split. It drops what a production script might add — a loop over several generations, per-run random seeds, first + last-frame conditioning, output run-numbering, and custom scheduler shifts — none of which change how the two cards are divided.

If you don’t have the NVFP4 transformer

The script above reuses a transformer that was already converted to NVFP4. You do not need to reproduce that conversion. Current Diffusers documents a two-48-GB path that instead quantizes both large BF16 components to INT8 and then splits them across the cards — the more portable, fully supported recipe. Our NVFP4 + INT8 setup is included because it was actually run on the local Blackwell system and reduced the generation side enough to sit comfortably below 48 GB; it is a first-party tested configuration, not the upstream default.

Lower-memory GPUs

If you have neither an 80–96 GB card nor two 48 GB cards, current Diffusers documents progressively more aggressive single-GPU recipes. These are memory enablers, not performance optimizations — they make H3 runnable by streaming weights between CPU and GPU, at a real speed cost we have not benchmarked here.

24–32 GB

• INT8 weight-only quantization for the transformer and Qwen3-VL;
• block-level group offload for the transformer, leaf-level for Qwen3-VL;
• VAEs kept on the GPU.

12–16 GB

The same recipe pushed lower: additionally group-offload the video VAE and use a smaller canvas such as 960×544. Expect a large host-memory requirement — the current documentation calls for around 75 GB of system RAM to hold the INT8 weights.

For the exact, current group-offload code, follow the Diffusers H3 memory documentation (linked below) rather than copying a snapshot here — that code moves quickly.

Switching workflows

The three workflows differ mostly in which inputs they take and which transformer partition they load:

T2VA and FL2VA share the same transformer/ partition, which is exactly why FL2VA is a good primary example: it shows text conditioning, image conditioning, video and audio output, and the same checkpoint T2VA uses. Adding last_image=... to an FL2VA call turns it into a first + last frame constraint with no other changes.

REF2VA is more different. It uses the separate transformer_ref/ checkpoint and takes an ordered list of reference objects (MiniMaxH3ImageReference, MiniMaxH3VideoReference, MiniMaxH3AudioReference). The practical consequence for the setup above: our custom NVFP4 transformer is an FL2VA/T2VA partition, so it cannot simply be substituted into a REF2VA run — an equivalent optimized REF2VA path would need its own converted transformer_ref/ checkpoint. Use REF2VA when you need to drive the result from multiple ordered image/video/audio references rather than a single keyframe; for a first local setup, FL2VA covers most of the same ground with less machinery.

Resolution and duration

For the current Diffusers integration:

24 fps
5–15 second generation range
width and height both multiples of 32

The tested examples use 960×544, 124 frames (about a 5-second clip), and 30 inference steps. The integration requires both dimensions to be multiples of 32 (the effective spatial downsampling is 32×), so keep them ÷32-clean; the Video Resolution Calculator has a MiniMax H3 preset that lists exactly those sizes for your chosen aspect ratio.

Important limitations

• Modular Diffusers only. Do not reach for a generic DiffusionPipeline example — it does not describe the current H3 integration.
• Not yet in a stable release. At the time this guide was verified, upstream documentation still described installing Diffusers from the H3 pull request rather than a normal release. Follow the install steps in the linked Diffusers H3 docs and record the exact commit or version you built — do not treat any one-liner as permanent.
• Guidance-distilled. There is no negative_prompt or guidance_scale to tune; do not port those from other video models.
• Version-sensitive memory. The GB figures here are observations for a specific software stack, resolution, and component placement, not fixed requirements.

Practical recommendation

If you are choosing hardware specifically for H3:

• 80–96 GB single GPU — simplest practical high-memory workflow with CPU offload.
• 2×48 GB — excellent practical target for an optimized split; in our two-GPU test neither side exceeded ~44 GB.
• 2×80 GB — the easiest full-BF16 multi-GPU split, keeping both halves resident without ongoing host eviction.
• 24–32 GB — possible with INT8 and heavy group offload; slower.
• 12–16 GB — possible with still more offload and a smaller canvas, but system RAM becomes critical (~75 GB).

For the tested local setup, the most interesting balance was the two-GPU NVFP4 + INT8 split: it kept both large components resident and stayed under 48 GB on each card.

Sources and upstream documentation

• Diffusers — MiniMax H3 documentation — ModularPipeline-only integration, workflow names, component sizes, the 80 GB / two-48-GB / two-80-GB and 24–32 / 12–16 GB memory recipes, and resolution/duration guidance.
• Diffusers — H3 modular pipeline source — current workflow behaviour, guidance-distilled semantics, and frame/audio pipeline implementation.
• Diffusers — H3 workflow blocks — T2VA vs FL2VA vs REF2VA selection, transformer/ vs transformer_ref/, keyframe and reference block behaviour.
• Diffusers — H3 reference objects — MiniMaxH3ImageReference / VideoReference / AudioReference and REF2VA ordered-reference behaviour.
• MiniMax on Hugging Face — official model repositories and cards (MiniMax-H3).

The two-GPU NVFP4 + INT8 configuration is first-party material: tested by How Quant on the local Blackwell system, not an upstream Diffusers recipe.