Can this work on quantized models too?

[markg85]

Hi,

AMD user here, 7900XT 20GB vram. Just too little for most text-to-image models, unfortunately.
But running a quantized version (fp8 for example, like this) already works well but it's so slow.

Can cache-dit work on that too? And if so, how?

[DefTruth]

If you use the quantized model through the Difffusion Pipeline, cache-dit still supports the acceleration of the quantized model.

import cache_dit
from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("Clybius/FLUX.1-Krea-dev-scaled-fp8")

cache_dit.enable_cache(pipe)

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
image = pipe(prompt).images[0]

cache_dit.summary(pipe)

[markg85]

Ohh, that looks very promising @DefTruth!
However, i'm facing some difficulties apparently.

Do you have a working example perhaps? I would make it if i had the python skill and the skill in code for image generation but i unfortunately lack both. That makes and edits like black box magic for me.

Your example gives me this error:

Traceback (most recent call last):
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 409, in hf_raise_for_status
    response.raise_for_status()
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/Clybius/FLUX.1-Krea-dev-scaled-fp8/resolve/main/model_index.json

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/dockerx/test/test.py", line 4, in <module>
    pipe = DiffusionPipeline.from_pretrained("Clybius/FLUX.1-Krea-dev-scaled-fp8")
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/diffusers/pipelines/pipeline_utils.py", line 833, in from_pretrained
    cached_folder = cls.download(
                    ^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/diffusers/pipelines/pipeline_utils.py", line 1485, in download
    config_file = hf_hub_download(
                  ^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1010, in hf_hub_download
    return _hf_hub_download_to_cache_dir(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1073, in _hf_hub_download_to_cache_dir
    (url_to_download, etag, commit_hash, expected_size, xet_file_data, head_call_error) = _get_metadata_or_catch_error(
                                                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1546, in _get_metadata_or_catch_error
    metadata = get_hf_file_metadata(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 1463, in get_hf_file_metadata
    r = _request_wrapper(
        ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 286, in _request_wrapper
    response = _request_wrapper(
               ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/file_download.py", line 310, in _request_wrapper
    hf_raise_for_status(response)
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/huggingface_hub/utils/_http.py", line 420, in hf_raise_for_status
    raise _format(EntryNotFoundError, message, response) from e
huggingface_hub.errors.EntryNotFoundError: 404 Client Error. (Request ID: Root=1-68a71bbc-0c8cb3f342dfa742584ae8d4;1b0dea38-c879-42eb-9249-0870fdd5f7ce)

Entry Not Found for url: https://huggingface.co/Clybius/FLUX.1-Krea-dev-scaled-fp8/resolve/main/model_index.json.

Now i can read ;) So i tried to find a different safetensors repo with all these files. John6666/flux1-dev-fp8-flux seems to be structurally ok. From that repo it gives a whole load of output but this is probably the most important part:

WARNING 08-21 13:36:31 [cache_adapters.py:439] cache_context_kwargs is empty, use default cache options: {'cache_type': <CacheType.DBCache: 'Dual_Block_Cache'>, 'residual_diff_threshold': 0.12, 'warmup_steps': 8, 'max_cached_steps': -1, 'Fn_compute_blocks': 8, 'Bn_compute_blocks': 0, 'max_Fn_compute_blocks': 16, 'max_Bn_compute_blocks': 16, 'Fn_compute_blocks_ids': [], 'Bn_compute_blocks_ids': []}
WARNING 08-21 13:36:31 [flux.py:240] Patched Flux for cache-dit.
INFO 08-21 13:36:31 [cache_adapters.py:633] Match Block Forward Pattern: FluxTransformerBlock
INFO 08-21 13:36:31 [cache_adapters.py:633]  IN(['hidden_states', 'encoder_hidden_states'],
INFO 08-21 13:36:31 [cache_adapters.py:633] OUT(['encoder_hidden_states', 'hidden_states']))
Traceback (most recent call last):
  File "/dockerx/test/test.py", line 9, in <module>
    image = pipe(prompt).images[0]
            ^^^^^^^^^^^^
  File "/dockerx/cache-dit/src/cache_dit/cache_factory/cache_adapters.py", line 463, in new_call
    return original_call(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 802, in __call__
    ) = self.encode_prompt(
        ^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 367, in encode_prompt
    prompt_embeds = self._get_t5_prompt_embeds(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/diffusers/pipelines/flux/pipeline_flux.py", line 253, in _get_t5_prompt_embeds
    prompt_embeds = self.text_encoder_2(text_input_ids.to(device), output_hidden_states=False)[0]
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 1930, in forward
    encoder_outputs = self.encoder(
                      ^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.py", line 1040, in forward
    causal_mask = (1.0 - causal_mask) * torch.finfo(inputs_embeds.dtype).min
                   ~~~~^~~~~~~~~~~~~
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/_tensor.py", line 39, in wrapped
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/py_3.12/lib/python3.12/site-packages/torch/_tensor.py", line 1073, in __rsub__
    return _C._VariableFunctions.rsub(self, other)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: "add_stub" not implemented for 'Float8_e4m3fn'

Where the last few lines are the issue:

RuntimeError: "add_stub" not implemented for 'Float8_e4m3fn'

I have version 0.2.16.dev3

I do not even know where to begin looking for a solution for that. because nothing in the test even refers to Float8_e4m3fn, nothing in the stack trace does either. It's like an error thrown down upon your from a random place where you first have to figure out who even threw that error.

[DefTruth]

it seems not relate to cache-dit.