Optimum 2.0.0: Bettertransformer deprecated

[noobHappylife]

System Info

When install with pip install infinity[all] it will install optimum 2.0.0. Resulting in the following error when using the infinity cli.

infinity_emb v2 --help
Traceback (most recent call last):
  File "/media/data4/yh/uvenv/infinity077/bin/infinity_emb", line 4, in <module>
    from infinity_emb.cli import cli
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/__init__.py", line 7, in <module>
    from infinity_emb.engine import AsyncEmbeddingEngine, AsyncEngineArray  # noqa: E402
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/engine.py", line 11, in <module>
    from infinity_emb.inference import (
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/inference/__init__.py", line 4, in <module>
    from infinity_emb.inference.batch_handler import BatchHandler
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/inference/batch_handler.py", line 39, in <module>
    from infinity_emb.transformer.utils import get_lengths_with_tokenize
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/transformer/utils.py", line 9, in <module>
    from infinity_emb.transformer.classifier.torch import SentenceClassifier
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/transformer/classifier/torch.py", line 8, in <module>
    from infinity_emb.transformer.acceleration import (
  File "/media/data4/yh/uvenv/infinity077/lib/python3.12/site-packages/infinity_emb/transformer/acceleration.py", line 11, in <module>
    from optimum.bettertransformer import (  # type: ignore[import-untyped]
ModuleNotFoundError: No module named 'optimum.bettertransformer'

Optimum installed:

pip freeze | grep optimum
optimum==2.0.0

Checking the optimum released notes, https://github.com/huggingface/optimum/releases/tag/v2.0.0 bettertransformer has been deprecated and removed.

Information

• Docker + cli
• pip + cli
• pip + usage of Python interface

Tasks

• An officially supported CLI command
• My own modifications

Reproduction

Installed with

pip install infinity-emb[all]

Failed to run cli as mentioned above.

[Learner-feng]

me too

[wirthual]

There is an experimental PR #641 which should work around this issue.

Can you try it out and report back if it works for you?

git clone -b better_transformer_clone https://github.com/michaelfeil/infinity.git
cd infinity/libs/infinity_emb/
pip install -e ".[all]"

[noobHappylife]

I tried your branch and see a different error.

infinity_emb v2 --help
/home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/torch/onnx/_internal/registration.py:159: OnnxExporterWarning: Symbolic function 'aten::scaled_dot_product_attention' already registered for opset 14. Replacing the existing function with new function. This is unexpected. Please report it on https://github.com/pytorch/pytorch/issues.
  warnings.warn(
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ /home/akk/micromamba/envs/infinity077/bin/infinity_emb:7 in <module>                                                                                                                                                                                                                 │
│                                                                                                                                                                                                                                                                                      │
│   4 if __name__ == '__main__':                                                               ╭──────────── locals ─────────────╮                                                                                                                                                     │
│   5 │   if sys.argv[0].endswith('.exe'):                                                     │ sys = <module 'sys' (built-in)> │                                                                                                                                                     │
│   6 │   │   sys.argv[0] = sys.argv[0][:-4]                                                   ╰─────────────────────────────────╯                                                                                                                                                     │
│ ❱ 7 │   sys.exit(cli())                                                                                                                                                                                                                                                              │
│   8                                                                                                                                                                                                                                                                                  │
│                                                                                                                                                                                                                                                                                      │
│ /media/data4/yh/git/infinity/libs/infinity_emb/infinity_emb/cli.py:413 in cli                                                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                      │
│   410 │   │   │   )                                                                                                                                                                                                                                                                  │
│   411 │   │   │                                                                                                                                                                                                                                                                      │
│   412 │   │   │   sys.argv.insert(1, "v2")                                                                                                                                                                                                                                           │
│ ❱ 413 │   tp()                                                                                                                                                                                                                                                                       │
│   414                                                                                                                                                                                                                                                                                │
│   415                                                                                                                                                                                                                                                                                │
│   416 if __name__ == "__main__":                                                                                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:338 in __call__                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:321 in __call__                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:360 in get_command                                                                                                                                                                                  │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:342 in get_group                                                                                                                                                                                    │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:496 in get_group_from_info                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:590 in get_command_from_info                                                                                                                                                                        │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:566 in get_params_convertors_ctx_param_name_from_function                                                                                                                                           │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/main.py:900 in get_click_param                                                                                                                                                                              │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/typer/core.py:427 in __init__                                                                                                                                                                                     │
│                                                                                                                                                                                                                                                                                      │
│ /home/akk/micromamba/envs/infinity077/lib/python3.12/site-packages/click/core.py:2793 in __init__                                                                                                                                                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Secondary flag is not valid for non-boolean flag.

[wirthual]

Thanks for trying, can you try to downgrade click:

pip install click==8.1.8

[noobHappylife]

Yes, with click==8.1.8 it works. Thank you.