Installation and Setup

[DelaramRajaei]

This issue is where I will record any problems or issues I come across during the installation process of RePair project.

[DelaramRajaei]

I reviewed the installation manual and started by creating a new environment from the environment.yml file. An error occurred because of faiss-cpu, leading to ignoring all other libraries listed in the pip section.

faiss-gpu didn't install using pip installation. To address the installation problem, I resolved it by running the following code:

conda install -c conda-forge faiss-gpu

Accordingly, this is the command for the CPU:

conda install -c conda-forge faiss-cpu

For all the others libraries I used pip. For example: installing t5 using pip:

pip install t5

I'm currently encountering a few bugs and working on resolving them.

[DelaramRajaei]

I've managed to install the RePair project successfully. I obtained all the required files for the T5 model and ran the project using t5model: small.local.

While working with the main.py file, I encountered issues with reading the original file path, which required some minor code adjustments. Additionally, to address library conflicts, I made substitutions for certain commands, such as replacing progress_apply() with apply().

This is the log file generated after successfully running the program.
log_first_run.txt

[hosseinfani]

Hi @DelaramRajaei , thanks for the update. can we keep progress_apply() as it shows the progress when we have big runs.

[DelaramRajaei]

@hosseinfani
I'm going to figure out how to keep the progress_apply().

Right now, I've added the expanders and Query class, and I'm fixing any related issues and problems. I'm still facing bugs. After a successful run, I'll sort it out.

[yogeswarl]

Hey @DelaramRajaei, Let me know if you need any help. I have worked with the pipeline extensively,I can help you with a workaround.

[DelaramRajaei]

Hey @yogeswarl. Thank you for your offer. I'll reach out if I have any questions.

[tanshkoul]

Python 3.8 is deprecated. Using Python 3.9+ fixes w/ Pyserini.

[tanshkoul]

$ python -u main.py -data ../data/raw/robust04 -domain robust04
-------------------------------------------------- Executing the project with the robust04 dataset --------------------------------------------------
2025-02-13 15:08:21.417189: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI AVX512_BF16 AVX_VNNI AMX_TILE AMX_INT8 AMX_BF16 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-02-13 15:08:21.521083: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2025-02-13 15:08:22.072080: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2025-02-13 15:08:22.072152: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2025-02-13 15:08:22.072161: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
------------------------------ Ranking and evaluating by bm25 and map
Evaluating documents for query changes using map ...

Following breakage at eval stage. @hosseinfani

[hosseinfani]

Hi @tanshkoul
I texted you in our channel. I need to reproduce the error. are config files set as you when you see the error?

[hosseinfani]

Hi @tanshkoul
I had a second look at the stacktrace and the codeline. Seems it's just a warning by tensorflow. I think it does not break the pipeline as it goes to the ranking step. When you run it, do you see IR files created in the output folder?

[tanshkoul]

Hi @hosseinfani -- The IR files are not created in the output folder when I run the eval stage. Are we ignoring the tensorflow warning for now?