Not able to reproduce using the uploaded model

[codeboy5]

Hey,

Can you please help on how to run the evaluation code using the model here?

I am using this command to run the evaluation.

python run_ddro_evaluation.py \ --epoch 10 \ --per_gpu_batch_size 4 \ --save_path "ddro_ms_tu_model_final.pkl" \ --log_path "final_result.log" \ --doc_file_path "resources/datasets/processed/msmarco-docs-sents.top.300k.json.gz" \ --pretrain_model_path resources/transformer_models/t5-base \ --docid_path resources/datasets/processed/msmarco-data/encoded_docid/url_docid.txt \ --test_file_path resources/datasets/processed/msmarco-data/eval_data_top_300k/query_dev.jsonl \ --dataset_script_dir resources/data/data_scripts \ --dataset_cache_dir resources/negs_tutorial_cache \ --num_beams 5 \ --add_doc_num 6144 \ --max_seq_length 64 \ --max_docid_length 100 \ --output_every_n_step 1000 \ --save_every_n_epoch 2 \ --operation "testing" \ --use_docid_rank True

[kidist-amde]

Hi @codeboy5 — thanks for reaching out! 🙌

If you’re using the released HF model, you can now evaluate with the new script I added here:
[src/pretrain/hf_eval/eval_hf_docid_ranking.py](https://github.com/kidist-amde/ddro/tree/main/src/pretrain/hf_eval)

---

Example: MS MARCO + URL-title docids

encoding="url_title" # Choose from: "url_title", "pq"
python src/pretrain/hf_eval/eval_hf_docid_ranking.py \
  --per_gpu_batch_size 4 \
  --log_path logs/msmarco/dpo_HF_url.log \
  --pretrain_model_path kiyam/ddro-msmarco-tu \
  --docid_path resources/datasets/processed/msmarco-data/encoded_docid/${encoding}_docid.txt \
  --test_file_path resources/datasets/processed/msmarco-data/eval_data/query_dev.${encoding}.jsonl \
  --dataset_script_dir src/data/data_scripts \
  --dataset_cache_dir ./cache \
  --num_beams 15 \
  --add_doc_num 6144 \
  --max_seq_length 64 \
  --max_docid_length 100 \
  --use_docid_rank True \
  --docid_format msmarco \
  --lookup_fallback True \
  --device cuda:0

---

What is --test_file_path?

The --test_file_path parameter should point to your evaluation dataset - typically query_dev.{encoding}.jsonl.

File Format:
This is a JSONL file where each line contains a query-document pair with:

• input_ids: tokenized query as integer IDs
• query_id: document identifier, e.g. "[d123]"
• doc_id: encoded docid string (comma-separated integers, usually ending with 1)

Generation:
Generate this file using:

python src/data/data_prep/build_t5_data/gen_eval_data_pipline.py --encoding "url_title"

Output Location:
The file will be created at:

resources/datasets/processed/msmarco-data/eval_data/query_dev.{encoding}.jsonl

Important: Ensure the --encoding parameter matches the encoding type used during training data generation ("url_title" or "pq").

---

Docid map (required)

Make sure you download the matching encoded docid file from the HF collection, e.g.:
[url_title_docid.txt](https://huggingface.co/collections/kiyam/ddro-generative-document-retrieval-680f63f2e9a72033598461c5)

If you’re using a different encoding (e.g., pq), adjust:

• --docid_path → path to that encoding
• --docid_format → set to msmarco for MS MARCO, nq for NQ

⚠️ Please note that the repository is still under development. I’ve verified this evaluation setup, and it should work as described, but parts of the codebase are not yet fully cleaned up or finalized. If you encounter any issues, feel free to reach out. I’ll be happy to clarify and update accordingly.