Rank-K-32B Integration [Draft]

[Yaohui2019]

Pull Request Checklist

Reference Issue

Please provide the reference to issue this PR is addressing (# followed by the issue number). If there is no associated issue, write "N/A".

ref: N/A

Checklist Items

Before submitting your pull request, please review these items:

• Have you followed the contributing guidelines?
• Have you verified that there are no existing Pull Requests for the same update/change?
• Have you updated any relevant documentation or added new tests where needed?

PR Type

What kind of change does this PR introduce?

• Bugfix
• Feature
• Code style update (formatting, local variables)
• Refactoring (no functional changes, no API changes)
• Documentation content changes
• Reproduction logs
• Other...
  • Description:

Description:

Integrating Rank-K-32B into Rank-LLM (https://github.com/hltcoe/rank-k)

Old PR

Current Status:
The current performance is close to that of the original authors. The nDCG@10 results for DL20 are 0.61 using BM25 and 0.82 using SPLADE++_EnsembleDistil_ONNX. The official nDCG@10 score is 0.643 using BM25.

What have I tried?

• Ran an experiment using the parse_ranking_more from the original repo and observed no significant change (around 0.05%)
• Ran an experiment truncating every document to 450 tokens and observed no changes at all

What could go wrong?

• Difference in calling the model
  • The official repo uses the vllm serve command and uses the OpenAI client to call the self-hosted model
  • I am pretty sure this is equivalent to loading the model using vllm.LLM and calling apply_chat_template before generation

Other Changes:

• Added a prompt template to follow the design pattern

The command I used is:

python src/rank_llm/scripts/run_rank_llm.py  --model_path='hltcoe/Rank-K-32B'  --top_k_candidates=20 --dataset=dl20 \
--retrieval_method=bm25  --context_size=4096 --print_prompts_responses  --variable_passages --num_gpus 2 --populate_invocations_history