DeepCodeBench

All commands run from the DeepCodeBench/ directory unless noted.

For a deeper explanation of how answers are generated and how the LLM judge computes fact recall, see DeepCodeBench/ANSWER_AND_JUDGE.md.

Baselines

Baseline	Context
B2	PR title, body, and code blocks (PR-anchored)
B3	Repository URL only
B4 (BM25)	Top-k code chunks from repo via BM25 retrieval
B4 (CodeRankEmbed)	Top-k code chunks from repo via dense retrieval
B5 (BM25)	PR title/body + top-k PR blocks via BM25 (legacy script; see below)
B5 (CodeRankEmbed)	PR-level dense retrieval (separate scripts; see repo)

Answer scripts are split by backend:

HuggingFace: baseline2_answers_hf.py, baseline3_answers_hf.py, baseline4_answers_hf.py
OpenAI (sync or batch): baseline2_answers_openai.py, baseline3_answers_openai.py, baseline4_answers_openai.py

Outputs default to data/results/answers_out/<model_label>/ with filenames like <tag>.b2.jsonl, <tag>.b3.jsonl, <tag>.b4.jsonl.

Precomputation

PR contexts — required for B2 (and B5)

python pr/precompute_contexts.py --split test --out_dir data/cache/pr_contexts
python pr/precompute_contexts.py --split train --out_dir data/cache/pr_contexts

Repo chunk indexes — required for B4 (BM25 and CodeRankEmbed)

Chunks each repo at the relevant commit and stores them under data/cache/indexes/repo. Both B4 variants read from this cache.

python indexing/precompute_indexes.py --split test  --cache_root ./data/cache/indexes/repo --workdir ./data/cache/repos
python indexing/precompute_indexes.py --split train --cache_root ./data/cache/indexes/repo --workdir ./data/cache/repos

BM25 retrieval — required for B4 (BM25)

python bm25/precompute_retrieval.py --split test  --cache_root ./data/cache/indexes/repo --k 16 --out_jsonl data/results/bm25_retrieval_test.jsonl
python bm25/precompute_retrieval.py --split train --cache_root ./data/cache/indexes/repo --k 16 --out_jsonl data/results/bm25_retrieval_train.jsonl

PR chunk indexes + BM25 retrieval — required for B5 (BM25)

Chunks each PR’s blocks into data/cache/indexes/pr.

python indexing/precompute_pr_indexes.py --split test  --pr_contexts_dir data/cache/pr_contexts --cache_root ./data/cache/indexes/pr
python indexing/precompute_pr_indexes.py --split train --pr_contexts_dir data/cache/pr_contexts --cache_root ./data/cache/indexes/pr

python bm25/precompute_pr_retrieval.py --split test  --cache_root ./data/cache/indexes/pr --k 16 --out_jsonl data/results/pr_bm25_retrieval_test.jsonl
python bm25/precompute_pr_retrieval.py --split train --cache_root ./data/cache/indexes/pr --k 16 --out_jsonl data/results/pr_bm25_retrieval_train.jsonl

CodeRankEmbed retrieval — required for B4 (CodeRankEmbed)

# Requires chunks.jsonl + bm25.pkl from indexing/precompute_indexes.py (--cache_root must match).
# Use --overwrite when changing --top_k or rebuilding; otherwise the script resumes and skips existing ids (file stays stale).
python code_rank_embed/scripts/run_coderankembed_dcb.py --split test  --top_k 16 --out_jsonl data/results/coderankembed_retrieval_test.jsonl  --cache_root data/cache/indexes/repo --overwrite
python code_rank_embed/scripts/run_coderankembed_dcb.py --split train --top_k 16 --out_jsonl data/results/coderankembed_retrieval_train.jsonl --cache_root data/cache/indexes/repo --overwrite

Running baselines (HuggingFace)

B2 — PR full context

python baseline2_answers_hf.py \
  --answer_model <MODEL_PATH> \
  --split test --limit 232 --offset 0 \
  --pr_contexts_dir data/cache/pr_contexts \
  --out_dir data/results/answers_out/<MODEL_NAME> \
  --tag test_run

B3 — Repo only

python baseline3_answers_hf.py \
  --answer_model <MODEL_PATH> \
  --split test --limit 232 --offset 0 \
  --out_dir data/results/answers_out/<MODEL_NAME> \
  --tag test_run

B4 — BM25 or CodeRankEmbed RAG

B4 requires --cache_root and --retrieval_jsonl (BM25 or CodeRankEmbed JSONL).

python baseline4_answers_hf.py \
  --answer_model <MODEL_PATH> \
  --split test --limit 232 --offset 0 \
  --cache_root ./data/cache/indexes/repo \
  --retrieval_jsonl data/results/bm25_retrieval_test.jsonl \
  --out_dir data/results/answers_out/<MODEL_NAME> \
  --tag test_run_bm25

B5 — PR-BM25 RAG (legacy monolithic script)

python generate_baseline5_answers.py \
  --answer_model <MODEL_PATH> \
  --split test --limit 232 --offset 0 \
  --pr_contexts_dir data/cache/pr_contexts \
  --pr_bm25_cache_root ./data/cache/indexes/pr \
  --pr_bm25_retrieval_jsonl data/results/pr_bm25_retrieval_test.jsonl \
  --out_dir data/results/answers_out \
  --tag test_run

OpenAI Batch API experiments

All commands run from the DeepCodeBench/ directory. Precomputation is the same as for HuggingFace.

Setup

export OPENAI_API_KEY=sk-...

Default model is often gpt-5.3-codex in examples; outputs go under data/results/answers_out/<model_label>/.

Step 1 — Prepare batch inputs

Each OpenAI baseline script takes --batch_out PATH.jsonl (and writes PATH_meta.jsonl). You must also pass the mode flags required by each script (see baseline*_answers_openai.py).

B3 — Repo only

python baseline3_answers_openai.py \
  --split test --limit 232 --offset 0 \
  --batch_out data/batch_jobs/b3_test_batch_input.jsonl

B2 — PR full context

python baseline2_answers_openai.py \
  --split test --limit 232 --offset 0 \
  --pr_contexts_dir data/cache/pr_contexts \
  --batch_out data/batch_jobs/b2_test_batch_input.jsonl

B4 — BM25 or CodeRankEmbed RAG

python baseline4_answers_openai.py \
  --split test --limit 232 --offset 0 \
  --cache_root ./data/cache/indexes/repo \
  --retrieval_jsonl data/results/bm25_retrieval_test.jsonl \
  --batch_out data/batch_jobs/b4_bm25_test_batch_input.jsonl

Step 2 — Submit to OpenAI

python batch_api/submit_batch.py --input_file data/batch_jobs/b3_test_batch_input.jsonl

Step 3 — Collect results

python batch_api/collect_batch.py \
  --batch_id batch_abc123 \
  --out data/batch_jobs/b3_test_batch_output.jsonl

Step 4 — Parse answers

python batch_api/parse_results.py answers \
  --batch_output data/batch_jobs/b3_test_batch_output.jsonl \
  --meta_jsonl   data/batch_jobs/b3_test_batch_input_meta.jsonl \
  --out          data/results/answers_out/gpt-5.3-codex/test_run.b3.jsonl

Step 5 — Judge (batch)

python score_baselines.py \
  --answers_jsonl data/results/answers_out/gpt-5.3-codex/test_run.b3.jsonl \
  --batch_out data/batch_jobs/b3_test_judge_input.jsonl

Submit, collect, parse with parse_results.py scores (see batch_api/ and score_baselines.py help).

Step 6 — Aggregate fact recall

python eval_scripts/evaluate_fact_recall.py --scores_glob "data/results/scores_out/gpt-5.3-codex/*.scores.jsonl"

OpenAI synchronous API experiments

Use --openai_sync on the OpenAI baseline scripts for immediate answers (no 24h batch wait). For reasoning-capable models, cluster scripts often use --max_new_tokens 4096 because the Responses API uses one shared budget for reasoning and final text (not a fixed paper constant).

Example:

python baseline3_answers_openai.py \
  --split test --limit 232 --offset 0 \
  --openai_sync \
  --openai_model gpt-5.3-codex \
  --max_new_tokens 4096 \
  --tag test_run

Judge scores (synchronous)

python score_baselines.py \
  --answers_jsonl data/results/answers_out/gpt-5.3-codex/test_run.b3.jsonl \
  --openai_sync \
  --openai_model gpt-5.3-codex \
  --reasoning_effort low

Evaluation (HuggingFace judge)

Score answers (fact recall)

python score_baselines.py \
  --judge_model <MODEL_PATH> \
  --answers_jsonl data/results/answers_out/<MODEL_NAME>/<TAG>.b4.jsonl \
  --out_dir data/results/scores_out/<MODEL_NAME>

Aggregate fact recall

python eval_scripts/evaluate_fact_recall.py --scores_glob "data/results/scores_out/<MODEL_NAME>/*.scores.jsonl"

SLURM jobs

Submit from the repository root (hindsight/), not from DeepCodeBench/.

Script	Purpose
`jobs/precompute/run_precompute_pr_contexts.sbatch`	PR contexts
`jobs/precompute/run_precompute_indexes.sbatch`	Build repo indexes (`chunks.jsonl`, `bm25.pkl`)
`jobs/precompute/run_precompute_bm25_retrieval.sbatch`	Write repo BM25 retrieval JSONL (run after indexes exist)
`jobs/precompute/run_coderankembed_dcb.sbatch`	CodeRankEmbed retrieval
`jobs/baselines/run_b2.sbatch`	B2 HF answers
`jobs/baselines/run_b2_openai_sync.sbatch`	B2 OpenAI sync
`jobs/baselines/run_b3.sbatch`	B3 HF answers
`jobs/baselines/run_b3_openai_sync.sbatch`	B3 OpenAI sync
`jobs/baselines/run_b4_bm25.sbatch`	B4 BM25 HF answers
`jobs/baselines/run_b4_bm25_openai_sync.sbatch`	B4 BM25 OpenAI sync
`jobs/baselines/run_b4_coderankembed.sbatch`	B4 CodeRankEmbed HF answers
`jobs/baselines/run_b4_coderankembed_openai_sync.sbatch`	B4 CodeRankEmbed OpenAI sync
`jobs/evaluation/run_eval.sbatch`	Judge over answer files
`jobs/evaluation/run_eval_openai_sync.sbatch`	OpenAI sync judge
`jobs/evaluation/evaluate_fact_recall_dir.sbatch`	Aggregate fact recall
`jobs/evaluation/evaluate_quality_dir.sbatch`	Aggregate quality scores
`jobs/evaluation/evaluate_coderankembed_fact_recall.sbatch`	Aggregate fact recall (CodeRankEmbed layout)

sbatch jobs/baselines/run_b4_bm25.sbatch
squeue -u $USER
tail -f logs/<JOB_NAME>_<JOBID>.out
scancel <JOBID>

Older B5 jobs may live under jobs/old/ if present.

Name		Name	Last commit message	Last commit date
Latest commit History 161 Commits
CodeRepoQA		CodeRepoQA
DeepCodeBench		DeepCodeBench
LocAgent		LocAgent
QA/logs		QA/logs
final_report		final_report
jobs		jobs
.DS_Store		.DS_Store
.gitignore		.gitignore
README.md		README.md
README_LOCAGENT.md		README_LOCAGENT.md
environment_openai.yml		environment_openai.yml
requirements.txt		requirements.txt

Navigation Menu

Repository files navigation

DeepCodeBench

Baselines

Precomputation

PR contexts — required for B2 (and B5)

Repo chunk indexes — required for B4 (BM25 and CodeRankEmbed)

BM25 retrieval — required for B4 (BM25)

PR chunk indexes + BM25 retrieval — required for B5 (BM25)

CodeRankEmbed retrieval — required for B4 (CodeRankEmbed)

Running baselines (HuggingFace)

B2 — PR full context

B3 — Repo only

B4 — BM25 or CodeRankEmbed RAG

B5 — PR-BM25 RAG (legacy monolithic script)

OpenAI Batch API experiments

Setup

Step 1 — Prepare batch inputs

Step 2 — Submit to OpenAI

Step 3 — Collect results

Step 4 — Parse answers

Step 5 — Judge (batch)

Step 6 — Aggregate fact recall

OpenAI synchronous API experiments

Judge scores (synchronous)

Evaluation (HuggingFace judge)

Score answers (fact recall)

Aggregate fact recall

SLURM jobs

About

Releases

Packages

Contributors

Languages

Folders and files

Latest commit

History

Repository files navigation

DeepCodeBench

Baselines

Precomputation

PR contexts — required for B2 (and B5)

Repo chunk indexes — required for B4 (BM25 and CodeRankEmbed)

BM25 retrieval — required for B4 (BM25)

PR chunk indexes + BM25 retrieval — required for B5 (BM25)

CodeRankEmbed retrieval — required for B4 (CodeRankEmbed)

Running baselines (HuggingFace)

B2 — PR full context

B3 — Repo only

B4 — BM25 or CodeRankEmbed RAG

B5 — PR-BM25 RAG (legacy monolithic script)

OpenAI Batch API experiments

Setup

Step 1 — Prepare batch inputs

Step 2 — Submit to OpenAI

Step 3 — Collect results

Step 4 — Parse answers

Step 5 — Judge (batch)

Step 6 — Aggregate fact recall

OpenAI synchronous API experiments

Judge scores (synchronous)

Evaluation (HuggingFace judge)

Score answers (fact recall)

Aggregate fact recall

SLURM jobs

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages