Questions about reproducing EPInformer benchmarks and model checkpoints

[sqhang]

Dear EPInformer authors,

Thank you for sharing this great work and for maintaining the EPInformer codebase! I have a few questions regarding reproducing the benchmark results shown in Figure 2 of the paper.

1. Model checkpoints for reproducing Figure 2a

   I tried running inference using the RNA-seq model checkpoints from [EPInformer_v1_PE_Activity_HiC](https://github.com/pinellolab/EPInformer/tree/main/trained_models/expression_models/EPInformer_v1_PE_Activity_HiC), together with the input data from the Zenodo records: https://doi.org/10.5281/zenodo.13233337. My understanding is that this corresponds to the most advanced model (EPInformer-PE-Activity-HiC) described in the paper.

   However, when I evaluated the 12-fold cross-chromosome predictions, I obtained a Pearson correlation of around 0.670, which is much lower than the 0.863 reported in Figure 2a of the paper. In comparison, running inference using the RNA-seq model checkpoints from [EPInformer_v2_PE_Activity](https://github.com/pinellolab/EPInformer/tree/main/trained_models/expression_models/EPInformer_v2_PE_Activity) gives a Pearson correlation of around 0.857.

   This makes me wonder whether the currently released EPInformer_v1_PE_Activity_HiC checkpoints are the same checkpoints used for Figure 2a. Could you clarify this? If the latest checkpoints are not currently released, would it be possible to make them available?

2. Enformer and Borzoi fold splits/checkpoints

   The file [leave_chrom_out_crossvalidation_split_18377genes.csv](https://github.com/pinellolab/EPInformer/blob/main/data/leave_chrom_out_crossvalidation_split_18377genes.csv) appears to list the Enformer-style fold split used in the paper. Could you also provide the Borzoi fold split used in the paper?

   In addition, would it be possible to release the RNA-seq prediction model checkpoints trained using the Enformer fold split and the Borzoi fold split?

3. Hi-C datasets used for EPInformer_PE_Activity_HiC

   Could you clarify which Hi-C datasets were used for training and evaluating the EPInformer_PE_Activity_HiC model?

   In the Data Availability section, the paper mentions 4DN in-situ Hi-C data with accession 4DNFITUOMFUQ for K562 and 4DNFI1UEG1HD for GM12878. However, in [download_EPInformer_training_data.sh](https://github.com/pinellolab/EPInformer/blob/main/data/download_EPInformer_training_data.sh), the script appears to use ENCODE intact Hi-C with accession ENCFF318GOM for GM12878, and I did not see a corresponding row for K562.

   Could you confirm which Hi-C files were used for the reported benchmark results?

4. Seq-GraphReg benchmark numbers

   In the paper, you report Seq-GraphReg performance in the benchmark comparison. Was this performance obtained by running Seq-GraphReg inference directly, or was it taken from the previously reported numbers in the GraphReg paper?

   If the number was taken from the original paper, could you point me to the table or figure from which it was obtained?

Thank you very much for your help!

[JasonLinjc]

Thank you for raising these questions, and apologies for the delayed response.

We recommend using the updated pipeline workflow for training and evaluation. It is substantially faster while achieving performance comparable to our published results. The README includes the complete K562/GM12878 workflow and a guided Jupyter notebook.

For the Enformer and Borzoi comparisons, the chromosome-fold definitions are available from the original datasets: Genentech/enformer-data and Genentech/borzoi-data, respectively.

The updated pipeline uses released ENCODE Hi-C datasets where available. The original analysis used 4DN in-situ Hi-C datasets as described in the paper; in our evaluation, this difference did not substantially affect prediction performance. If a usable cell-specific Hi-C file is unavailable, the updated pipeline automatically falls back to the ENCODE/rE2G average Hi-C reference.

For Seq-GraphReg, we used the reported performance from the released Seq-GraphReg supplementary materials rather than rerunning Seq-GraphReg inference in our pipeline.

We are continuing to improve the documentation and appreciate the feedback!