Some questions about the performance in the paper and training details.

[sdfsfdx]

Hi, thank you for open source! I executed train_base.py and have some questions as follows:

1. The performance on de dataset. I paste the results in the paper and highlight the values that correspond to train_base.py

I executed train_base.py and record the results as follows:

I noticed my GMF's NDCG@10 is far higher than that in the paper, and the rest 5 values are far lower. I did not change the content in train_base.py, what should I do to reproduce the models in the paper.

2. Whether the results in the paper are taken from the validation or the test? I set num_epoch as 175 and I found some metrics in the validation set are much higher than that in the paper, but the results of the test set are still lower.
3. If the results in the paper are taken from the test set, how did you set up the early stopping strategy? (i.e., the value of patience)
4. I did not find the description of the batch size in the paper, but it is 1024 in the file. Is the batch size in the paper 1024?

I hope you can reply to these questions that puzzle me when you have time, thank you very much!

[hamedrab]

Hi, thanks for the complete report on the issue. A few explanations points.

1. Our test and evaluation parts are taken out with the data we released due to a conflict with another work. We might be able to release those parts as well, though it is not clear when we can do this. As a result, the data that is processed here is to some good degree different from what we used for the paper.

2. In the time of running experiments for the paper, we were not able to get review dates, however, with this release, we are able to access review dates and sorted reviews based on date. Due to these, we are working (well, slowly) on releasing our numbers for the data that is used here. Please expect to see these changes in about two weeks.

3. Generally, we have seen that GMF numbers change significantly with different runs. However, NMF should be stable. I would recommend comparing NMF numbers.

4. For your other questions,

• re question 2. we used our test data for reporting. As mentioned on 1. there is some differences with the data here with the paper.
• re question 3. we didn't use early stopping. we set a fix num_epoch for every step as 50.
• re question 4. we used 1024 as the batch size.

Hope these explanations helped you. Please expect some changes to our codebase by early Feburary. There are a few small bugs with the code we released that will be fixed as well. Please let us know if you happen to see any other inconsistencies...

[sdfsfdx]

@hamedrab Thanks for your reply, looking forward to the new version!
If I find other problems, I will continue to list them here.

[miziha-zp]

@hamedrab thanks to your great dataset, I am so interested in your work. when does the version used in paper of your dataset come out?

[Charmve]

I am also looking forward to your update ...

My training test result:

I executed train_base.py for de and record the results, all log:

▶ python train_base.py
loading DATA/proc_data/de_5core.txt
loaded target data!
concatenating target and source data...
preparing test/valid data...
====> config['use_cuda']: False
model is GMF!
GMF(
  (embedding_user): Embedding(1852, 8)
  (embedding_item): Embedding(2180, 8)
  (affine_output): Linear(in_features=8, out_features=1, bias=True)
  (logistic): Sigmoid()
)
------------------------------------------------
Start to testing...
[pytrec_based] tgt_valid: 	 NDCG@10: 0.21813552287563903 	 R@10: 0.3490005402485143
[pytrec_based] tgt_test: 	 NDCG@10: 0.15258397792366213 	 R@10: 0.2614802809292274


====> config['use_cuda']: False
model is MLP!
loading pretrained gmf...
MLP(
  (embedding_user): Embedding(1852, 8)
  (embedding_item): Embedding(2180, 8)
  (fc_layers): ModuleList(
    (0): Linear(in_features=16, out_features=64, bias=True)
    (1): Linear(in_features=64, out_features=32, bias=True)
    (2): Linear(in_features=32, out_features=16, bias=True)
    (3): Linear(in_features=16, out_features=8, bias=True)
  )
  (affine_output): Linear(in_features=8, out_features=1, bias=True)
  (logistic): Sigmoid()
)
------------------------------------------------
Start to testing...
[pytrec_based] tgt_valid: 	 NDCG@10: 0.20339182863875818 	 R@10: 0.391139924365208
[pytrec_based] tgt_test: 	 NDCG@10: 0.1499018400244101 	 R@10: 0.2922744462452728


====> config['use_cuda']: False
model is NeuMF!
loading pretrained gmf and mlp...
NeuMF(
  (embedding_user_mlp): Embedding(1852, 8)
  (embedding_item_mlp): Embedding(2180, 8)
  (embedding_user_mf): Embedding(1852, 8)
  (embedding_item_mf): Embedding(2180, 8)
  (fc_layers): ModuleList(
    (0): Linear(in_features=16, out_features=64, bias=True)
    (1): Linear(in_features=64, out_features=32, bias=True)
    (2): Linear(in_features=32, out_features=16, bias=True)
    (3): Linear(in_features=16, out_features=8, bias=True)
  )
  (affine_output): Linear(in_features=16, out_features=1, bias=True)
  (logistic): Sigmoid()
)
------------------------------------------------
Start to testing...
[pytrec_based] tgt_valid: 	 NDCG@10: 0.22594107216002543 	 R@10: 0.4143706104808212
[pytrec_based] tgt_test: 	 NDCG@10: 0.16254332160838558 	 R@10: 0.31388438681793623


Experiment finished success!