How to reproduce results of the paper？ #96
Description
I read the paper and downloaded the AG news dataset，and tested PET model on it，but there is a great margin between my results and the author's results. I set parameters as below. To be specifically, I just used 10 examples for train(10 shot),model type is roberta,model_name_or_path is roberta-large,and I used all patterns for AG news. I did not change other parameters. And here is my results:
==============my results===============
acc-p0: 0.6450877192982456 +- 0.053859095516898825
acc-p1: 0.7874561403508772 +- 0.01841603941808791
acc-p2: 0.5642543859649123 +- 0.06912621607498706
acc-p3: 0.6119298245614034 +- 0.09528808314997761
acc-p4: 0.7537719298245614 +- 0.07473549078343446
acc-all-p: 0.6725 +- 0.10462149351553651
===============parameters setting===========
parser.add_argument("--train_examples", default=10, type=int,
help="The total number of train examples to use, where -1 equals all examples.")
parser.add_argument("--method", required=False, default='pet', choices=['pet', 'ipet', 'sequence_classifier'],
help="The training method to use. Either regular sequence classification, PET or iPET.")
parser.add_argument("--data_dir", default="./agnews/", type=str, required=False,
help="The input data dir. Should contain the data files for the task.")
parser.add_argument("--model_type", default="roberta", type=str, required=False, choices=MODEL_CLASSES.keys(),
help="The type of the pretrained language model to use")
parser.add_argument("--model_name_or_path", default="roberta-large", type=str, required=False,
help="Path to the pre-trained model or shortcut name")
parser.add_argument("--task_name", default="agnews", type=str, required=False, choices=PROCESSORS.keys(),
help="The name of the task to train/evaluate on")
parser.add_argument("--output_dir", default="./output/", type=str, required=False,
help="The output directory where the model predictions and checkpoints will be written")
# PET-specific optional parameters
parser.add_argument("--wrapper_type", default="mlm", choices=WRAPPER_TYPES,
help="The wrapper type. Set this to 'mlm' for a masked language model like BERT or to 'plm' "
"for a permuted language model like XLNet (only for PET)")
parser.add_argument("--pattern_ids", default=[0,1,2,3,4], type=int, nargs='+',
help="The ids of the PVPs to be used (only for PET)")