Dataset comprehensiveness

[Data-Designer]

Great resource, especially regarding the datasets. However, I would like to clarify:

1. In the Appendix, as you described, the complete datasets of MedQA, PubMedQA, MedBullets, MedExQA, MedXpertQA, and the MMLU-Medical set were used. But the authors only used a small subset of MMLU-Pro and MedMCQA, correct?

2. What criteria did the authors use for filtering the datasets?

[super-dainiu]

Hi @Data-Designer,

Thank you for your interest. Here is the reply to your questions.

1. We use the full sets to benchmark base models, but use a subset to evaluate agent frameworks.
2. We selected the hardest questions from each dataset (i.e., most base models failed to answer). The threshold is 5/10.