Issues with evaluating multilearn models using fit-evaluate or crossvalidation

[capriceZ]

Thank you for maintaining this library! It has been super helpful!

I was testing individualized BKT by setting multilearn to student_id. My understanding is this model setup will estimate separate learn rates for each student. Therefore, when we split data into train and test at the student level (i.e., train and test include data for different students), we cannot predict/evaluate the trained model with the test data simply because we have no estimated learn rate parameters for students in the test data set. This is actually the error I got "Learn rate x not fitted".

However, when I use model.crossvalidate with multilearn set to student_id, I DID get an auc metric without any errors. I thought the default crossvalidation is also splitting train and test data at the student level. So, I would expect such crossvalidation to fail due to the same reason above.

Can you help me understand what happened?

Thank you!

[abadrinath947]

As I recall, we will construct "best effort" predictions (0.5) if we are unsure of the parameter value during cross-validation to prevent errors. As of now, the behaviour is not entirely consistent between fit + predict and crossvalidate for this specific case, but you will notice that during fit + predict, we will make similar "best effort" predictions for skills we have not trained on.

I can take a closer look at this and see if we can make the behaviour more consistent or output some kind of warning. Let me know if you have any suggestions or ideas.