HuggingFaceNmtEngine does not use SDPA attention implementation

[ddaspit]

The HuggingFaceNmEngine class sets output_attentions=True when calling the generate method on the model. This is not compatible with the SDPA attention implementation, so Huggingface reverts to using the default eager implementation. The class outputs the attentions so that they can be used to infer the alignment. A config option should be added to the class that allows the caller to disable inferring alignment from attention. This will allow Huggingface to use the SDPA attention implementation when it is configured.

[Enkidu93]

The HuggingFaceNmEngine class sets output_attentions=True when calling the generate method on the model. This is not compatible with the SDPA attention implementation, so Huggingface reverts to using the default eager implementation. The class outputs the attentions so that they can be used to infer the alignment. A config option should be added to the class that allows the caller to disable inferring alignment from attention. This will allow Huggingface to use the SDPA attention implementation when it is configured.

Is this causing the slow-downs observed in Serval jobs? I wondered about SDPA being the cause of the differences and searched for it in past issues only to find that you created this issue 😆.

[ddaspit]

I think this only affects inferencing, so it wouldn't explain slower training.

[Enkidu93]

I think this only affects inferencing, so it wouldn't explain slower training.

Oh, right - good point