Would FlexAttention be useful over SDPA for masked language modeling?

[abdulfatir]

Thank you for the examples in this repo. I am wondering if FlexAttention would bring any value over SDPA in the case of masked language modeling? In masked language modeling, random input positions are masked and their contribution is ignored during softmax. However, since the masking is random, there's no structure to potentially exploit. Does FlexAttention make sense in this case?

[drisspg]

It seems less likely in terms of perf, but you may be able to get some memory savings. If you can come up w/ a compact way to represent your masking as opposed to a full attention_bias