[QUESTION]why f and g must conjucates each other?

[bescks]

Your question

According to my understanding, there are two facts:

1. the grad op of communication op is still communication(e.g. allreduce's grad is still allreduce, allgather's grad is reduce-scatter)
2. without considering the gradient allreduce in data parallel, if no communication op exists in forward, neither nor in backward.

So, why not use a single function g, whose both forward and backward operation are allreduce instead the two conjucation functions;

[github-actions[bot]]

Marking as stale. No activity in 60 days.