Skip to content

Add a new mode to module so accGradInput will override the gradient instead of accumulate #1524

@yiheng

Description

@yiheng
Contributor

Module has an accGradInput method, which accumulates grad into a tensor. So we need to zero that tensor at the beginning of each iteration.

This is not necessary if we only accumulate once in each iteration. We can provide a mode flag. If the user set that flag, the accumulating gradient will become overwrite the tensor, which can save the zeroGrad time in the training.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @yiheng

        Issue actions

          Add a new mode to module so accGradInput will override the gradient instead of accumulate · Issue #1524 · intel/ipex-llm