8-bit C-Optim optimizers #1430
Description
Feature request
Implementing C-optim (https://github.com/kyleliang919/C-Optim) for the supported optimizers, especially Adam and AdamW, optionally or as separate optimizers.
Motivation
C-optim is a new development in optimizer research, which very slightly tweaks any optimizer update rule with significant convergence improvements, both theoretical and empirical.
Your contribution
unfortunately, very unfamiliar with C development so will have a hard time implementing this myself