SFAO uses gradient-subspace cosine similarity toselectively determine whether to project, accept, or discard each update, with a per-layer gating mechanism that can be tuned and efficiently approximated via Monte Carlo sampling. This design enables fine-grained control over the degree of alignment with past tasks while reducing unnecessary computation. Experiments on standard continual learning benchmarks demonstrate that SFAO improves accuracy, mitigates forgetting, and lowers energy consumption.
As neural networks are increasingly deployed in dynamic environments, they face the challenge of catastrophic forgetting, the tendency to overwrite previously learned knowledge when adapting to new tasks, resulting in severe performance degradation on earlier tasks. We propose Selective Forgetting-Aware Optimization (SFAO), a dynamic method that regulates gradient directions via cosine similarity and per-layer gating, enabling controlled forgetting while balancing plasticity and stability. SFAO selectively projects, accepts, or discards updates using a tunable mechanism with efficient Monte Carlo approximation. Experiments on standard continual learning benchmarks shows that SFAO significantly reduces memory overhead while maintaining comparable or superior accuracy, making it suitable for resource-constrained scenarios.