PINN loss and DRM loss help getting rid of local minima by gradient selection

[stevengogogo]

Idea: Gradient belief to help getting out local minima via mutual global minima

Both PINN loss and DRM loss lead to the same solution. The major task we want is to avoid local minima. Thus, we should trust the large gradient than the small one if one of loss in local minima.

Does PINN loss and DRM loss has same local mimia? We only know they have same global minima

Strategy:
grad1 and grad2 can be applied to different optimizer

1. Max(norm(grad1), norm(grad2))
2. Record grad1 and grad2 to check if they are in local minima. Belief the one that is not. This minimize the noise in SGD