Add more optimizers and losses

[Shashi456]

Just a small-ish roadmap to different Optimizers and losses we can look at to add :

Optimizers:-

• Adam
• Adagrad
• SGD
• RMSprop
• AdaDelta
  - [x] Riemann SGD Removed by [Optimizers] Removed the 'RiemannSGD' optimizer. #311
• Adabound ->ICLR 2019 -> Repo
• LBFGS
• AdaMax
• SparseAdam

Losses :-

• L1Loss
• L2Loss
• MeanSquaredError
• SoftmaxCrossEntropy
• SoftmaxCrossEntropyWithLogits
• SigmoidCrossEntropy
• MeanAbsoluteError
• MeanAbsolutePercentageError
• MeanSquaredLogarithmicError
• HingeLoss
• SquaredHinge
• CategoricalHinge
• Logcosh
• CategoricalCrossEntropy
• Kullback Leibler Divergence
  NegativeLogLikelihood
• Cosine
• TripletMarginLoss
• Poisson

Removed NegativeLogLikelihood. Check this discussion for more.

Note: If you have suggestions for losses and optimizers, please suggest them in this issue.

[rxwei]

Thanks for listing these!

[Shashi456]

@rxwei just a small roadmap, so that we have in mind what we need to add.

[liberium]

Maybe it would be better to organise this as a project?

[Shashi456]

@liberium. This is not priority right now. I wrote this task list so that we know what we have and what we don't. We are currently implementing all the layers. This is just a simple checklist so that we know where we are, and if someone wants to implement they can help us. Issues are a good way to inform people about open source contributions. In this case, a "good first issue"

[Shashi456]

@dan-zheng @rxwei so this Google group thread here talks about regularization. Although we are a little far from that stage right now.

One thing I was going to suggest him to look at was to retrieve the layer.weight iteratively, but then realized a lot of layers don't have the weight parameter.

I think one functionality we should look at adding is retrieving model weights. Is there an easy/hack-y way to add this ? Could we start looking at this I would definitely like to help in adding this functionality.

[eaplatanios]

@Shashi456 I believe that regularization schemes are heavily intertwined with the training abstractions, which do not yet exist in this repo. It would be nice to start a discussion around them at some point though.