Add derivative tests for layers

[dan-zheng]

Layers currently lack derivative tests.

• Tensor operation derivatives are tested in Tests/TensorFlowTests/TensorAutoDiffTests.swift.
• Layers are tested in Tests/TensorFlowTests/LayerTests.swift. (Adding Test Cases for Layers in Layer.swift file #77)

Consider adding layer derivative tests to Tests/TensorFlowTests/LayerTests.swift, alongside layer "forward-pass" tests.

[Shashi456]

@dan-zheng might be the right time to make a LayerTests directory and have forwardpass and differentiation tests as files?

[jkrukowski]

@Shashi456 @dan-zheng just wanted to ask you, what's the preferred way of implementing derivative tests for layers. Here is one of the first attempts #443 I'm thinking to use pullback function for that, something like

// ...
let seed = Tensor<Float>([[1, 1]])
let backprop = pullback(at: input, layer) { $1($0) }
let (_, 𝛁layer) = backprop(seed)
// ...

[Shashi456]

The preferred way is at it is done for batch normalization in TensorAutoDiffTests.swift. You can check that here.

[jon-tow]

Layer Gradient Tests

• Sequential
• Conv1D
• Conv2D
• Conv3D
• DepthConv2D
• SeparableConv1D
• SeparableConv2D
• ZeroPadding1D
• ZeroPadding2D
• ZeroPadding3D
• MaxPool1D
• MaxPool2D
• MaxPool3D
• AvgPool1D
• AvgPool2D
• AvgPool3D
• GlobalAvgPool1D
• GlobalAvgPool2D
• GlobalAvgPool3D
• GlobalMaxPool1D
• GlobalMaxPool2D
• GlobalMaxPool3D
• UpSampling1D
• UpSampling2D
• UpSampling3D
• Reshape
• Flatten
• Embedding
• Dense
• RNN
• SimpleRNNCell
• LSTM
• Function
• BatchNorm
• LayerNorm

[rxwei]

Thanks for the checklist @jon-tow !

[Shashi456]

#443 #478 #470 #460 #469