question about standardization

[yuffon]

I find that in the training and testing phase, the dataset is standardized as a whole batch, including computing the mean and variance from the entire cifar10 dataset.
However, when the model is deployed, the image is feed individually, how should we preprocess the image?
What mean and variance should we use when the input image is of the same category but not included in the cifar10 dataset?

[liuzhuang13]

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

[yuffon]

Hi @yuffon I think we just subtract the training means and stds in all testing, this is standard in machine learning.

Yes, this is common in ML. What if the input image is not from the 10 categories in cifar10 dataset?
If I want to do transfer learning or other tasks, how should I preprocess the data?
I think it is more convenient with per image standardization.
So I replaced the whole-batch standardization with per image standardization, the acc doesn't converge, untill that I add an BN layer without scale and shift variables at the start point of the densenet. I have not checked the final accuracy results in different scenarios.

[liuzhuang13]

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

[yuffon]

In TensorFlow's official tutorial Resnet repository, they use per image standardization.
https://github.com/tensorflow/models/tree/master/official/resnet

[yuffon]

I don't think one should preprocess the images by its own mean and stds, that causes different input to be changed by different amounts.

CIFAR is not a good dataset to transfer from, I think if you do transfer learning it's better to start with ImageNet models. In this case, you still subtract imageNet training data's mean and stds.

If I make a new network that trained on cifar10, but I want to test the model on image out of the cifar10 dataset and observe the network behavior on the out-of-distribution data. How should I normalize the data?

[liuzhuang13]

I think the common practice is to just normalize the data as the way it is normalized during training (and using the training stats).