Performance drop with 1 batch size per GPU

[zx-bit]

Excellent work! But I encountered an issue while trying to reproduce your results. When utilizing 4 GPUs with a batch size of 2 per GPU, I successfully replicated the results, about 39 mIoU. Nevertheless, when employing 8 GPUs with a batch size of 1 per GPU, it yielded significantly inferior results, about 20+ mIoU. The only change I made was replacing BatchNorm with SyncBatchNorm. What might be the underlying cause of this problem, and how could it be resolved?

[chenchen235]

Is the learning rate adjusted accordingly?

[zx-bit]

Is the learning rate adjusted accordingly?

I tried reducing the learning rate by half, from 2e-4 to 1e-4, but the performance is still poor.

[chenchen235]

Is the learning rate adjusted accordingly?学习率是否相应调整？

I tried reducing the learning rate by half, from 2e-4 to 1e-4, but the performance is still poor.我尝试将学习率降低一半，从 2e-4 降低到 1e-4，但性能仍然很差。

My swin-related models are also single-batch per GPU, and there is no significant drop in accuracy. Maybe you need to check if there are any other changes.