Inference is slow when the classes is large.

@YuwenXiong 
When I testing on Mapillary vistas dataset, the classes is set to 66. The post_time is changed from 0.8s (19 classes on cityscapes) to 4.187. Log is shown as listed, 
```
upsnet_end2end_test.py | line 276: Batch 151/2000, data_time:0.017, net_time:0.134, post_time:4.187
```
Do you know the cause? Is that means the code can not deal with large numbers of classes?