RuntimeError: CUDA error: device-side assert triggered #15
Description
2020-09-11 00:36:15,849 - mmdet - INFO - Start running, host: ni.jiang@hobot-job-203203-0, work_dir: /running_package/work_dirs/reppoints_v2_r50_fpn_1x_coco, hdfs_dir:hdfs://hobot-bigdata-aliyun/user/ni.jiang/reppoints/train_models/reppoints_v2-gt-r50-1x
2020-09-11 00:36:15,849 - mmdet - INFO - workflow: [('train', 1)], max: 12 epochs
2020-09-11 00:37:16,898 - mmdet - INFO - Epoch [1][50/6175] lr: 9.890e-04, eta: 1 day, 1:06:44, time: 1.221, data_time: 0.259, memory: 4945, loss_cls: 0.8208, loss_pts_init: 0.6830, loss_pts_refine: 0.7008, loss_heatmap: 0.7156, loss_offset: 0.4417, loss_sem: 0.1059, loss: 3.4678, grad_norm: 17.0503
2020-09-11 00:38:03,562 - mmdet - INFO - Epoch [1][100/6175] lr: 1.988e-03, eta: 22:08:24, time: 0.933, data_time: 0.009, memory: 4945, loss_cls: 0.3879, loss_pts_init: 0.2207, loss_pts_refine: 0.6300, loss_heatmap: 0.3815, loss_offset: 0.4177, loss_sem: 0.0373, loss: 2.0752, grad_norm: 16.6666
2020-09-11 00:38:50,347 - mmdet - INFO - Epoch [1][150/6175] lr: 2.987e-03, eta: 21:09:25, time: 0.936, data_time: 0.008, memory: 4945, loss_cls: 0.3399, loss_pts_init: 0.1843, loss_pts_refine: 0.5266, loss_heatmap: 0.3059, loss_offset: 0.4155, loss_sem: 0.0247, loss: 1.7969, grad_norm: 14.2407
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [64,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [65,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [66,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [67,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [68,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [69,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [70,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [71,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [36,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [37,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [38,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [39,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [41,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [42,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [43,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [52,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [53,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [32,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [0,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [4,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [5,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [6,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [7,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [11,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [12,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [13,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [14,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [15,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [16,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [17,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [18,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [19,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [20,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [21,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [22,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [23,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [24,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [25,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [30,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [31,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [96,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [97,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [98,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [99,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [100,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [101,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [102,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [103,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [104,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [105,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [106,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [107,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [108,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [109,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [110,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [111,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [112,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [113,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [114,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [115,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [116,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [117,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [118,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [119,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [120,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [121,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [122,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [123,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [124,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [1,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [2,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [3,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [4,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [5,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [6,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [58,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [90,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [91,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [92,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [93,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [94,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
/tmp/pip-req-build-4baxydiv/aten/src/ATen/native/cuda/IndexKernel.cu:60: lambda ->auto::operator()(int)->auto: block: [0,0,0], thread: [95,0,0] Assertion index >= -sizes[i] && index < sizes[i] && "index out of bounds" failed.
Traceback (most recent call last):
File "/running_package/./tools/train.py", line 161, in
main()
File "/running_package/./tools/train.py", line 157, in main
meta=meta)
File "/running_package/mmdet/apis/train.py", line 129, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 123, in run
epoch_runner(data_loaders[i], **kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train
**kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 36, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/running_package/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/running_package/mmdet/core/fp16/decorators.py", line 51, in new_func
return old_func(*args, **kwargs)
File "/running_package/mmdet/models/detectors/base.py", line 172, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/running_package/mmdet/models/detectors/reppoints_v2_detector.py", line 35, in forward_train
*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 861, in loss
label_channels=label_channels)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 585, in get_targets
unmap_outputs=unmap_outputs)
File "/running_package/mmdet/core/utils/misc.py", line 54, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 480, in _point_target_single
assign_result = assigner.assign(proposals, num_level_proposals_inside, gt_bboxes, gt_bboxes_ignore, gt_labels)
File "/running_package/mmdet/core/bbox/assigners/atss_assigner.py", line 145, in assign
-INF).t().contiguous().view(-1)
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /tmp/pip-req-build-4baxydiv/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x6d (0x7fb9649e5e7d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x21a83 (0x7fb986a16a83 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fb9649d122d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x10b6a9 (0x7fb985faf6a9 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x4b1abd (0x7fb986355abd in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x4b1af2 (0x7fb986355af2 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x1809da (0x55cf48c749da in /running_package/./open-mmlab/bin/python)
frame #7: + 0xfc039 (0x55cf48bf0039 in /running_package/./open-mmlab/bin/python)
frame #8: + 0xfa938 (0x55cf48bee938 in /running_package/./open-mmlab/bin/python)
frame #9: + 0xfa938 (0x55cf48bee938 in /running_package/./open-mmlab/bin/python)
frame #10: + 0xfa348 (0x55cf48bee348 in /running_package/./open-mmlab/bin/python)
frame #11: + 0xfadd8 (0x55cf48beedd8 in /running_package/./open-mmlab/bin/python)
frame #12: + 0xfadec (0x55cf48beedec in /running_package/./open-mmlab/bin/python)
frame #13: + 0xfadec (0x55cf48beedec in /running_package/./open-mmlab/bin/python)
frame #14: + 0xfadec (0x55cf48beedec in /running_package/./open-mmlab/bin/python)
frame #15: + 0xfadec (0x55cf48beedec in /running_package/./open-mmlab/bin/python)
frame #16: + 0xfadec (0x55cf48beedec in /running_package/./open-mmlab/bin/python)
frame #17: + 0x12adc7 (0x55cf48c1edc7 in /running_package/./open-mmlab/bin/python)
frame #18: PyDict_SetItemString + 0x89 (0x55cf48c2b889 in /running_package/./open-mmlab/bin/python)
frame #19: PyImport_Cleanup + 0x9c (0x55cf48ca080c in /running_package/./open-mmlab/bin/python)
frame #20: Py_FinalizeEx + 0x64 (0x55cf48d14d04 in /running_package/./open-mmlab/bin/python)
frame #21: + 0x23232e (0x55cf48d2632e in /running_package/./open-mmlab/bin/python)
frame #22: _Py_UnixMain + 0x3c (0x55cf48d2667c in /running_package/./open-mmlab/bin/python)
frame #23: __libc_start_main + 0xf5 (0x7fb9960bd495 in /usr/lib64/libc.so.6)
frame #24: + 0x1d7101 (0x55cf48ccb101 in /running_package/./open-mmlab/bin/python)
Traceback (most recent call last):
File "/running_package/./tools/train.py", line 161, in
main()
File "/running_package/./tools/train.py", line 157, in main
meta=meta)
File "/running_package/mmdet/apis/train.py", line 129, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 123, in run
epoch_runner(data_loaders[i], **kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train
**kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 36, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/running_package/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/running_package/mmdet/core/fp16/decorators.py", line 51, in new_func
return old_func(*args, **kwargs)
File "/running_package/mmdet/models/detectors/base.py", line 172, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/running_package/mmdet/models/detectors/reppoints_v2_detector.py", line 35, in forward_train
*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 861, in loss
label_channels=label_channels)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 585, in get_targets
unmap_outputs=unmap_outputs)
File "/running_package/mmdet/core/utils/misc.py", line 54, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 480, in _point_target_single
assign_result = assigner.assign(proposals, num_level_proposals_inside, gt_bboxes, gt_bboxes_ignore, gt_labels)
File "/running_package/mmdet/core/bbox/assigners/atss_assigner.py", line 126, in assign
candidate_idxs[:, gt_idx] += gt_idx * num_bboxes
RuntimeError: CUDA error: device-side assert triggered
Traceback (most recent call last):
File "/running_package/./tools/train.py", line 161, in
main()
File "/running_package/./tools/train.py", line 157, in main
meta=meta)
File "/running_package/mmdet/apis/train.py", line 129, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 123, in run
epoch_runner(data_loaders[i], **kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train
**kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 36, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/running_package/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/running_package/mmdet/core/fp16/decorators.py", line 51, in new_func
return old_func(*args, **kwargs)
File "/running_package/mmdet/models/detectors/base.py", line 172, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/running_package/mmdet/models/detectors/reppoints_v2_detector.py", line 35, in forward_train
*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 861, in loss
label_channels=label_channels)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 585, in get_targets
unmap_outputs=unmap_outputs)
File "/running_package/mmdet/core/utils/misc.py", line 54, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 480, in _point_target_single
assign_result = assigner.assign(proposals, num_level_proposals_inside, gt_bboxes, gt_bboxes_ignore, gt_labels)
File "/running_package/mmdet/core/bbox/assigners/atss_assigner.py", line 146, in assign
index = candidate_idxs.view(-1)[is_pos.view(-1)]
RuntimeError: copy_if failed to synchronize: device-side assert triggered
Traceback (most recent call last):
File "/running_package/./tools/train.py", line 161, in
main()
File "/running_package/./tools/train.py", line 157, in main
meta=meta)
File "/running_package/mmdet/apis/train.py", line 129, in train_detector
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 123, in run
epoch_runner(data_loaders[i], **kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/runner/epoch_based_runner.py", line 32, in train
**kwargs)
File "/running_package/open-mmlab/lib/python3.7/site-packages/mmcv/parallel/distributed.py", line 36, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/running_package/mmdet/models/detectors/base.py", line 237, in train_step
losses = self(**data)
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/module.py", line 541, in call
result = self.forward(*input, **kwargs)
File "/running_package/mmdet/core/fp16/decorators.py", line 51, in new_func
return old_func(*args, **kwargs)
File "/running_package/mmdet/models/detectors/base.py", line 172, in forward
return self.forward_train(img, img_metas, **kwargs)
File "/running_package/mmdet/models/detectors/reppoints_v2_detector.py", line 35, in forward_train
*loss_inputs, gt_bboxes_ignore=gt_bboxes_ignore)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 861, in loss
label_channels=label_channels)
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 585, in get_targets
unmap_outputs=unmap_outputs)
File "/running_package/mmdet/core/utils/misc.py", line 54, in multi_apply
return tuple(map(list, zip(*map_results)))
File "/running_package/mmdet/models/dense_heads/reppoints_v2_head.py", line 480, in point_target_single
assign_result = assigner.assign(proposals, num_level_proposals_inside, gt_bboxes, gt_bboxes_ignore, gt_labels)
File "/running_package/mmdet/core/bbox/assigners/atss_assigner.py", line 136, in assign
t = ep_bboxes_cy[candidate_idxs].view(-1, num_gt) - gt_bboxes[:, 1]
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /tmp/pip-req-build-4baxydiv/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x6d (0x7f0eceda2e7d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x21a83 (0x7f0ef0dd3a83 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f0eced8e22d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x10b6a9 (0x7f0ef036c6a9 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x4b1abd (0x7f0ef0712abd in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x4b1af2 (0x7f0ef0712af2 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x1809da (0x557dbc4a89da in /running_package/./open-mmlab/bin/python)
frame #7: + 0xfc039 (0x557dbc424039 in /running_package/./open-mmlab/bin/python)
frame #8: + 0xfa938 (0x557dbc422938 in /running_package/./open-mmlab/bin/python)
frame #9: + 0xfa938 (0x557dbc422938 in /running_package/./open-mmlab/bin/python)
frame #10: + 0xfa348 (0x557dbc422348 in /running_package/./open-mmlab/bin/python)
frame #11: + 0xfadd8 (0x557dbc422dd8 in /running_package/./open-mmlab/bin/python)
frame #12: + 0xfadec (0x557dbc422dec in /running_package/./open-mmlab/bin/python)
frame #13: + 0xfadec (0x557dbc422dec in /running_package/./open-mmlab/bin/python)
frame #14: + 0xfadec (0x557dbc422dec in /running_package/./open-mmlab/bin/python)
frame #15: + 0xfadec (0x557dbc422dec in /running_package/./open-mmlab/bin/python)
frame #16: + 0xfadec (0x557dbc422dec in /running_package/./open-mmlab/bin/python)
frame #17: + 0x12adc7 (0x557dbc452dc7 in /running_package/./open-mmlab/bin/python)
frame #18: PyDict_SetItemString + 0x89 (0x557dbc45f889 in /running_package/./open-mmlab/bin/python)
frame #19: PyImport_Cleanup + 0x9c (0x557dbc4d480c in /running_package/./open-mmlab/bin/python)
frame #20: Py_FinalizeEx + 0x64 (0x557dbc548d04 in /running_package/./open-mmlab/bin/python)
frame #21: + 0x23232e (0x557dbc55a32e in /running_package/./open-mmlab/bin/python)
frame #22: _Py_UnixMain + 0x3c (0x557dbc55a67c in /running_package/./open-mmlab/bin/python)
frame #23: __libc_start_main + 0xf5 (0x7f0f0047a495 in /usr/lib64/libc.so.6)
frame #24: + 0x1d7101 (0x557dbc4ff101 in /running_package/./open-mmlab/bin/python)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /tmp/pip-req-build-4baxydiv/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x6d (0x7f0e393a8e7d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x21a83 (0x7f0e5b3d9a83 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f0e3939422d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x10b6a9 (0x7f0e5a9726a9 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x4b1abd (0x7f0e5ad18abd in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x4b1af2 (0x7f0e5ad18af2 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x1809da (0x55d15f6809da in /running_package/./open-mmlab/bin/python)
frame #7: + 0xfc039 (0x55d15f5fc039 in /running_package/./open-mmlab/bin/python)
frame #8: + 0xfa938 (0x55d15f5fa938 in /running_package/./open-mmlab/bin/python)
frame #9: + 0xfa938 (0x55d15f5fa938 in /running_package/./open-mmlab/bin/python)
frame #10: + 0xfa348 (0x55d15f5fa348 in /running_package/./open-mmlab/bin/python)
frame #11: + 0xfadd8 (0x55d15f5fadd8 in /running_package/./open-mmlab/bin/python)
frame #12: + 0xfadec (0x55d15f5fadec in /running_package/./open-mmlab/bin/python)
frame #13: + 0xfadec (0x55d15f5fadec in /running_package/./open-mmlab/bin/python)
frame #14: + 0xfadec (0x55d15f5fadec in /running_package/./open-mmlab/bin/python)
frame #15: + 0xfadec (0x55d15f5fadec in /running_package/./open-mmlab/bin/python)
frame #16: + 0xfadec (0x55d15f5fadec in /running_package/./open-mmlab/bin/python)
frame #17: + 0x12adc7 (0x55d15f62adc7 in /running_package/./open-mmlab/bin/python)
frame #18: PyDict_SetItemString + 0x89 (0x55d15f637889 in /running_package/./open-mmlab/bin/python)
frame #19: PyImport_Cleanup + 0x9c (0x55d15f6ac80c in /running_package/./open-mmlab/bin/python)
frame #20: Py_FinalizeEx + 0x64 (0x55d15f720d04 in /running_package/./open-mmlab/bin/python)
frame #21: + 0x23232e (0x55d15f73232e in /running_package/./open-mmlab/bin/python)
frame #22: _Py_UnixMain + 0x3c (0x55d15f73267c in /running_package/./open-mmlab/bin/python)
frame #23: __libc_start_main + 0xf5 (0x7f0e6aa80495 in /usr/lib64/libc.so.6)
frame #24: + 0x1d7101 (0x55d15f6d7101 in /running_package/./open-mmlab/bin/python)
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /tmp/pip-req-build-4baxydiv/c10/cuda/CUDACachingAllocator.cpp:569)
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0x6d (0x7f2c1f713e7d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x21a83 (0x7f2c4172da83 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f2c1f6ff22d in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x10b6a9 (0x7f2c40cdd6a9 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #4: + 0x4b1abd (0x7f2c41083abd in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x4b1af2 (0x7f2c41083af2 in /running_package/open-mmlab/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x1809da (0x561b3bbaf9da in /running_package/./open-mmlab/bin/python)
frame #7: + 0xfc039 (0x561b3bb2b039 in /running_package/./open-mmlab/bin/python)
frame #8: + 0xfa938 (0x561b3bb29938 in /running_package/./open-mmlab/bin/python)
frame #9: + 0xfa938 (0x561b3bb29938 in /running_package/./open-mmlab/bin/python)
frame #10: + 0xfa348 (0x561b3bb29348 in /running_package/./open-mmlab/bin/python)
frame #11: + 0xfadd8 (0x561b3bb29dd8 in /running_package/./open-mmlab/bin/python)
frame #12: + 0xfadec (0x561b3bb29dec in /running_package/./open-mmlab/bin/python)
frame #13: + 0xfadec (0x561b3bb29dec in /running_package/./open-mmlab/bin/python)
frame #14: + 0xfadec (0x561b3bb29dec in /running_package/./open-mmlab/bin/python)
frame #15: + 0xfadec (0x561b3bb29dec in /running_package/./open-mmlab/bin/python)
frame #16: + 0xfadec (0x561b3bb29dec in /running_package/./open-mmlab/bin/python)
frame #17: + 0x12adc7 (0x561b3bb59dc7 in /running_package/./open-mmlab/bin/python)
frame #18: PyDict_SetItemString + 0x89 (0x561b3bb66889 in /running_package/./open-mmlab/bin/python)
frame #19: PyImport_Cleanup + 0x9c (0x561b3bbdb80c in /running_package/./open-mmlab/bin/python)
frame #20: Py_FinalizeEx + 0x64 (0x561b3bc4fd04 in /running_package/./open-mmlab/bin/python)
frame #21: + 0x23232e (0x561b3bc6132e in /running_package/./open-mmlab/bin/python)
frame #22: _Py_UnixMain + 0x3c (0x561b3bc6167c in /running_package/./open-mmlab/bin/python)
frame #23: __libc_start_main + 0xf5 (0x7f2c50e1a495 in /usr/lib64/libc.so.6)
frame #24: + 0x1d7101 (0x561b3bc06101 in /running_package/./open-mmlab/bin/python)
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.
Traceback (most recent call last):
File "/running_package/open-mmlab/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/running_package/open-mmlab/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 253, in
main()
File "/running_package/open-mmlab/lib/python3.7/site-packages/torch/distributed/launch.py", line 249, in main
cmd=cmd)