RuntimeError when training

[ianz27]

I only change the config file (free_anchor_R-50-FPN_test.txt), and got the error

2020-01-16 15:49:05,738 maskrcnn_benchmark.trainer INFO: eta: 3:46:57 iter: 244400 loss: 1.7938 (1.8977) loss_retina_positive: 1.6451 (1.7404) loss_retina_negative: 0.1402 (0.1573) time: 0.1097 (0.1178) data: 0.0042 (0.0045) lr: 0.010000 max mem: 1404
2020-01-16 15:49:08,161 maskrcnn_benchmark.trainer INFO: eta: 3:46:55 iter: 244420 loss: 1.7646 (1.8977) loss_retina_positive: 1.6248 (1.7404) loss_retina_negative: 0.1239 (0.1573) time: 0.1109 (0.1178) data: 0.0041 (0.0045) lr: 0.010000 max mem: 1404
2020-01-16 15:49:10,560 maskrcnn_benchmark.trainer INFO: eta: 3:46:52 iter: 244440 loss: 1.8001 (1.8977) loss_retina_positive: 1.6412 (1.7404) loss_retina_negative: 0.1554 (0.1573) time: 0.1126 (0.1178) data: 0.0040 (0.0045) lr: 0.010000 max mem: 1404
2020-01-16 15:49:12,817 maskrcnn_benchmark.trainer INFO: eta: 3:46:50 iter: 244460 loss: 1.7907 (1.8977) loss_retina_positive: 1.6191 (1.7404) loss_retina_negative: 0.1470 (0.1573) time: 0.1076 (0.1178) data: 0.0037 (0.0045) lr: 0.010000 max mem: 1404
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=59 : device-side assert triggered
/opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [2887,0,0], thread: [16,0,0] Assertion *input >= 0. && *input <= 1. failed.
Traceback (most recent call last):
File "tools/train_net.py", line 171, in
main()
File "tools/train_net.py", line 164, in main
model = train(cfg, args.local_rank, args.distributed)
File "tools/train_net.py", line 73, in train
arguments,
File "/home/zz/work/FreeAnchor/maskrcnn_benchmark/engine/trainer.py", line 70, in do_train
loss_dict_reduced = reduce_loss_dict(loss_dict)
File "/home/zz/work/FreeAnchor/maskrcnn_benchmark/engine/trainer.py", line 28, in reduce_loss_dict
all_losses = torch.stack(all_losses, dim=0)
RuntimeError: cuda runtime error (59) : device-side assert triggered at /opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THC/THCCachingHostAllocator.cpp:265
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /opt/conda/conda-bld/pytorch_1556653215914/work/c10/cuda/CUDACachingAllocator.cpp:564)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7fcb2ed3fdc5 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x14792 (0x7fcb2bc1c792 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x50 (0x7fcb2ed2f640 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x3067fb (0x7fcb2c33c7fb in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
frame #4: + 0x14019b (0x7fcb54b2019b in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x3bfc84 (0x7fcb54d9fc84 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3bfcd1 (0x7fcb54d9fcd1 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x19dfce (0x56446760afce in /home/zz/anaconda3/envs/fa/bin/python)
frame #8: + 0x113a6b (0x564467580a6b in /home/zz/anaconda3/envs/fa/bin/python)
frame #9: + 0x103948 (0x564467570948 in /home/zz/anaconda3/envs/fa/bin/python)
frame #10: + 0x114267 (0x564467581267 in /home/zz/anaconda3/envs/fa/bin/python)
frame #11: + 0x11427d (0x56446758127d in /home/zz/anaconda3/envs/fa/bin/python)
frame #12: + 0x11427d (0x56446758127d in /home/zz/anaconda3/envs/fa/bin/python)
frame #13: + 0x11427d (0x56446758127d in /home/zz/anaconda3/envs/fa/bin/python)
frame #14: PyDict_SetItem + 0x502 (0x5644675cc602 in /home/zz/anaconda3/envs/fa/bin/python)
frame #15: PyDict_SetItemString + 0x4f (0x5644675cd0cf in /home/zz/anaconda3/envs/fa/bin/python)
frame #16: PyImport_Cleanup + 0x9e (0x56446760c91e in /home/zz/anaconda3/envs/fa/bin/python)
frame #17: Py_FinalizeEx + 0x67 (0x564467682367 in /home/zz/anaconda3/envs/fa/bin/python)
frame #18: + 0x227d93 (0x564467694d93 in /home/zz/anaconda3/envs/fa/bin/python)
frame #19: _Py_UnixMain + 0x3c (0x5644676950bc in /home/zz/anaconda3/envs/fa/bin/python)
frame #20: __libc_start_main + 0xe7 (0x7fcb651ccb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: + 0x1d0990 (0x56446763d990 in /home/zz/anaconda3/envs/fa/bin/python)

free_anchor_R-50-FPN_test.txt

[zhangxiaosong18]

I think the problem may be in the INPUT setting, it's best to use a scale that keeps the horizontal/vertical ratio in object detection, so I recommend using (MIN_SIZE=800,MAX_SIZE=1333), or a lower resolution (MIN_SIZE=600, MAX_SIZE=833). I want to know why you want to change these parameters, is it the memory limit?

[ianz27]

Thank you for your reply.
In my application, the input size is 512x512. I am using mmdetection to train my dataset and want to test the performace of FreeAnchor. I can get better performace in my dataset BUT is seems the training process is unstable. For example, the toal epoch is 60, the error occured in epoch17 and epoch 39, I have to resume the training process when the error occured (open-mmlab/mmdetection#1989), so I train your original FreeAnchor and found similar error.

[ianz27]

@zhangxiaosong18 I change the config file like (MIN_SIZE=600, MAX_SIZE=833), and still got error.

2020-01-19 05:29:33,311 maskrcnn_benchmark.trainer INFO: eta: 1:48:16 iter: 319500 loss: 1.6501 (2.0420) loss_retina_positive: 1.4884 (1.8823) loss_retina_negative: 0.1492 (0.1597) time: 0.1532 (0.1604) data: 0.0056 (0.0063) lr: 0.000100 max mem: 2270
2020-01-19 05:29:36,486 maskrcnn_benchmark.trainer INFO: eta: 1:48:13 iter: 319520 loss: 1.6764 (2.0420) loss_retina_positive: 1.5364 (1.8823) loss_retina_negative: 0.1335 (0.1597) time: 0.1517 (0.1604) data: 0.0059 (0.0063) lr: 0.000100 max mem: 2270
2020-01-19 05:29:39,661 maskrcnn_benchmark.trainer INFO: eta: 1:48:10 iter: 319540 loss: 1.6727 (2.0420) loss_retina_positive: 1.4854 (1.8823) loss_retina_negative: 0.1593 (0.1597) time: 0.1542 (0.1604) data: 0.0054 (0.0063) lr: 0.000100 max mem: 2270
2020-01-19 05:29:42,879 maskrcnn_benchmark.trainer INFO: eta: 1:48:07 iter: 319560 loss: 1.6427 (2.0419) loss_retina_positive: 1.4790 (1.8822) loss_retina_negative: 0.1520 (0.1597) time: 0.1535 (0.1604) data: 0.0056 (0.0063) lr: 0.000100 max mem: 2270
/opt/conda/conda-bld/pytorch_1556653215914/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [16266,0,0], thread: [302,0,0] Assertion *input >= 0. && *input <= 1. failed.
Traceback (most recent call last):
File "tools/train_net.py", line 171, in
main()
File "tools/train_net.py", line 164, in main
model = train(cfg, args.local_rank, args.distributed)
File "tools/train_net.py", line 73, in train
arguments,
File "/home/zz/work/FreeAnchor/maskrcnn_benchmark/engine/trainer.py", line 72, in do_train
meters.update(loss=losses_reduced, **loss_dict_reduced)
File "/home/zz/work/FreeAnchor/maskrcnn_benchmark/utils/metric_logger.py", line 48, in update
v = v.item()
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: device-side assert triggered (insert_events at /opt/conda/conda-bld/pytorch_1556653215914/work/c10/cuda/CUDACachingAllocator.cpp:564)
frame #0: c10::Error::Error(c10::SourceLocation, std::string const&) + 0x45 (0x7f6836c3fdc5 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1: + 0x14792 (0x7f6833b1c792 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x50 (0x7f6836c2f640 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #3: + 0x3067fb (0x7f683423c7fb in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch.so.1)
frame #4: + 0x14019b (0x7f685ca2019b in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #5: + 0x3bfc84 (0x7f685cc9fc84 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x3bfcd1 (0x7f685cc9fcd1 in /home/zz/anaconda3/envs/fa/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x19dfce (0x561e3cab2fce in /home/zz/anaconda3/envs/fa/bin/python)
frame #8: + 0x113a6b (0x561e3ca28a6b in /home/zz/anaconda3/envs/fa/bin/python)
frame #9: + 0x103948 (0x561e3ca18948 in /home/zz/anaconda3/envs/fa/bin/python)
frame #10: + 0x114267 (0x561e3ca29267 in /home/zz/anaconda3/envs/fa/bin/python)
frame #11: + 0x11427d (0x561e3ca2927d in /home/zz/anaconda3/envs/fa/bin/python)
frame #12: + 0x11427d (0x561e3ca2927d in /home/zz/anaconda3/envs/fa/bin/python)
frame #13: + 0x11427d (0x561e3ca2927d in /home/zz/anaconda3/envs/fa/bin/python)
frame #14: PyDict_SetItem + 0x502 (0x561e3ca74602 in /home/zz/anaconda3/envs/fa/bin/python)
frame #15: PyDict_SetItemString + 0x4f (0x561e3ca750cf in /home/zz/anaconda3/envs/fa/bin/python)
frame #16: PyImport_Cleanup + 0x9e (0x561e3cab491e in /home/zz/anaconda3/envs/fa/bin/python)
frame #17: Py_FinalizeEx + 0x67 (0x561e3cb2a367 in /home/zz/anaconda3/envs/fa/bin/python)
frame #18: + 0x227d93 (0x561e3cb3cd93 in /home/zz/anaconda3/envs/fa/bin/python)
frame #19: _Py_UnixMain + 0x3c (0x561e3cb3d0bc in /home/zz/anaconda3/envs/fa/bin/python)
frame #20: __libc_start_main + 0xe7 (0x7f686d0ccb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #21: + 0x1d0990 (0x561e3cae5990 in /home/zz/anaconda3/envs/fa/bin/python)

[ianz27]

@zhangxiaosong18 anything update ？

[huangruofei]

/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [32,0,0] Assertion *input >= 0. && *input <= 1. failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [33,0,0] Assertion *input >= 0. && *input <= 1. failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [34,0,0] Assertion *input >= 0. && *input <= 1. failed.
/opt/conda/conda-bld/pytorch_1579022060824/work/aten/src/THCUNN/BCECriterion.cu:57: void bce_updateOutput_no_reduce_functor<Dtype, Acctype>::operator()(const Dtype *, const Dtype *, Dtype *) [with Dtype = float, Acctype = float]: block: [0,0,0], thread: [35,0,0] Assertion *input >= 0. && *input <= 1. failed.
Traceback (most recent call last):
File "tools/train_net.py", line 171, in
main()
File "tools/train_net.py", line 164, in main
model = train(cfg, args.local_rank, args.distributed)
File "tools/train_net.py", line 73, in train
arguments,
File "/media/user/1T_DISK/lost+found/HRF/FreeAnchor-master/maskrcnn_benchmark/engine/trainer.py", line 65, in do_train
loss_dict = model(images, targets)
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/media/user/1T_DISK/lost+found/HRF/FreeAnchor-master/maskrcnn_benchmark/modeling/detector/retinanet.py", line 50, in forward
(anchors, detections), detector_losses = self.rpn(images, rpn_features, targets)
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/media/user/1T_DISK/lost+found/HRF/FreeAnchor-master/maskrcnn_benchmark/modeling/rpn/retinanet.py", line 146, in forward
return self.forward_train(anchors, box_cls, box_regression, targets)
File "/media/user/1T_DISK/lost+found/HRF/FreeAnchor-master/maskrcnn_benchmark/modeling/rpn/retinanet.py", line 153, in forward_train
anchors, box_cls, box_regression, targets
File "/media/user/1T_DISK/lost+found/HRF/FreeAnchor-master/maskrcnn_benchmark/modeling/rpn/free_anchor_loss.py", line 121, in call
indices = torch.stack([torch.arange(len(labels)).type_as(labels), labels_], dim=0)
RuntimeError: CUDA error: device-side assert triggered
Traceback (most recent call last):
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/site-packages/torch/distributed/launch.py", line 263, in
main()
File "/home/user/anaconda3/envs/wrnd/lib/python3.7/site-packages/torch/distributed/launch.py", line 259, in main
cmd=cmd)
subprocess.CalledProcessError: Command '['/home/user/anaconda3/envs/wrnd/bin/python', '-u', 'tools/train_net.py', '--local_rank=0', '--config-file', 'configs/free_anchor_R-101-FPN_e2x.yaml']' returned non-zero exit status 1.

[huangruofei]

我也有同样问题 您解决了吗