memory leak in boxlist_iou?

[twmht]

I found out that memory leak may happen in box_list_iou (https://github.com/zhangxiaosong18/FreeAnchor/blob/master/maskrcnn_benchmark/modeling/rpn/free_anchor_loss.py#L108)

when it processes large number of ground truth, of course the gpu memory grows up. but it never frees the memory it holds.

turning to cpu mode is a workaround but very slow.

[twmht]

I set batch size to 32, if i turn box_list_iou into cpu mode, gpu memory consumption is only 6GB for each of two gpus, but when turning box_list_iou into gpu mode, it would grow up and finally out of memory.