Support multiple images per gpu for testing the model.

[lartpang]

Describe the feature

Support multiple images per gpu for testing the model.

Motivation
A clear and concise description of the motivation of the feature.
Ex1. It is inconvenient when I have a lot of data to evaluate. In this case, I hope that a larger batch size can be used.
Ex2. (https://github.com/open-mmlab/mmsegmentation/blob/master/tools/test.py#L103) This TODO should be considered to be completed.

Additional context

For single_gpu_test, this problem is easy to solve. But I am not sure how such a modification will affect multi_gpu_test. Can anyone provide some suggestions?

For single_gpu_test, we can modify

mmsegmentation/mmseg/apis/test.py

Lines 31 to 45 in c3e4dbc

	for i, data in enumerate(data_loader):
	with torch.no_grad():
	result = model(return_loss=False, **data)
	if isinstance(result, list):
	results.extend(result)
	else:
	results.append(result)
	if show or out_dir:
	img_tensor = data['img'][0]
	img_metas = data['img_metas'][0].data[0]
	imgs = tensor2imgs(img_tensor, **img_metas[0]['img_norm_cfg'])
	assert len(imgs) == len(img_metas)
	for img, img_meta in zip(imgs, img_metas):

to:

...
    for i, data in enumerate(data_loader):
        with torch.no_grad():
            result_per_batch = model(return_loss=False, **data)
        if isinstance(result_per_batch, list):
            results.extend(result_per_batch)
        else:
            results.append(result_per_batch)

        if show or out_dir:
            img_tensor = data['img'][0]
            img_metas = data['img_metas'][0].data[0]
            imgs = tensor2imgs(img_tensor, **img_metas[0]['img_norm_cfg'])
            assert len(imgs) == len(img_metas)

            for i, img, img_meta in enumerate(zip(imgs, img_metas)):
                    result = result_per_batch[i:i+1]

[xvjiarui]

Hi @lartpang
For multi-gpu should be fine. But you need to handle padding carefully when the datasets consist of images with different sizes.

You are very welcome to open a PR about it.

[wla-coder]

Hi, @xvjiarui
For multiple images testing per gpu, why the inference time "batch_size=8" is almsot the same as "batch_size=1"? Why there is no inferece speedup with larger batch size? Thank you very much and look forward to you answer.

[xvjiarui]

Hi @wla-coder
Did you modify the samples_per_gpu here? I haven't tried by myself. The reason may be that the image spatial scale is large.

[kavyasreedhar]

Hi, are there any updates since this issue was closed on handling padding when datasets consists of images with different image sizes? When modifying samples_per_gpu in test.py, there is a call to torch.stack which requires the tensors to be the same size. As a result, I believe this issue occurs for these datasets regardless of single or multi gpu inference and wanted to check if there was an existing solution that supports batch_size > 1 for inference for datasets such as ADE20K with images of different sizes.

For this thread, please see issue #1370.