Calling cv::cuda::Stream::Null() results in a stray cudaMalloc() call

[nglee]

System information (version)

• OpenCV => master
• Operating System / Platform => Windows 10 64-Bit , CUDA 8.0
• Compiler => Visual Studio 2015

Detailed description

If I call cv::cuda::Stream::Null() in my code, it seems it calls cudaMalloc() without freeing the allocated memory, resulting in a memory leak error when tested with the command line memory checker cuda-memcheck --leak-check full.

Steps to reproduce

__global__ void hello()
{
	printf("%u: hello\n", threadIdx.x);
}
int main()
{
	hello<<<1, 16, 0, cv::cuda::StreamAccessor::getStream(cv::cuda::Stream::Null())>>>();
	CheckCudaError(cudaDeviceSynchronize());
	CheckCudaError(cudaDeviceReset());
}

If you profile above code with nvprof then it calls cudaMalloc() without any cudaFree() followed.

[nglee]

This comment is a summary note for the merged PR that fixed this issue.

Investigation

• When cv::cuda::Stream::Null() is called in my sample code, a MemoryPool object is allocated some GPU memory with cudaMalloc.
• The MemoryPool object is owned by DefaultDeviceInitializer object, and a global instance of it is declared.
• The GPU memory allocated with cudaMalloc is intended to be destroyed in the destructor of DefaultDeviceInitializer, which, in turn, calls MemoryPool::release().

Experiment

• When I added cudaSafeCall to cudaFree in MemoryPool::release(), which is responsible for deallocating the GPU memory, it returned driver shutting down (cudaErrorCudartUnloading = 29) error code.

Reasoning

• With help from this and this, it seems that cuda runtime environment has been shut down before the destructor of DefaultDeviceInitializer is called. This should be the reason why driver shutting down (cudaErrorCudartUnloading = 29) error was returned in the experiment.
• CUDA context being destroyed before cudaFree is called, cuda-memcheck may assume this situation as memory leak.

Solutions

• (Option 1) Do not use a global instance that de-allocates CUDA memory in its destructor. This is already mentioned in the OpenCV doc for GpuMat, that says:

You are not recommended to leave static or global GpuMat variables allocated, that is, to rely on its destructor. The destruction order of such variables and CUDA context is undefined. GPU memory release function returns error if the CUDA context has been destroyed before.

• (Option 2) Do not pre-allocate CUDA memory for each Stream object, so that the de-allocation does not happen at the destructor of a global instance. The merged PR that fixes this issue takes this path.