stubgen tests are flaking

[emmatyping]

I had noticed this a few days ago, it seems that stugen tests are flaking every once in a while, I just was able to get a repro:

_________________________ testVarDefsNotInAll_import __________________________
[gw3] win32 -- Python 3.6.4 C:\Python36\python.exe
data: C:\Users\ethanhs\Documents\mypy\test-data\unit\stubgen.test:369:
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:97: in run_case
    test_stubgen(testcase)
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:136: in test_stubgen
    a = load_output(out_dir)
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:158: in load_output
    assert entries, 'No files generated'
E   AssertionError: No files generated
E   assert []
---------------------------- Captured stdout call -----------------------------
Failed to import prog_ha2bjeqt; skipping it
_______________________ testIncludeClassNotInAll_import _______________________
[gw3] win32 -- Python 3.6.4 C:\Python36\python.exe
data: C:\Users\ethanhs\Documents\mypy\test-data\unit\stubgen.test:379:
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:97: in run_case
    test_stubgen(testcase)
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:136: in test_stubgen
    a = load_output(out_dir)
..\..\..\..\Documents\mypy\mypy\test\teststubgen.py:158: in load_output
    assert entries, 'No files generated'
E   AssertionError: No files generated
E   assert []
---------------------------- Captured stdout call -----------------------------
Failed to import prog_fp1d8vup; skipping it

I haven't looked into what is causing the issue yet.

[emmatyping]

I believe this may have to do with #4571? Since it is failing to import programs.

[emmatyping]

See also this Travis log: https://gist.github.com/ethanhs/5169bf3bc1ecee491b1cd9bd3f1ebcc6

[miedzinski]

How did you repro this? I did

until ! ./runtests.py stubgen; do :; done

for 20 minutes and nothing came up. I'm on 064c8d33 commit, Python 3.6.4, macOS 10.3.3.

[emmatyping]

It happens ephemerally, and I'm not doing anything to repro at the moment, the errrors that have come up are random. If it was reproducible I would have looked at what was causing it :)

Taking a look at the stubgen tests this seems like it could cause trouble since AIUI they are run concurrently

[msullivan]

That was my first hunch as well, @ethanhs, but each test is run in its own temp dir.

I made a testing branch that reraised the exception from importlib in stubgen and managed to get it to flake once: https://travis-ci.org/python/mypy/jobs/347964924
There is a ModuleNotFound error coming from importlib. I then added a bunch more debug code to that branch, but haven't been able to get it to flake since.
I think I'm going to merge a variant of that debug code into master and wait for another flake.

[msullivan]

It looks like there are actually only 11 tests that use the code path that tries to import modules with importlib, and it looks like they tend to fail all together.