7.7.* degraded performance

[alexei]

I noticed our CI workflows have gotten slower lately. I did some investigations and found it happened on Apr 1st and so far I was able to isolate it to when we upgraded coverage from 7.6.12 to 7.8.0

In short, previously our tests took 10-15 min. now they take ~15-20 min:

I think we have added 1.5% more tests in the past month, so I ran all the tests we have today in main under multiple versions of the coverage library:

We use Python 3.12.7 and fwiw we run tests with COVERAGE_CORE=sysmon since Mar 2024.

I'd appreciate any suggestions in helping either me or you figure this out.

[nedbat]

The other factors are the same? Same version of Python? What are your coverage settings?

[alexei]

Thanks for the response!

The workflows in the screenshot differ in the coveragepy version only.

Config is:

[tool.coverage.run]
branch = true
source = ["."]

[tool.coverage.report]
omit = [
    "*/migrations/*",
]
skip_empty = true
fail_under=79
precision=2

[nedbat]

Using 3.12, you don't want COVERAGE_CORE=sysmon and branch=true. If you want branch coverage, use the default CTracer core. Branch coverage isn't fast until 3.14.

[alexei]

We've been using branch=true for at least 4 years.

On Mar 6, 2024 we upgraded to Python 3.12 and we found tests took ~45 min.

Then on Mar 18, 2024 we started using COVERAGE_CORE=sysmon and time went down to ~25 min.

Now you're suggesting to unset it. I'm a confused about this suggestion, but I'll set up branches to measure all combinations.

[nedbat]

It's not that branch=true is slow. It's branch=true combined with COVERAGE_CORE=sysmon on Python before 3.14. I recommend unsetting COVERAGE_CORE.

[miketheman]

I've just found the same - and noted it here: pypi/warehouse#17872 (comment)
What's odd to me is that 7.6.12 works with sysmon, branch, and contexts, and is faster than ctrace.

[nedbat]

I have details and some suggestions on the other issue, but maybe I should have put it here...

[alexei]

I ran the workflows with different combinations both on Thursday:

And today:

(ignore the "failed" workflows - there's a flaky test that's not fixed in main)

It looks like on Python 3.12, so far, the fastest is either:

• COVERAGE_CORE=sysmon and branch = false, with any coverage version;
• or COVERAGE_CORE=sysmon and branch = true, with coverage==7.6.12.

I'm thinking of sticking to coverage==7.6.12 for now, and revisiting the situation once we upgrade to Python 3.13 and later 3.14 if needed.