2.4.8 Integration test annotations not working because of phpunit 9->10 upgrade - event management system

[convenient]

This is a similar kind of issue to #39463 in that we have code looking to hook into TestResult which is a class that no longer exists.

In short, I believe all php annotations used in integration tests are no longer functioning as expected. This prevents us from properly writing tests to verify our work.

I have not fully tested all of them, but can verify it for annotations like

/**
 * @magentoAdminConfigFixture some/config/value 1
 */

The reason for this is that Magento has bumped the phpunit dependency as follows

• 2.4.7-p5 requires "phpunit/phpunit": "^9.5",
• 2.4.8 requires "phpunit/phpunit": "^10.5",

This major version changes the event system, which does not appear to have corresponding fixes in the codebase.

I have been able to reproduce this on a vanilla 2.4.8 installation.

How it works in 2.4.7-p5 / phpunit ^9.5

phpunit would call $listener->startTest($test) before each test execution on the TestResult class

https://github.com/sebastianbergmann/phpunit/blob/43d2cb18d0675c38bd44982a5d1d88f6d53d8d95/src/Framework/TestResult.php#L435-L443

The magento framework has an eventmanager system which would fire a subsequent event for startTest

magento2/dev/tests/integration/framework/Magento/TestFramework/Event/PhpUnit.php

Lines 122 to 128 in dc87600

	public function startTest(\PHPUnit\Framework\Test $test): void
	{
	if (!$test instanceof \PHPUnit\Framework\TestCase || $test instanceof \PHPUnit\Framework\Warning) {
	return;
	}
	$this->_eventManager->fireEvent('startTest', [$test]);
	}

magento2/dev/tests/integration/framework/Magento/TestFramework/EventManager.php

Lines 43 to 52 in dc87600

	public function fireEvent($eventName, array $parameters = [], $reverseOrder = false)
	{
	$subscribers = $reverseOrder ? array_reverse($this->_subscribers) : $this->_subscribers;
	foreach ($subscribers as $subscriberInstance) {
	$callback = [$subscriberInstance, $eventName];
	if (is_callable($callback)) {
	call_user_func_array($callback, $parameters);
	}
	}
	}

This would be picked up by the Annotation/AdminConfigFixture.php class which would handle startTest, and stub in the necessary config values

magento2/dev/tests/integration/framework/Magento/TestFramework/Annotation/AdminConfigFixture.php

Lines 104 to 108 in dc87600

	public function startTest(TestCase $test)
	{
	$this->_currentTest = $test;
	$this->_assignConfigData($test);
	}

How it fails to work in 2.4.8 / phpunit ^10.5

The entire event system has changed, and I cannot see how it is compatible. I do not know how the current integration test fixtures are running on this repository, given that they should also be using phpunit 10 series.

• See that no startTest event exists https://docs.phpunit.de/en/10.5/events.html#test
• See that the Magento phpunit event class still functions based on startTest https://github.com/magento/magento2/blob/2.4.8/dev/tests/integration/framework/Magento/TestFramework/Event/PhpUnit.php#L122

Verification of issue

The phpunit 9 series has startTest references, so the magento startTest events hook into that

$ git clone https://github.com/sebastianbergmann/phpunit/ --depth=1 --branch 9.6 phpunit-9-6
Cloning into 'phpunit-9-6'...
remote: Enumerating objects: 1602, done.
remote: Counting objects: 100% (1602/1602), done.
remote: Compressing objects: 100% (1043/1043), done.
remote: Total 1602 (delta 918), reused 764 (delta 480), pack-reused 0 (from 0)
Receiving objects: 100% (1602/1602), 5.49 MiB | 12.57 MiB/s, done.
Resolving deltas: 100% (918/918), done.

$ grep -r startTest phpunit-9-6 | wc -l
      56

The phpunit 10 series clearly has no references, so the magento startTest events never trigger

$ git clone https://github.com/sebastianbergmann/phpunit/ --depth=1 --branch 10.5 phpunit-10-5
Cloning into 'phpunit-10-5'...
remote: Enumerating objects: 2944, done.
remote: Counting objects: 100% (2944/2944), done.
remote: Compressing objects: 100% (1807/1807), done.
remote: Total 2944 (delta 1858), reused 1464 (delta 985), pack-reused 0 (from 0)
Receiving objects: 100% (2944/2944), 5.77 MiB | 12.86 MiB/s, done.
Resolving deltas: 100% (1858/1858), done.

$ grep -r startTest phpunit-10-5 | head -10 | wc -l
       0

Example repository

Additionally I have spun up a repository with running integration tests, see https://github.com/convenient/magento-broken-integration-tests-events

It has a test like so

/**
 * @magentoAdminConfigFixture some/config/value 1
 */
public function testConfigFixtureSet()
{
    $this->assertTrue(
        Bootstrap::getObjectManager()->get(ScopeConfigInterface::class)->isSetFlag('some/config/value'),
        'some/config/value should be set by the fixture'
    );
}

And it works on the 2.4.7-p5 series, but not 2.4.8

The actions can be seen here https://github.com/convenient/magento-broken-integration-tests-events/actions/runs/14926109084

The test failure is like

There was 1 failure:
1) Convenient\Hello\Test\Integration\TestConfigFixtureTest::testConfigFixtureSet
some/config/value should be set by the fixture
Failed asserting that false is true.

[m2-assistant]

Hi @convenient. Thank you for your report.
To speed up processing of this issue, make sure that the issue is reproducible on the vanilla Magento instance following Steps to reproduce.

• For more details, review the Magento Contributor Assistant documentation.
• Add a comment to assign the issue: @magento I am working on this
• To learn more about issue processing workflow, refer to the Code Contributions.

---

Join Magento Community Engineering Slack and ask your questions in #github channel.
⚠️ According to the Magento Contribution requirements, all issues must go through the Community Contributions Triage process. Community Contributions Triage is a public meeting.
🕙 You can find the schedule on the Magento Community Calendar page.
📞 The triage of issues happens in the queue order. If you want to speed up the delivery of your contribution, join the Community Contributions Triage session to discuss the appropriate ticket.

[convenient]

Closing after discussion with @michielgerritsen and his example here.