Issues with test runner configuration

[Bodigrim]

hackage-server/exes/BuildClient.hs

Lines 654 to 666 in 1217eb0

	testPackage :: Verbosity -> BuildOpts -> DocInfo -> IO (Maybe String, Maybe FilePath, Maybe FilePath)
	testPackage verbosity opts docInfo = do
	let pkgid = docInfoPackage docInfo
	testLogFile = (installDirectory opts) </> display pkgid <.> "test"
	testReportFile = (installDirectory opts) </> "reports" </> display pkgid <.> "test"
	testResultFile = (resultsDirectory opts) </> display pkgid <.> "test"
	pkg_flags =
	["all",
	"--enable-coverage",
	"--test-log=" ++ testReportFile,
	"--test-show-details=never",
	"--disable-optimization"]
	notice verbosity ("Testing " ++ display pkgid)

IMO there are a few issues with this:

• Hiding --test-show-details is very unhelpful. Presumably users are supposed to access --test-log somehow, but I cannot find any link to it in UI.
• Why --disable-optimizations? This breaks inspection testing (fusion rules and such), here is an example: Inspection tests fail if -O0 is forced text#634
• --enable-coverage prevents test execution for any package depending on vector (so pretty much any non-trivial package) because of

Resolving dependencies...
Error:
    Internal libraries only supported with per-component builds.
    Per-component builds were disabled because program coverage is enabled
    In the package 'vector-0.13.2.0'

[gbaz]

The first issue is a good call! I believe the plan was to add a ui to give access to the test build reports, but I can't find an issue for that. Imho that's the right solution.

On point two, I believe optimizations are disabled because they interact poorly with coverage.

Finally, on point three imho the right long term solution is to fix coverage with internal libraries. There is a workaround, but it appears to be somewhat tricky to implement as part of docbuilder, and really we "shouldn't" need it: haskell/cabal#6440

[ulysses4ever]

Fun fact: cabal-install adopted --test-show-details=direct as the default since version 3.12 because people kept asking for it.

[Bodigrim]

The first issue is a good call! I believe the plan was to add a ui to give access to the test build reports, but I can't find an issue for that. Imho that's the right solution.

While we wait for the long-term solution to arrive, could we enable test logs being reported directly for time being? It's very challenging to figure out what failed and why when there are no logs available.

On point two, I believe optimizations are disabled because they interact poorly with coverage.

Hmm, I don't recall any particular issues with regards to this. After all, the coverage metric we care about is when a package is used "normally", which means "with optimizations".

[ysangkok]

the plan was to add a ui to give access to the test build reports

The test logs are already available, as far as I can see. For example, for lens-5.3.5, click "build log" in the side bar. Then at the very bottom, there is a section called "Test log". The section is only shown if logs are available, as you can see in report.html.st.

[Bodigrim]

That's build logs, but not test logs. If you look at the bottom of https://hackage.haskell.org/package/text-2.1.2/reports/2, it says "Error: cabal: Tests failed for text-2.1.2", but the failure is nowhere to be found.

Running 1 test suites...
Test suite tests: RUNNING...
Test suite tests: FAIL
Test suite logged to:
/var/lib/hackage-doc-builder/build-cache/tmp-install/reports/text-2.1.2.test

There is no way to access /var/lib/hackage-doc-builder/build-cache/tmp-install/reports/text-2.1.2.test.