Record detailed test failure modes

[jdries]

Currently, test scenarios either fail or succeed.
We notice that there are however different failure modes:

• The service was not able to produce any result.
• The service produced a result, it was X% different for the same input data
• The service produced a result, it was X% different, a change in input data was detected

Tracking these separately upon failure would give us a more nuanced picture, allowing to spot faster where fast intervention is relevant.

This plugin seems to do this:
https://pypi.org/project/pytest-custom-outputs/

[soxofaan]

Already added new tracking of separate phases of a benchmark:
connect, create-job, run-job, collect-metadata, download-actual, download-reference and compare.

If an exception happens, it's now possible to see from the "metrics" in which phase this happened:
e.g. run 1007 has metric

['test:phase:exception', 'run-job']

which means an exception occurred during the run-job phase

There is also an initial feature for "change in input data was detected": if there is a change in derived_from links, the metric will be

["test:phase:exception", "compare:derived_from-change"],

e.g see test_track_phase_describe_derived_from_change