[RFC] Storage of code coverage results in KCIDB

[a-wai]

With kernelci/kernelci-pipeline#1211 Maestro will gain the ability to extract the code coverage results (functions and lines coverage percents) for each test and consolidate/merge those results at the build level.

In order to make this data more useful to end users, we need to store it in KCIDB so we can display historical data and trends in the dashboard. This would e.g. provide factual data points supporting the addition of new tests or spotting regressions due to failed or modified tests.

The plan so far is to store the coverage results in a new table, referencing the build on one hand, and storing the following data as separate fields:

• percentage of lines covered
• percentage of functions covered
• URL of the generated HTML report (which includes coverage data for each source file)

Individual test results doesn't seem relevant as those aren't supposed to change over time.

Comments and suggestions are obviously welcome.

cc @padovan

[tales-aparecida]

Happy to see that kernelci/kernelci-pipeline#1211 will start submitting data in misc fields.
I'm looking forward to see what we get there, to take part in the discussions about adding another table and views 🤗

[padovan]

This issue is more for https://github.com/kernelci/kcidb-ng

Happy to see that kernelci/kernelci-pipeline#1211 will start submitting data in misc fields. I'm looking forward to see what we get there, to take part in the discussions about adding another table and views 🤗

I don't see misc fields being possible here due to perfomance issues. Unless you are talking about the misc fields in the submission json only, but with a proper coverage table in KCIDB. I don't see the point of misc field for this in the submission json either.

First of all, I believe the API/standard should be the submission json, which leaves room for us to work on schema improvements as needed in the background. Having the schema as a standard really slows the process down in my opinion. We need to move away from that.

Now when it comes to coverage data. It relates to a build and huge collection of tests ran to create the coverage report. However, only a very small fraction of the builds will have a coverage report. So if you are asking us to execute queries searching for data on misc field on the build tables in a scenario where a only a fraction of search space will have valid data, I'd be really against that.

In the new kcidb, we should be driving migrations from an ORM hopefully, maybe on the Dashboard Django, so it won't be that problematic to add such a table and work with it. We can easily change/rollback the schema as needed. And querying would be trivial. We select from the coverage table and..done!

We really need to look at being good at delivering value to the users vs running experimentations to understand how to fit the data in disregarding the users need to see coverage report data in the Dashboard.

The legacy KICDB development model is broken IMO. We need to be more agile in general to keep up with the needs of a growing user base.

[tales-aparecida]

I don't see misc fields being possible here due to performance issues. Unless you are talking about the misc fields in the submission json only, but with a proper coverage table in KCIDB. I don't see the point of misc field for this in the submission json either.

Absolutely, if you already have clear data structures in mind, creating those tables makes perfect sense!

I'm inviting you to use of the "misc" fields as soft clay on a potter’s wheel. You shape your ideas freely, test their form, and see how they hold up in real use. If the shape works and the community finds it useful, we can glaze it with proper Dashboard code and fire it into a real DB table.

My perspective here is to make the most of PostgreSQL JSON fields with a couple of weeks of real data, which should be performant enough to query and experiment with a few report prototypes without having to touch Django backend or the React frontend.

But that’s just my angle, if you’ve got solid requirements already scoped out, then your approach is definitely the more direct path.

[tales-aparecida]

First of all, I believe the API/standard should be the submission json, which leaves room for us to work on schema improvements as needed in the background. Having the schema as a standard really slows the process down in my opinion. We need to move away from that.

Definitely agree. The KCIDB JSON schema doesn’t need to map 1:1 to the KernelCI dashboard’s DB tables.

That's pretty much how CKI interacts with KCIDB, we have own schema covering each and every misc field, and every now-and-then we suggested bringing a field that already proved useful to the KernelCI community.

FWIW, my vision for what KCIDB could be is more like a NoSQL-style datalake:

1. companies submit data against a slow-moving KCIDB schema,
2. and consumers, like the KernelCI dashboard, query what’s relevant to them.

My vision for the KernelCI dashboard, therefore, is that it is its own service with its own database, which would have its own refined, fast-evolving schema for the misc fields, and its goal is to clean, structure and report on that data.

Now, storing the raw KCIDB json files put us closer to my NOSQL datalake in a way, and you are talking about separating the KCIDB schema for submissions from the database used by the KernelCI dashboard, which I can only take as step towards what I had in my heart, so I'm happy 😁

[tales-aparecida]

Now when it comes to coverage data. It relates to a build and huge collection of tests ran to create the coverage report. However, only a very small fraction of the builds will have a coverage report. So if you are asking us to execute queries searching for data on misc field on the build tables in a scenario where a only a fraction of search space will have valid data, I'd be really against that.

Just to reinforce, I am saying to "execute queries searching for data on misc field on the build tables in a scenario where a only a fraction of search space will have valid data", but only until we work the requirements for the reports. If the requirements are already scoped out, then it's time to write the tables 💚

[tales-aparecida]

In the new kcidb, we should be driving migrations from an ORM hopefully, maybe on the Dashboard Django, so it won't be that problematic to add such a table and work with it. We can easily change/rollback the schema as needed. And querying would be trivial. We select from the coverage table and..done!

Just to clarify, I’m comparing the workflow of multiple people experimenting directly on the DB (e.g. prototyping tables/charts in Grafana) vs. Profusion building Django querysets, APIs, tests, React code, reviews, and deployment. Both valid paths, just different trade-offs.

I completely agree that we should avoid querying JSON fields in long-term production deployments.

[tales-aparecida]

We really need to look at being good at delivering value to the users vs running experiments to understand how to fit the data in disregarding the users need to see coverage report data in the Dashboard.
The legacy KICDB development model is broken IMO. We need to be more agile in general to keep up with the needs of a growing user base.

Finally, I totally agree, we shouldn’t invest time in hypothetical future use cases where data from different origins might fit together. That’s exactly why I’m suggesting we let the real data guide us first, before putting effort, even if minimal via Django ORM, into building tables. That way, Profusion can stay focused while we iterate on actual reports.