Document our Why / How etc #12
Conversation
Co-authored-by: Hippie Hacker <[email protected]>
Co-authored-by: Hippie Hacker <[email protected]>
README.md
Outdated
| ## Welcome to our really new / exploration level project. | ||
|
|
||
| ### Why | ||
| Our initial goal is to provide a useful indicator as to which Kubernetes APIs are used the most and don't yet have conformance tests. This is specifically to ensure we are creating testing API's that are relevant, rather than just hitting all the endpoints. Our K8s API consumer/user journeys are an important model to drive this prioritization, in addition to providing insight into _how_ the APIs are being used. |
README.md
Outdated
| ## Welcome to our really new / exploration level project. | ||
|
|
||
| ### Why | ||
| Our initial goal is to provide a useful indicator as to which Kubernetes APIs are used the most and don't yet have conformance tests. This is specifically to ensure we are creating testing API's that are relevant, rather than just hitting all the endpoints. Our K8s API consumer/user journeys are an important model to drive this prioritization, in addition to providing insight into _how_ the APIs are being used. |
README.md
Outdated
| ### Why | ||
| Our initial goal is to provide a useful indicator as to which Kubernetes APIs are used the most and don't yet have conformance tests. This is specifically to ensure we are creating testing API's that are relevant, rather than just hitting all the endpoints. Our K8s API consumer/user journeys are an important model to drive this prioritization, in addition to providing insight into _how_ the APIs are being used. | ||
|
|
||
| Our currently rough output is the most promising visualazion of how api groups endpoints and verbs are used today. The APISnoop Visualization presents in very clear terms how poor our current coverage actually is. Our high level graph shows our stable API's are mostly untested ( the grey outer donut sections). |
|
Thanks for doing this, @hh :-) BTW, I have one question about this kind of thing. Do we want to count all APIs which include ones returned error status codes(like HTTP 404, etc) as tested? or do we want to count succeeded APIs only as we verify happy path of these APIs? |
README.md
Outdated
|
|
||
| In order to identify target applications to test, we define a Kubernetes API Consumer as a _KAPIC_. | ||
|
|
||
| We inspect the [advanced audit logs](https://kubernetes.io/docs/tasks/debug-application-cluster/audit/) to describe which APIs are called during KAPIC operations. For our initial data run we installed a small set of KAPIC helm charts and observing which API groups are called. We used the [d3 libary](https://github.com/d3/d3/wiki/Gallery) to create [Sunburst Partition Graphs](https://bl.ocks.org/mbostock/4063423) center stable/beta/alpha with partitions for APIGroups then APICalls/Verbs. |
|
Thanks a lot for doing this @hh and also for the documentation :) |
|
/lgtm |
|
@oomichi regarding including 404s as tested: If the e2e / conformance suite is hitting 404s, I think it should be considered a failed test. If KAPICs are hitting 404s, there is activity at that endpoint but it's likely that application isn't mean to run against that version of k8s. I think the number of 404s we see from applications should also be quite small. A larger data set (more applications) will give us a better understanding here. We will try and streamline creating and contributing apisnoop data to get us there faster. |
show technical debt in conformance progress in top conformance progress chart
No description provided.