This repository is meant to host all the code, documents, reports, configuration files, etc. relevant to working with Antithesis platform to test various components and configurations of the Cardano network.
Important
This project is currently in a very early stage: A team at the Cardano Foundation is responsible to testing the platform and evaluating how it fits the needs of Cardano. We strive to work in the open, engage with the community as much as possible, and expects questions and contributions, but the requirement to move fast and do a thorough due diligence work might lead us to not be as responsive as expected.
This project is conceptually split in potentially three phases, with different goals:
- Proof-of-concept and evaluation: Does Antithesis significantly improves how we can test Cardano network and components?
- Community validation: Is the "Cardano Community" at large of interested in such a tool?
- Production phase: Here the goal is simply to keep operating the platform in the most satisfying and cost-effective way on behalf of the community
All dates are expressed relative to project's start date:
| Time | Phase | Deliverable |
|---|---|---|
| T+1m | PoC | PoC report detailing: |
| - experiments run using cardano-node only | ||
| - results (eg. did we find the bugs we expected to find, and possibly other ones?) | ||
| - analysis of fitness-for-purpose of the platform w.r.t testing cardano-node, possibly detailing needed adjustments to the code (eg. tracing) | ||
| - feedback from community (eg. core team, other node implementors, SMEs, SPOs, etc. | ||
| - Answer to the question: Shall we keep evaluating the platform to test the cardano-node and/or other components of Cardano? | ||
| T+3m | Prototyping | Evaluation report detailing: |
| - as above plus | ||
| - possible plan(s) to make the platform acccessible to community members | ||
| - evaluation of cost/benefit for node/components implementors | ||
| - economic analysis of operating model for the project | ||
| - Answer to the question: should the Cardano community pursue investment in the platform? | ||
| T+ (k * 3m) | Production | Produce quarterly reports on the use of the platform |
Here is a sketch of the expected structure of this repository, and the overall process to run and analyse tests results.
There shall be one directory per test, containing all the material to run the test on Antithesis along with outcomes:
docker-composedefinition- Dockerfile to build the image if needed
- workload and test drivers
- Test results artifacts and analysis
Each test run will be triggered from a Github action:
- Initially through a manual trigger with a parameter pointing at the right directory:
- Later on through submitting a Pull Request
Test results will be available through an authenticated access to antithesis platform, so they won't be directly shareable. Moreover, in the case test results expose some potential security vulnerability, we certainly don't want to disclose those publicly before notifying interested parties. The general process should look like:
- Once test results are available, acknowledge the end of test by updating the Logbook with some basic data from the result (time run, number of tests executed)
- Analyse the result
- Write a report and publish it in the relevant directory
Note
It's still unclear how the results are available, what kind of reports are produced, and how to analyse and use those. Things will become clearer after onboarding and a few tests have been run.
We'll use GH Issue(s) to keep track of the list of interesting tests we want to run. At the time of this writing, we mostly want to have a baseline (or smoke) test to onboard the team with the platform, then a few tests to reproduce known bugs in the network, consensus, or even ledger parts, in order to assess effectiveness of the platform and understand how tests can be best structured to reap the most benefits out of it.
At least during the PoC/Prototyping period, it's critical the team updates the Logbook on frequent basis, at least weekly and ideally daily, to keep track of experiments run, issues, successes, problems, etc.
We are using radicle to track the code and the issues. You can observe
- code at rad:z2a7Te5b28CX5YyPQ7ihrdG2EEUsC - issues at issues
Developers: