midenup and miden

[bitwalker]

The following is a comment I posted in the DevRel repo, which isn't public, so I'm reposting it here for discussion within the context of midenup. The context of the discussion preceding my comment was about unifying the Miden toolchain under a single CLI tool, called miden, which would have some kind of basic install process, e.g. cargo install miden. That discussion was started in parallel with our design of midenup, and I think both midenup and miden are essentially two sides of the same coin - one handling installation/updates/toolchain management, and one handling usage of a selected toolchain, by surfacing all of the toolchain components using a single entrypoint, miden.

---

I definitely think we should avoid trying to combine everything literally into a single CLI (as opposed to a Git-style porcelain approach). There are multiple reasons for that:

• The surface area of such a CLI would be enormous, would be critical to every team of the Miden project, and would certainly be a bottleneck for development and release.
• Dependency management would almost certainly be a nightmare.
• Huge CLIs are slow to start and consume much more memory
• The value in smaller components that work together is that they can tailor themselves to their purpose, be released more frequently, and with less coordination, and be maintained by the respective team as part of their normal development flow (akin to how cargo-miden is maintained within the compiler repo, as part of our regular development process).

As an aside, I think we should avoid conflating two closely related activities:

• Toolchain Management (bootstrapping the environment, integrating with the user's shell, fetching/installing components, versioning and updates)
• Toolchain Usage (creating a project, building/testing/deploying it, etc., using the currently installed/selected toolchain)

With that in mind, I think our approach should be comprised of two tools, midenup and miden, as follows:

• Use midenup for toolchain management. This would involve a user running cargo install midenup && midenup init to walk through configuring their environment and installing the desired set of initial components. Updating the currently installed toolchain components would be as simple as running midenup update. The midenup tool gets information about available toolchains, and their components, by fetching it from GitHub (published as JSON via GitHub Pages). Thus new versions and components can be made available without having to release a new version of midenup itself.
• Toolchains would be described either in the form of channels, e.g. stable, or explicit versions. Toolchain versions I think could be based on the Miden VM version initially, since it is currently the most foundational component, and the one most likely to break compatibility; but we can also version toolchains independently, so that we can more accurately represent changes between toolchain versions. Channels are a way of staying up to date with latest versions of all components without pinning a version. For example, the stable channel would always reflect the latest versions of all components, and running midenup update stable would update to those latest versions, even if there are breaking changes since the last time you updated it. A channel corresponding to a specific Miden version, e.g. 0.15.0, would always reflect the latest versions of all components compatible with that toolchain version, i.e. if a new version of the Miden node is released, but it isn't backwards-compatible with 0.15.0, then running midenup update 0.15.0 would ignore that new version.
• The midenup CLI would provide shell integration so that switching to a directory with a miden-toolchain.toml file would activate the toolchain specified in that file. If the toolchain is not currently installed, midenup would do so the first time miden is run. Toolchains are activated at a low-level by setting MIDEN_SYSROOT to the appropriate toolchain root directory, and prepending $MIDEN_SYSROOT/bin to the current PATH, making all of the executables of that toolchain available for use.
• Provide a miden executable which is a thin wrapper around the set of binaries installed by midenup, and integrates with midenup when necessary. The miden executable would be installed by midenup init. This executable would consist of any number of subcommands implemented as one of the following:
  1. Native subcommands, i.e. functionality which is implemented as part of the miden binary. It is unlikely that there would be many of these, but things related to global settings and such might be appropriate for this kind of subcommand.
  2. External subcommands, i.e. functionality which is implemented in an executable that is expected to be present in the user's PATH, and which requires no pre-processing of its arguments. When invoking this type of subcommand, the miden CLI would execute the external tool as a subprocess, forwarding all of the arguments. For example, miden foo [ARGS..] would expect an executable in the current PATH called miden-foo, which would be invoked as miden-foo [ARGS..]. If miden-foo is not found, we could either inform the user that they need to install that component; prompt them if they'd like to install it; or automatically install it.
  3. Aliased subcommands, i.e. functionality which is implemented externally as in 2, but in a way that we wish to expose differently via miden. For example, miden send might correspond to something like miden-client transaction send. This form of subcommand essentially converts/forwards its arguments to whatever is expected by the underlying executable, as if it had been invoked as a regular external subcommand.

Putting all of this together, we get the following:

• A streamlined installer and updater for all Miden toolchain components
• Simple toolchain management (i.e. for doing development in different projects that have different version requirements)
• The benefits of a unified CLI interface, without the downsides of trying to build everything into a single giant executable (if we could even do so)
• Individual components can be released/versioned/updated independently, but end users don't need to install/manage them all, they can simply run midenup update when they wish to pick up updates that are compatible with their current channel/toolchain.

Work on midenup has already started here (the repo was just created today, so it may be a few days before the repo where prototyping started gets merged in, but that's where things will live going forward). The immediate goal is to build an MVP of the midenup functionality, i.e. bootstrapping the stable toolchain with a subset of the Miden components, namely the VM, compiler, cargo-miden, and standard and transaction kernel libraries.

From there, the next step is to define the miden CLI as described above. Following that, we'll want to create/rename all of the executables that we wish to expose via miden wherever they live today (e.g. miden executables in the VM and client repos would be renamed to miden-run (my suggestion) and miden-client respectively). Once that's done, we can expose more components for installation via midenup. Further improvements to both midenup and miden can be identified and implemented from there.

---

/cc @igaray @greenhat

[bitwalker]

Closing this in favor of #2