Update workflow files to use self-hosted runners

[prop-opentensor]

Description

Updates workflow files to use ephemeral self-hosted runners. Standby runners of each used instance type are available. If no standby runner is available, a new runner is provisioned. Each job uses a separate runner, after which it is re-imaged. Runners are put in recycle mode for up to an hour, which means that if extra runners were spun up during a period of high activity, they will serve as standby runners for the remainder of their billed hour.

In the event of capacity issues that prevent further scaling, only the standby runners will be available. Jobs will queue until they are finished and re-provisioned to pick up the next one.

Related Issue(s)

x

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Other (please describe):

Breaking Change

Most workflows have been tested and adapted already where possible in local testing, but some issues may arise. Github hosted runners have a lot of dependencies pre-installed, while these runners use a base Ubuntu 24.04 installation with only Docker added. No assumptions can be made about pre-existing dependencies, and jobs need to install them.

Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have run cargo fmt and cargo clippy to ensure my code is formatted and linted correctly
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules

Screenshots (if applicable)

Please include any relevant screenshots or GIFs that demonstrate the changes made.

Additional Notes

Please provide any additional information or context that may be helpful for reviewers.

[sam0x17]

looks like 2 little tests failing related to the weight changes