Skip to content

fix: error for multi-runner setup when using extra_labels #4587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

raford-ice
Copy link

@raford-ice raford-ice commented May 19, 2025

I have found a issue when configuring a multi-runner module setup when using runner_extra_labels. I am going to simplify this example not adding extra irrelevant configuration.

Suppose you have a runner map with this configuration (in this example I will only show one runner):

runners_config = {
  "amazon-arm64" = {
    "matcherConfig" = {
      "exactMatch" = true
      "labelMatchers" = [
        [
          "self-hosted",
          "linux",
          "arm64",
          "amazon",
        ],
      ]
    }
    "runner_config" = {
      "enable_organization_runners" = true
      "instance_types" = [
        "t4g.small",
      ]
      "runner_architecture" = "arm64"
      "runner_extra_labels" = [
        "test",
      ]
      "runner_name_prefix" = "gh-runner-arm64-"
      "runner_os" = "linux"
    }
  }
}

So no pool is configured, after running terraform, no instances will be running.

When I trigger a new workflow run with:

    runs-on: [self-hosted, amazon, arm64]

it works as expected, webhook pick up the job and a new runner is created and pick the job.

When I trigger a new workflow run with:

    runs-on: [self-hosted, amazon, arm64, test]

Note now, I have added test label, the workflow run is picked up by Github and it stays in pending status with no runner picking up the job.

dispatch-to-runner lambda rejects the job with the following log:

    "message": "Received event contains runner labels 'self-hosted,amazon,arm64,test' that are not accepted.",
    ...

Changes introduced in this commit fixes the issue for multi-runner module

@raford-ice raford-ice requested a review from a team as a code owner May 19, 2025 17:02
@raford-ice raford-ice changed the title Fix error for multi-runner setup when using extra_labels. Dispatch la… fix: error for multi-runner setup when using extra_labels May 19, 2025
@raford-ice raford-ice force-pushed the fix/multi-runner-dispatch branch from 6be2dca to 5081205 Compare May 19, 2025 17:31
@npalm
Copy link
Member

npalm commented May 24, 2025

Sorry have not much time. The labels is complex and should be read als capabilities. In you example you have set exact match to true. This will let the lambda try to match all the labels. And since your job is not containing all, no event is put on the queue to scale. Indeed a runner with those labels could run your job. Can you trie it with exact match set to false?

@raford-ice
Copy link
Author

Sorry for my late reply. I can confirm that if you set the flag exactMatch to false it will work, because it is not asserting all the labels are present which makes sense. But this is against the definition of exactMatch and runner_extra_labels variables:

exactMatch: "If set to true all labels in the workflow job must match the GitHub labels (os, architecture and self-hosted). When false if any workflow label matches it will trigger the webhook."

runner_extra_labels: "Extra (custom) labels for the runners (GitHub). Separate each label by a comma. Labels checks on the webhook can be enforced by setting multi_runner_config.matcherConfig.exactMatch. GitHub read-only labels should not be provided."

If you can not use exactMatch to true to make it work with runner_extra_labels what is the purpose of runner_extra_labels? From my perspective runner_extra_labels is a way to extend labelMatchers list but with the same functionality.

@raford-ice
Copy link
Author

@npalm any update on this?

@npalm
Copy link
Member

npalm commented Jun 16, 2025

I had no time yet to dig in this PR. Changes to the wya labels are processed are proven to be risky :(

Anyone that has time to review the PR feel free to dig.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants