Error: Failed preparing Gitlab runner: Forbidden in cml-cloud-case

[DavidGOrtega]

I have updated the images in the yaml to discover that the repo is failing . My personal fashion_mnist uses the images and its OK. The only difference is the machine size.

{"level":"info","message":"Preparing workdir /home/runner..."}
{"level":"info","message":"Deploying cloud runner plan..."}
{"level":"info","message":"Terraform apply..."}
{"level":"error","message":"terraform -chdir='/home/runner' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  \u001b[32m+\u001b[0m create\n\u001b[0m\nTerraform will perform the following actions:\n\n\u001b[1m  # iterative_cml_runner.runner\u001b[0m will be created\u001b[0m\u001b[0m\n\u001b[0m  \u001b[32m+\u001b[0m\u001b[0m resource \"iterative_cml_runner\" \"runner\" {\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mcloud\u001b[0m\u001b[0m                = \"aws\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mdriver\u001b[0m\u001b[0m               = \"gitlab\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mid\u001b[0m\u001b[0m                   = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0midle_timeout\u001b[0m\u001b[0m         = 300\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mimage\u001b[0m\u001b[0m                = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_hdd_size\u001b[0m\u001b[0m    = 64\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_ip\u001b[0m\u001b[0m          = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_launch_time\u001b[0m\u001b[0m = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_type\u001b[0m\u001b[0m        = \"g3.4xlarge\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mlabels\u001b[0m\u001b[0m               = \"cml-runner-gpu\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mname\u001b[0m\u001b[0m                 = \"cml-gp06y1oau8\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mregion\u001b[0m\u001b[0m               = \"us-west\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mrepo\u001b[0m\u001b[0m                 = \"https://gitlab.com/iterative.ai/cml-cloud-case\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0msingle\u001b[0m\u001b[0m               = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot\u001b[0m\u001b[0m                 = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot_price\u001b[0m\u001b[0m           = -1\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mssh_public\u001b[0m\u001b[0m           = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mtoken\u001b[0m\u001b[0m                = (sensitive value)\n    }\n\n\u001b[0m\u001b[1mPlan:\u001b[0m 1 to add, 0 to change, 0 to destroy.\n\u001b[0m\u001b[0m\u001b[1miterative_cml_runner.runner: Creating...\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [30s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [40s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [50s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m0s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m30s elapsed]\u001b[0m\u001b[0m\n\n\t\u001b[31m╷\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mError checking the runner status\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[0m  with iterative_cml_runner.runner,\n\u001b[31m│\u001b[0m \u001b[0m  on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n\u001b[31m│\u001b[0m \u001b[0m  14: resource \"iterative_cml_runner\" \"runner\" \u001b[4m{\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m-- Logs begin at Thu 2021-09-30 13:32:55 UTC, end at Mon 2021-12-06\n\u001b[31m│\u001b[0m \u001b[0m12:33:58 UTC. --\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:35 ip-172-31-2-224 systemd[1]: Started cml.service.\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Preparing workdir\n\u001b[31m│\u001b[0m \u001b[0m/tmp/tmp.PdTeIq49Jr/.cml/cml-gp06y1oau8...\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Launching gitlab runner\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"error\",\"message\":\"Failed preparing Gitlab runner:\n\u001b[31m│\u001b[0m \u001b[0mForbidden\",\"stack\":\"Error: Failed preparing Gitlab runner: Forbidden\\n\n\u001b[31m│\u001b[0m \u001b[0mat Gitlab.startRunner\n\u001b[31m│\u001b[0m \u001b[0m(/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:209:13)\\n    at\n\u001b[31m│\u001b[0m \u001b[0mprocessTicksAndRejections\n\u001b[31m│\u001b[0m \u001b[0m(internal/process/task_queues.js:97:5)\",\"status\":\"terminated\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"waiting 20 seconds before exiting...\"}\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m╵\u001b[0m\u001b[0m\n","stack":"Error: terraform -chdir='/home/runner' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  \u001b[32m+\u001b[0m create\n\u001b[0m\nTerraform will perform the following actions:\n\n\u001b[1m  # iterative_cml_runner.runner\u001b[0m will be created\u001b[0m\u001b[0m\n\u001b[0m  \u001b[32m+\u001b[0m\u001b[0m resource \"iterative_cml_runner\" \"runner\" {\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mcloud\u001b[0m\u001b[0m                = \"aws\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mdriver\u001b[0m\u001b[0m               = \"gitlab\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mid\u001b[0m\u001b[0m                   = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0midle_timeout\u001b[0m\u001b[0m         = 300\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mimage\u001b[0m\u001b[0m                = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_hdd_size\u001b[0m\u001b[0m    = 64\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_ip\u001b[0m\u001b[0m          = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_launch_time\u001b[0m\u001b[0m = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0minstance_type\u001b[0m\u001b[0m        = \"g3.4xlarge\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mlabels\u001b[0m\u001b[0m               = \"cml-runner-gpu\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mname\u001b[0m\u001b[0m                 = \"cml-gp06y1oau8\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mregion\u001b[0m\u001b[0m               = \"us-west\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mrepo\u001b[0m\u001b[0m                 = \"https://gitlab.com/iterative.ai/cml-cloud-case\"\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0msingle\u001b[0m\u001b[0m               = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot\u001b[0m\u001b[0m                 = false\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mspot_price\u001b[0m\u001b[0m           = -1\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mssh_public\u001b[0m\u001b[0m           = (known after apply)\n      \u001b[32m+\u001b[0m \u001b[0m\u001b[1m\u001b[0mtoken\u001b[0m\u001b[0m                = (sensitive value)\n    }\n\n\u001b[0m\u001b[1mPlan:\u001b[0m 1 to add, 0 to change, 0 to destroy.\n\u001b[0m\u001b[0m\u001b[1miterative_cml_runner.runner: Creating...\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [30s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [40s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [50s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m0s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m10s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m20s elapsed]\u001b[0m\u001b[0m\n\u001b[0m\u001b[1miterative_cml_runner.runner: Still creating... [1m30s elapsed]\u001b[0m\u001b[0m\n\n\t\u001b[31m╷\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[1m\u001b[31mError: \u001b[0m\u001b[0m\u001b[1mError checking the runner status\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\u001b[0m  with iterative_cml_runner.runner,\n\u001b[31m│\u001b[0m \u001b[0m  on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n\u001b[31m│\u001b[0m \u001b[0m  14: resource \"iterative_cml_runner\" \"runner\" \u001b[4m{\u001b[0m\u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m│\u001b[0m \u001b[0m-- Logs begin at Thu 2021-09-30 13:32:55 UTC, end at Mon 2021-12-06\n\u001b[31m│\u001b[0m \u001b[0m12:33:58 UTC. --\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:35 ip-172-31-2-224 systemd[1]: Started cml.service.\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Preparing workdir\n\u001b[31m│\u001b[0m \u001b[0m/tmp/tmp.PdTeIq49Jr/.cml/cml-gp06y1oau8...\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:43 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"Launching gitlab runner\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"error\",\"message\":\"Failed preparing Gitlab runner:\n\u001b[31m│\u001b[0m \u001b[0mForbidden\",\"stack\":\"Error: Failed preparing Gitlab runner: Forbidden\\n\n\u001b[31m│\u001b[0m \u001b[0mat Gitlab.startRunner\n\u001b[31m│\u001b[0m \u001b[0m(/usr/lib/node_modules/@dvcorg/cml/src/drivers/gitlab.js:209:13)\\n    at\n\u001b[31m│\u001b[0m \u001b[0mprocessTicksAndRejections\n\u001b[31m│\u001b[0m \u001b[0m(internal/process/task_queues.js:97:5)\",\"status\":\"terminated\"}\n\u001b[31m│\u001b[0m \u001b[0mDec 06 12:33:50 ip-172-31-2-224 cml.sh[3549]:\n\u001b[31m│\u001b[0m \u001b[0m{\"level\":\"info\",\"message\":\"waiting 20 seconds before exiting...\"}\n\u001b[31m│\u001b[0m \u001b[0m\n\u001b[31m╵\u001b[0m\u001b[0m\n\n    at /usr/lib/node_modules/@dvcorg/cml/src/utils.js:14:27\n    at ChildProcess.exithandler (node:child_process:404:5)\n    at ChildProcess.emit (node:events:390:28)\n    at maybeClose (node:internal/child_process:1064:16)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)","status":"terminated"}
{"level":"info","message":"waiting 20 seconds before exiting..."}

[0x2b3bfa0]

Failed preparing Gitlab runner: Forbidden

Expired token or something?

[DavidGOrtega]

Updated token

[casperdcl]

The Error: Failed preparing Gitlab runner: Forbidden is rather hidden. Wonder if we can improve the messaging a bit :)

[mikolajpabiszczak]

Have a similar problem on GitHub, when specifying the instance type to g3.4xlarge (note that I have no problems with, e.g., g3s.xlarge instances).

Takeout of the log:

2022-02-11T16:08:19.9963307Z {"level":"warn","message":"ignoring RUNNER_NAME environment variable, use CML_RUNNER_NAME or --name instead"}
2022-02-11T16:08:24.5146428Z {"level":"info","message":"Preparing workdir /home/runner/.cml/cml-ohlqr0bdbb..."}
2022-02-11T16:08:24.5154973Z {"level":"info","message":"Deploying cloud runner plan..."}
2022-02-11T16:08:24.5159001Z {"level":"info","message":"Terraform apply..."}
2022-02-11T16:08:35.1654231Z {"level":"error","message":"terraform -chdir='/home/runner/.cml/cml-ohlqr0bdbb' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  + create\n\nTerraform will perform the following actions:\n\n  # iterative_cml_runner.runner will be created\n  + resource \"iterative_cml_runner\" \"runner\" {\n      + cloud                = \"aws\"\n      + docker_volumes       = []\n      + driver               = \"github\"\n      + id                   = (known after apply)\n      + idle_timeout         = 216000\n      + instance_hdd_size    = 35\n      + instance_ip          = (known after apply)\n      + instance_launch_time = (known after apply)\n      + instance_type        = \"g3.4xlarge\"\n      + labels               = \"training-fold\"\n      + name                 = \"cml-ohlqr0bdbb\"\n      + region               = \"eu-west-1\"\n      + repo                 = \"https://github.com/medsensio/abnormal-lung-sounds\"\n      + single               = true\n      + spot                 = true\n      + spot_price           = -1\n      + ssh_public           = (known after apply)\n      + token                = (sensitive value)\n    }\n\nPlan: 1 to add, 0 to change, 0 to destroy.\niterative_cml_runner.runner: Creating...\n\n\t╷\n│ Error: Failed creating the machine: Not able to decode: waiter state transitioned to Failure\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n","stack":"Error: terraform -chdir='/home/runner/.cml/cml-ohlqr0bdbb' apply -auto-approve\n\t\nTerraform used the selected providers to generate the following execution\nplan. Resource actions are indicated with the following symbols:\n  + create\n\nTerraform will perform the following actions:\n\n  # iterative_cml_runner.runner will be created\n  + resource \"iterative_cml_runner\" \"runner\" {\n      + cloud                = \"aws\"\n      + docker_volumes       = []\n      + driver               = \"github\"\n      + id                   = (known after apply)\n      + idle_timeout         = 216000\n      + instance_hdd_size    = 35\n      + instance_ip          = (known after apply)\n      + instance_launch_time = (known after apply)\n      + instance_type        = \"g3.4xlarge\"\n      + labels               = \"training-fold\"\n      + name                 = \"cml-ohlqr0bdbb\"\n      + region               = \"eu-west-1\"\n      + repo                 = \"https://github.com/medsensio/abnormal-lung-sounds\"\n      + single               = true\n      + spot                 = true\n      + spot_price           = -1\n      + ssh_public           = (known after apply)\n      + token                = (sensitive value)\n    }\n\nPlan: 1 to add, 0 to change, 0 to destroy.\niterative_cml_runner.runner: Creating...\n\n\t╷\n│ Error: Failed creating the machine: Not able to decode: waiter state transitioned to Failure\n│ \n│   with iterative_cml_runner.runner,\n│   on main.tf line 14, in resource \"iterative_cml_runner\" \"runner\":\n│   14: resource \"iterative_cml_runner\" \"runner\" {\n│ \n╵\n\n    at /usr/local/lib/node_modules/@dvcorg/cml/src/utils.js:21:27\n    at ChildProcess.exithandler (node:child_process:404:5)\n    at ChildProcess.emit (node:events:390:28)\n    at maybeClose (node:internal/child_process:1064:16)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:301:5)","status":"terminated"}
2022-02-11T16:08:35.1660770Z {"level":"info","message":"waiting 20 seconds before exiting..."}
2022-02-11T16:08:55.1917329Z ##[error]Process completed with exit code 1.

Additionally, in the AWS Console I can see that the corresponding instance request has status cancelled with status bad-parameters.