Webhook-go returns 500 error code on delete event if queue is disabled

[posledov]

While queue is disabled the webhook retruns 500 response code on delete event

# webhook-go -v
webhook-go version 2.9.0

Queue is NOT enabled (response code is 500)

Request headers:

Request URL: http://puppet.local.lan:4000/api/v1/r10k/environment
Request method: POST
Content-Type: application/json
X-GitHub-Delivery: 497cc251-78dd-4b46-8f90-9b334381a721
X-GitHub-Event: delete
X-GitHub-Event-Type: delete
X-Gitea-Delivery: 497cc251-78dd-4b46-8f90-9b334381a721
X-Gitea-Event: delete
X-Gitea-Event-Type: delete
X-Gitea-Signature: 
X-Gogs-Delivery: 497cc251-78dd-4b46-8f90-9b334381a721
X-Gogs-Event: delete
X-Gogs-Event-Type: delete
X-Gogs-Signature: 
X-Hub-Signature: sha1=
X-Hub-Signature-256: sha256=

Response body:

"ERROR\t -\u003e Environment(s) 'test2' cannot be found in any source and will not be deployed.\nINFO\t -\u003e Removing unmanaged path /etc/puppetlabs/code/environments/test2\n"

Webhook-go logs:

бер 07 16:12:12 puppet.local.lan webhook-go[543857]: time="2025-03-07T16:12:12+02:00" level=error msg="failed to execute local command `[/opt/puppetlabs/puppet/bin/r10k deploy environment test2 --config=/etc/puppetlabs/r10k/r10k.yaml --verbose --modules]` with error: `exit status 1` `ERROR\t -> Environment(s) 'test2' cannot be found in any source and will not be deployed.\nINFO\t -> Removing unmanaged path /etc/puppetlabs/code/environments/test2\n`"

бер 07 16:12:12 puppet.local.lan webhook-go[543857]: [GIN] 2025/03/07 - 16:12:12 | 500 |  1.879172362s |  192.168.13.130 | POST     "/api/v1/r10k/environment"

Queue is enabled (response code is 202)

Request headers:

Request URL: http://puppet.local.lan:4000/api/v1/r10k/environment
Request method: POST
Content-Type: application/json
X-GitHub-Delivery: 757cc692-907e-4bb3-a587-9be9157bc672
X-GitHub-Event: delete
X-GitHub-Event-Type: delete
X-Gitea-Delivery: 757cc692-907e-4bb3-a587-9be9157bc672
X-Gitea-Event: delete
X-Gitea-Event-Type: delete
X-Gitea-Signature: 
X-Gogs-Delivery: 757cc692-907e-4bb3-a587-9be9157bc672
X-Gogs-Event: delete
X-Gogs-Event-Type: delete
X-Gogs-Signature: 
X-Hub-Signature: sha1=
X-Hub-Signature-256: sha256=

Response body:

{"Id":"f25d7996-fb6d-11ef-84a3-525400f5d4fb","Name":"test","CommandType":"env","AddedAt":"2025-03-07T18:05:10.799401672+02:00","StartedAt":"0001-01-01T00:00:00Z","FinishedAt":"0001-01-01T00:00:00Z","Command":["/opt/puppetlabs/puppet/bin/r10k","deploy","environment","test","--config=/etc/puppetlabs/r10k/r10k.yaml","--verbose","--modules"],"Response":null,"State":"added"}

Webhook-go logs:

бер 07 18:05:10 puppet.local.lan webhook-go[549403]: [GIN] 2025/03/07 - 18:05:10 | 202 |     162.517µs |  192.168.13.130 | POST     "/api/v1/r10k/environment"

бер 07 18:05:10 puppet.local.lan webhook-go[549403]: time="2025-03-07T18:05:10+02:00" level=info msg="Worker picked Job f25d7996-fb6d-11ef-84a3-525400f5d4fb"

бер 07 18:05:13 puppet.local.lan webhook-go[549403]: time="2025-03-07T18:05:13+02:00" level=error msg="failed to execute local command `[/opt/puppetlabs/puppet/bin/r10k deploy environment test --config=/etc/puppetlabs/r10k/r10k.yaml --verbose --modules]` with error: `exit status 1` `ERROR\t -> Environment(s) 'test' cannot be found in any source and will not be deployed.\nINFO\t -> Removing unmanaged path /etc/puppetlabs/code/environments/test\n`"

Perhaps this problem is more related to the r10k, which should return 0 instead of 1 when attempting to deploy a non-existent branch, but I think that when deleting a branch, webhook-go could deploy not that branch itself, but the default_branch, which will also lead to the deletion of environments whose branches are missing in the git repository:

# ls -l /etc/puppetlabs/code/environments/
загалом 12
drwxr-xr-x 8 root root 4096 бер  7 17:52 production
drwxr-xr-x 8 root root 4096 бер  7 17:52 test1
drwxr-xr-x 8 root root 4096 бер  7 17:52 test2

# /opt/puppetlabs/puppet/bin/r10k deploy environment production --config=/etc/puppetlabs/r10k/r10k.yaml --verbose
INFO	 -> Deploying environment /etc/puppetlabs/code/environments/production
INFO	 -> Environment production is now at 8f184d7c768b0378c776c250417f83ffe743c9d1
INFO	 -> Using Puppetfile '/etc/puppetlabs/code/environments/production/Puppetfile'
INFO	 -> Removing unmanaged path /etc/puppetlabs/code/environments/test1
INFO	 -> Removing unmanaged path /etc/puppetlabs/code/environments/test2

# echo $?
0

[dhollinger]

That does appear to be an issue with the r10k command's return code. Likely a better option would be to only error on an actual HTTP error and not when the command is executed.

[elfranne]

Not sure it is relevant but the webhook would reply 500 for all events unless i have the queue enabled.

[dhollinger]

@elfranne Do you have any logs for those 500s? This particular ticket is related to how r10k itself resolves when it deletes a branch which leads to a 500 in the webhook due to the cli exit code

[elfranne]

@dhollinger should I open a new issue instead ?