NAS bacup fail on stopped VM

[levindecaro]

problem

An error occur when backup a stopped VM on NAS backup, same VM are able to backup when it is running.

Management Log

2025-07-02 14:48:37,464 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Executing AsyncJob {"accountId":9,"cmd":"org.apache.cloudstack.api.command.user.backup.CreateBackupCmd","cmdInfo":"{\"virtualmachineid\":\"627690b0-ec96-4eb4-a567-66121e2e1268\",\"response\":\"json\",\"ctxUserId\":\"9\",\"sessionkey\":\"EmegSGhP3Js2friBxUVZWYFh29Y\",\"httpmethod\":\"GET\",\"ctxStartEventId\":\"934564\",\"id\":\"7072\",\"ctxDetails\":\"{\\\"interface com.cloud.vm.VirtualMachine\\\":\\\"627690b0-ec96-4eb4-a567-66121e2e1268\\\"}\",\"ctxAccountId\":\"9\",\"cmdEventType\":\"BACKUP.CREATE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":138342,"initMsid":52239181762,"instanceId":7072,"instanceType":"Backup","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":9,"uuid":"e37acb9f-fa3f-4e1c-a37f-3d3f8a35ec4f"}
2025-07-02 14:48:37,467 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Access to Account [{"accountName":"test1.test","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}] granted to Account [{"accountName":"test1.test","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}] by DomainChecker on behalf of user test1.test
2025-07-02 14:48:37,478 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Access to VM instance {"id":7072,"instanceName":"i-9-7072-VM","state":"Stopped","type":"User","uuid":"627690b0-ec96-4eb4-a567-66121e2e1268"} granted to Account [{"accountName":"test1.test","id":9,"uuid":"d09266b9-4e22-40b8-a013-da17fc4a3cac"}] by DomainChecker on behalf of user test1.test
2025-07-02 14:48:37,501 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Wait time setting on org.apache.cloudstack.backup.TakeBackupCommand is 21600 seconds
2025-07-02 14:48:37,502 DEBUG [c.c.a.m.ClusteredAgentAttache] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Seq 105-6349231049662267510: Routed from 52239181762
2025-07-02 14:48:37,502 DEBUG [c.c.a.t.Request] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Seq 105-6349231049662267510: Sending  { Cmd , MgmtId: 52239181762, via: 105(compute-9.apac-1b.example.com), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-9-7072-VM","backupPath":"i-9-7072-VM/2025.07.02.14.48.37","backupRepoType":"nfs","backupRepoAddress":"secstor.apac-1b.example.com:/mnt/backupdg1/cloudstack-userbackup","volumePaths":["/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009","/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35"],"wait":"0","bypassHostMaintenance":"false"}}] } 
2025-07-02 14:48:37,704 DEBUG [c.c.a.t.Request] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Seq 105-6349231049662267510: Received:  { Ans: , MgmtId: 52239181762, via: 105(compute-9.apac-1b.example.com), Ver: v1, Flags: 110, { Answer } }
2025-07-02 14:48:37,704 WARN  [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Skipping delivering event Event {"description":"{\"event\":\"BACKUP.CREATE\",\"status\":\"Completed\"}","eventId":null,"eventType":"BACKUP.CREATE","eventUuid":null,"resourceType":null,"resourceUUID":null} to any webhook as account ID is missing
2025-07-02 14:48:37,704 WARN  [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342, ctx-3ea57770]) (logid:e37acb9f) Failed to publish event [category: ActionEvent, type: BACKUP.CREATE] on bus webhookEventBus
2025-07-02 14:48:37,723 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Complete async job-138342, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"class com.cloud.agent.api.Answer cannot be cast to class org.apache.cloudstack.backup.BackupAnswer (com.cloud.agent.api.Answer and org.apache.cloudstack.backup.BackupAnswer are in unnamed module of loader 'app')"}
2025-07-02 14:48:37,724 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Publish async job-138342 complete on message bus
2025-07-02 14:48:37,724 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Wake up jobs related to job-138342
2025-07-02 14:48:37,724 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Update db status for job-138342
2025-07-02 14:48:37,726 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Wake up jobs joined with job-138342 and disjoin all subjobs created from job- 138342
2025-07-02 14:48:37,733 DEBUG [c.c.a.ApiServer] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Retrieved cmdEventType from job info: BACKUP.CREATE
2025-07-02 14:48:37,736 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Done executing org.apache.cloudstack.api.command.user.backup.CreateBackupCmd for job-138342
2025-07-02 14:48:37,737 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-46:[ctx-87ec267f, job-138342]) (logid:e37acb9f) Remove job-138342 from job monitoring

versions

ACS: 4.20.1
OS: OL9 x86_64
hypervisors: KVM
storage: NFS
network: vxlan

The steps to reproduce the bug

1. Setup a NAS backup offering
2. Run a backup for a stopped VM

What to do about it?

No response