NAS Restore fail

[levindecaro]

problem

When restoring a VM, it complete instantly without errors, but nothing are being restored. Agent log shown I/O errors on a volumePath appear invalid.

2025-07-02 15:00:06,074 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:) Request:Seq 105-6349231049662267563:  { Cmd , MgmtId: 52239181762, via: 105, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.RestoreBackupCommand":{"vmName":"i-9-7072-VM","backupPath":"i-9-7072-VM/2025.07.02.14.58.03","backupRepoType":"nfs","backupRepoAddress":"secstor.apac-1b.example.com:/mnt/backupdg1/cloudstack-userbackup","volumePaths":["/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009","/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35"],"vmExists":"true","vmState":"Restoring","wait":"0","bypassHostMaintenance":"false"}}] }
2025-07-02 15:00:06,074 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:) Processing command: org.apache.cloudstack.backup.RestoreBackupCommand
2025-07-02 15:00:06,074 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Executing command [/bin/bash -c sudo mount -t nfs secstor.apac-1b.example.com:/mnt/backupdg1/cloudstack-userbackup /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335 ].
2025-07-02 15:00:06,174 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Successfully executed process [155358] for command [/bin/bash -c sudo mount -t nfs secstor.apac-1b.example.com:/mnt/backupdg1/cloudstack-userbackup /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335 ].
2025-07-02 15:00:06,175 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Executing command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/root.aac01ea4-e2ef-497f-925d-1a6467cef009.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009 ].
2025-07-02 15:00:06,181 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Execution of process [155373] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/root.aac01ea4-e2ef-497f-925d-1a6467cef009.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009 ] failed.
2025-07-02 15:00:06,181 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Exit value of process [155373] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/root.aac01ea4-e2ef-497f-925d-1a6467cef009.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009 ] is [3].
2025-07-02 15:00:06,181 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Process [155373] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/root.aac01ea4-e2ef-497f-925d-1a6467cef009.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/aac01ea4-e2ef-497f-925d-1a6467cef009 ] encountered the error: [rsync: [Receiver] change_dir#3 "/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b" failed: No such file or directory (2)rsync error: errors selecting input/output files, dirs (code 3) at main.c(818) [Receiver=3.2.5]].
2025-07-02 15:00:06,181 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Executing command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/datadisk.18d02046-6c8c-4c55-bb76-c82a3eaf7d35.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35 ].
2025-07-02 15:00:06,187 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Execution of process [155375] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/datadisk.18d02046-6c8c-4c55-bb76-c82a3eaf7d35.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35 ] failed.
2025-07-02 15:00:06,187 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Exit value of process [155375] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/datadisk.18d02046-6c8c-4c55-bb76-c82a3eaf7d35.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35 ] is [3].
2025-07-02 15:00:06,187 WARN  [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Process [155375] for command [/bin/bash -c rsync -az /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335/i-9-7072-VM/2025.07.02.14.58.03/datadisk.18d02046-6c8c-4c55-bb76-c82a3eaf7d35.qcow2 /mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b/18d02046-6c8c-4c55-bb76-c82a3eaf7d35 ] encountered the error: [rsync: [Receiver] change_dir#3 "/mnt/7178c5ce-bc96-4f6a-a83a-1024d025717b" failed: No such file or directory (2)rsync error: errors selecting input/output files, dirs (code 3) at main.c(818) [Receiver=3.2.5]].
2025-07-02 15:00:06,187 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Executing command [/bin/bash -c sudo umount /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335 ].
2025-07-02 15:00:06,214 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-5:[]) (logid:) Processing command: com.cloud.agent.api.GetHostStatsCommand
2025-07-02 15:00:06,221 DEBUG [utils.script.Script] (AgentRequest-Handler-3:[]) (logid:) Successfully executed process [155377] for command [/bin/bash -c sudo umount /usr/share/cloudstack-agent/tmp/csbackup.fOEkP12089916377865461335 ].
2025-07-02 15:00:06,221 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-3:[]) (logid:) Seq 105-6349231049662267563:  { Ans: , MgmtId: 52239181762, via: 105, Ver: v1, Flags: 110, [{"org.apache.cloudstack.backup.BackupAnswer":{"result":"true","wait":"0","bypassHostMaintenance":"false"}}] }

versions

ACS: 4.20.1
OS: OL9 x86_64
hypervisors: KVM
storage: NFS(SharedMountPoint)
network: vxlan

The steps to reproduce the bug

1. Backup a running VM
2. Stop the VM
3. Restore the VM

What to do about it?

No response