4.20.1 Can't attach addon disk to VM running on VMware - vCenter

[leduyquy]

problem

In new version Cloudstack: 4.20.1.0, I see it support VMware 8U3 so I did upgrade vCenter to 8.0.3.00500, But affter upgrade vCenter i testing attach disk to VM running on it. everything is normal when i rollback vcenter to old version:

Log on management:

2025-06-23 09:50:37,646 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:167cef34) Add job-132862 into job monitoring
2025-06-23 09:50:37,649 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-51180:[ctx-356feba0, ctx-fb5487f9]) (logid:0d6e0b53) submit async job-132862, details: AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}
2025-06-23 09:50:37,650 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Executing AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}
2025-06-23 09:50:37,658 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,662 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VM {"id":1544,"instanceName":"i-10-1544-VM","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,664 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Account [Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}]] has access to resource.
2025-06-23 09:50:37,690 DEBUG [c.c.h.d.HypervisorCapabilitiesDaoImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Hypervisor capabilities for hypervisor: VMware, version: 8.0.2.0 can not be found. Trying to find capabilities for the parent version: 8.0.2
2025-06-23 09:50:37,696 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VolumeObject {"dataStore":"StoragePool {"id":2,"name":"vsanDatastore","poolType":"PreSetup","uuid":"7baf972d-35f4-394c-b9c3-0c3e13b4144a"}","volumeVO":"Volume {"id":4274,"instanceId":null,"name":"data","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","volumeType":"DATADISK"}"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,697 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Access to VM instance {"id":1544,"instanceName":"i-10-1544-VM","state":"Running","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"} granted to Account [{"accountName":"<>","id":10,"uuid":"e653412e-93cf-472e-a96b-8a0efd8924be"}] by DomainChecker on behalf of user <>
2025-06-23 09:50:37,703 INFO [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Trying to attach volume [VolumeObject {"dataStore":"StoragePool {"id":2,"name":"vsanDatastore","poolType":"PreSetup","uuid":"7baf972d-35f4-394c-b9c3-0c3e13b4144a"}","volumeVO":"Volume {"id":4274,"instanceId":null,"name":"data","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","volumeType":"DATADISK"}"}] to VM instance [VM instance {"id":1544,"instanceName":"i-10-1544-VM","state":"Running","type":"User","uuid":"5007e13f-28e0-4f9e-b191-428c381c9288"}], update async job-132862 [AsyncJob {"accountId":10,"cmd":"org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd","cmdInfo":"{"virtualmachineid":"5007e13f-28e0-4f9e-b191-428c381c9288","response":"json","ctxUserId":"60","sessionkey":"aZTCm6g7HBWrL9uH08ocEqYxnl8","httpmethod":"GET","ctxStartEventId":"1265802","id":"d4a8082f-d4e4-415f-878e-667cba85d103","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"d4a8082f-d4e4-415f-878e-667cba85d103\",\"interface com.cloud.vm.VirtualMachine\":\"5007e13f-28e0-4f9e-b191-428c381c9288\"}","ctxAccountId":"10","uuid":"d4a8082f-d4e4-415f-878e-667cba85d103","cmdEventType":"VOLUME.ATTACH"}","cmdVersion":0,"completeMsid":null,"created":null,"id":132862,"initMsid":345050445235,"instanceId":4274,"instanceType":"Volume","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"0654201d-f27e-4b7d-8d64-0a067bc74a80"}] progress status
2025-06-23 09:50:37,705 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Update async-job attachment, job-132862, instanceType: Volume, instanceId: 4274
2025-06-23 09:50:37,707 DEBUG [c.c.a.ApiServer] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Retrieved cmdEventType from job info: VOLUME.ATTACH
2025-06-23 09:50:37,733 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Sync job-132863 execution on object VmWorkJobQueue.1544
2025-06-23 09:50:37,762 DEBUG [c.c.s.VolumeApiServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) New job VmWorkJobVO : {, step: Starting, vmType: Instance, vmInstanceId: 1544, secondaryObjectIdentifier: nullAsyncJob {"accountId":10,"cmd":"com.cloud.vm.VmWorkAttachVolume","cmdInfo":"rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrQXR0YWNoVm9sdW1lB62v-WGH4hwCAAJMAAhkZXZpY2VJZHQAEExqYXZhL2xhbmcvTG9uZztMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAoAAAAAAAAAPAAAAAAAAAYIdAAUVm9sdW1lQXBpU2VydmljZUltcGxwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAABCy","cmdVersion":0,"completeMsid":null,"created":null,"id":132863,"initMsid":345050445235,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":60,"uuid":"8d75d03d-44d6-463a-9150-94f7f6589203"}}, result field: null
2025-06-23 09:50:40,786 WARN [o.a.c.m.w.WebhookServiceImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Skipping delivering event Event {"description":"{"details":"Volume Id: d4a8082f-d4e4-415f-878e-667cba85d103 VmId: 5007e13f-28e0-4f9e-b191-428c381c9288","event":"VOLUME.ATTACH","status":"Completed"}","eventId":null,"eventType":"VOLUME.ATTACH","eventUuid":null,"resourceType":"Volume","resourceUUID":null} to any webhook as account ID is missing
2025-06-23 09:50:40,787 WARN [o.a.c.f.e.EventDistributorImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862, ctx-1557a9bc]) (logid:0654201d) Failed to publish event [category: ActionEvent, type: VOLUME.ATTACH] on bus webhookEventBus
2025-06-23 09:50:40,802 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Unexpected exception while executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd com.cloud.utils.exception.CloudRuntimeException: Failed to attach volume data to VM <>
2025-06-23 09:50:40,803 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Complete async job-132862, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed to attach volume data to VM <>"}
2025-06-23 09:50:40,803 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Publish async job-132862 complete on message bus
2025-06-23 09:50:40,804 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Wake up jobs related to job-132862
2025-06-23 09:50:40,804 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Update db status for job-132862
2025-06-23 09:50:40,805 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Wake up jobs joined with job-132862 and disjoin all subjobs created from job- 132862
2025-06-23 09:50:40,813 DEBUG [c.c.a.ApiServer] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Retrieved cmdEventType from job info: VOLUME.ATTACH
2025-06-23 09:50:40,817 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Done executing org.apache.cloudstack.api.command.user.volume.AttachVolumeCmd for job-132862
2025-06-23 09:50:40,817 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:[ctx-9935ac14, job-132862]) (logid:0654201d) Remove job-132862 from job monitoring

Since i had to rollback to vcenter to secure workload i can only provide more information from old log

versions

Cloudstack: 4.20.1.0
vCenter: 8.0.3.00500, Build number: 24674346
VMware ESXi : 8.0.2, 22380479

The steps to reproduce the bug

1. Upgrade vCenter from 8.0.0 to 8.0.3
2. Attach addon disk VM
3. Error
   ...

What to do about it?

Attach addon disk working normal

[sureshanaparti]

@leduyquy any errors noticed in vcenter tasks / events during attach volume operation?

[kiranchavala]

@leduyquy, what type of primary storage are you using

I have tested with Nfs and the volume attachment works fine on 4.20.1 and 80u3

[leduyquy]

I using vsan for primary storage, I don't see any error during attach volume operator (I'm not sure about it either because I've rolled back it to old version)

[sureshanaparti]

ok @leduyquy , I've verified attach volume with NFS primary storage, in 4.20.1 environment and VMware vCenter: 8.0.3.00500, Build number: 24674346 (vCenter Server 8.0 Update 3e); VMware ESXi : 8.0.3, 24674464 (ESXi 8.0 P05 / ESXi 8.0 Update 3e). No issues noticed.

[winterhazel]

@leduyquy could you check:

• in CloudStack, what values do the settings rootDiskController and dataDiskController have in your VM?
• in vCenter, what disk controllers does your VM have?

[leduyquy]

Hi @winterhazel

in CloudStack, what values do the settings rootDiskController and dataDiskController have in your VM?

I set rootDiskController and dataDiskController to pvscsi

in vCenter, what disk controllers does your VM have?

at vcenter it is set: VMware Paravirtual