PowerFlex/ScaleIO - MDM and host SDC connection enhancements #11047

sureshanaparti · 2025-06-17T07:00:03Z

Description

This PR enhances the PowerFlex/ScaleIO MDM and host SDC connections, includes the following changes (and some code improvements).

Introduced timeout configuration 'powerflex.mdm.change.apply.wait' (Default value: 1000 ms) at zone scope, to wait after MDM addition, and before & after MDM removal changes made on Host with ScaleIO SDC. Also, Changes to apply the wait time after making MDM changes for ScaleIO in prepare and unprepare logic.
Introduced configuration flag 'powerflex.block.sdc.unprepare' (Default is false) at zone scope, to enable/disable blocking unprepare ScaleIO SDC connection when SDC client restart required (upon PowerFlex MDM removal i.e. no support for --remove_mdm in drv_cfg cmd) and there are volumes attached to the Host. Added validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted.
Introduced configuration flag 'powerflex.mdm.validate.on.connect' (Default is false) at zone scope, to enable/disable validation of MDM addresses on Host, in the Configuration File and in CLI cmd (drv_cfg --query_mdms) output matches or not, during storage pool registration in agent.
Added detection of MDM removal support via CLI. If MDM removal support via CLI supported then use CLI, Otherwise fall back to edit drv_cfg.txt and restart scini as earlier. Tested with /opt/emc/scaleio/sdc/bin/drv_cfg --version: DellEMC PowerFlex Version: R3_6.4000.124, with cmd: /opt/emc/scaleio/sdc/bin/drv_cfg --remove_mdm.
Added agent property 'powerflex.sdc.service.wait' for the time (in secs) to wait after SDC service start/restart/stop, and retries to fetch SDC id/guid.
Updated to allow unprepare SDC when there are no volumes mapped on the host for other connected pools (with same SDC Id, i.e pools of same PowerFlex storage cluster).

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)
build/CI
test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

Tested the new settings, and PowerFlex SDC connections (MDM add/remove) with VM & Volume operations.

How did you try to break this feature and the system with this change?

sureshanaparti · 2025-06-17T07:05:11Z

@blueorangutan package

blueorangutan · 2025-06-17T07:06:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

codecov · 2025-06-17T07:12:24Z

Codecov Report

Attention: Patch coverage is 26.13982% with 243 lines in your changes missing coverage. Please review.

Project coverage is 16.58%. Comparing base (be22bfe) to head (ef3c3f1).
Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
...cloudstack/storage/datastore/util/ScaleIOUtil.java	28.85%	99 Missing and 7 partials ⚠️
.../hypervisor/kvm/storage/ScaleIOStorageAdaptor.java	25.00%	57 Missing and 12 partials ⚠️
...re/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java	14.81%	21 Missing and 2 partials ⚠️
...orage/datastore/manager/ScaleIOSDCManagerImpl.java	0.00%	21 Missing ⚠️
...torage/datastore/provider/ScaleIOHostListener.java	0.00%	12 Missing ⚠️
...s/src/main/java/com/cloud/utils/script/Script.java	0.00%	10 Missing ⚠️
...rapper/LibvirtModifyStoragePoolCommandWrapper.java	77.77%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               main   #11047    +/-   ##
==========================================
  Coverage     16.57%   16.58%            
- Complexity    13971    13989    +18     
==========================================
  Files          5743     5745     +2     
  Lines        510648   511093   +445     
  Branches      62105    62170    +65     
==========================================
+ Hits          84641    84753   +112     
- Misses       416534   416854   +320     
- Partials       9473     9486    +13

Flag	Coverage Δ
uitests	`3.91% <ø> (+<0.01%)`	⬆️
unittests	`17.47% <26.13%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull Request Overview

This PR enhances the handling of ScaleIO MDM and host SDC connections by introducing new configuration keys for timeouts, validation, and blocking behavior along with updating the underlying command execution and logging mechanisms.

Introduced new configuration keys, including MdmsChangeApplyTimeout, ValidateMdmsOnConnect, and BlockSdcUnprepareIfRestartNeededAndVolumesAreAttached.
Updated MDM add/remove logic to use varargs and improved command execution via Script.executeCommand.
Adjusted test cases and adapter logic to account for the new ScaleIO configurations.

Reviewed Changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
utils/src/main/java/com/cloud/utils/script/Script.java	Added a new executeCommand(String) method for command execution including stdout/stderr handling.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java	Refactored MDM add/remove methods and updated command templates, patterns, and file read operations.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManagerImpl.java	Updated configuration details sent to hosts by including new timeout settings.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/manager/ScaleIOSDCManager.java	Introduced new config key definitions and updated the getConfigKeys() return values.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/ScaleIOPrimaryDataStoreLifeCycle.java	Injected the new configuration details during maintain and cancellation procedures.
plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/client/ScaleIOGatewayClient.java	Added a short documentation comment for STORAGE_POOL_MDMS.
plugins/hypervisors/kvm/src/test/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptorTest.java	Updated test mocks to reflect changes in command executions for MDM removal.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/storage/ScaleIOStorageAdaptor.java	Introduced validation of MDM state and timeout application after MDM changes.
plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtModifyStoragePoolCommandWrapper.java	Wrapped storage pool creation in a try/catch block to better handle CloudRuntimeException.

Comments suppressed due to low confidence (1)

plugins/storage/volume/scaleio/src/main/java/org/apache/cloudstack/storage/datastore/util/ScaleIOUtil.java:315

[nitpick] The ordering of stdout and stderr here is reversed compared to other usages of Script.executeCommand; consider using a consistent ordering (stdout as first, stderr as second) to avoid confusion.

String stdErr = result.first(); String stdOut = result.second();

utils/src/main/java/com/cloud/utils/script/Script.java

blueorangutan · 2025-06-17T08:01:11Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 13811

sureshanaparti · 2025-06-17T08:13:05Z

@blueorangutan package

blueorangutan · 2025-06-17T08:14:05Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-17T09:38:10Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13812

harikrishna-patnala

code LGTM

sureshanaparti · 2025-06-19T06:12:37Z

@blueorangutan test

blueorangutan · 2025-06-19T06:14:03Z

@sureshanaparti a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-06-19T06:20:24Z

[SF] Trillian Build Failed (tid-13562)

harikrishna-patnala · 2025-06-23T06:21:07Z

@blueorangutan test

blueorangutan · 2025-06-23T06:22:03Z

@harikrishna-patnala a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

blueorangutan · 2025-06-24T02:36:46Z

[SF] Trillian test result (tid-13586)
Environment: kvm-ol8 (x2), Advanced Networking with Mgmt server ol8
Total time taken: 69826 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr11047-t13586-kvm-ol8.zip
Smoke tests completed. 136 look OK, 5 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestAccounts>:setup	`Error`	0.00	test_accounts.py
ContextSuite context=TestAddVmToSubDomain>:setup	`Error`	0.00	test_accounts.py
test_DeleteDomain	`Error`	15.92	test_accounts.py
test_forceDeleteDomain	`Failure`	15.66	test_accounts.py
ContextSuite context=TestRemoveUserFromAccount>:setup	`Error`	16.38	test_accounts.py
ContextSuite context=TestTemplateHierarchy>:setup	`Error`	1537.55	test_accounts.py
ContextSuite context=TestDeployVmWithAffinityGroup>:setup	`Error`	0.00	test_affinity_groups_projects.py
ContextSuite context=TestAnnotations>:setup	`Error`	0.00	test_annotations.py
ContextSuite context=TestAsyncJob>:setup	`Error`	0.00	test_async_job.py
ContextSuite context=TestClusterDRS>:setup	`Error`	0.00	test_cluster_drs.py

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java

sureshanaparti · 2025-06-25T06:41:19Z

@blueorangutan package

blueorangutan · 2025-06-25T06:42:07Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-06-25T08:04:14Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 13907

rohityadavcloud · 2025-06-26T07:52:12Z

@sureshanaparti as these are general bugfix/improvements for powerflex storage, should these be also applicable for 4.20 branch (esp these changes are scaleio/powerflex storage plugin related with no DB changes)?

sureshanaparti · 2025-06-27T14:26:16Z

@sureshanaparti as these are general bugfix/improvements for powerflex storage, should these be also applicable for 4.20 branch (esp these changes are scaleio/powerflex storage plugin related with no DB changes)?

@rohityadavcloud There are no DB changes, but changes in host SDC connection behavior earlier in this PR: #9903 (already part of main), where the SDC connection is controlled using MDMs addition/removal instead of SDC service scini start/stop. This PR changes are on top of these, and introduces some configurations to validate the MDMs and apply timeout after add/remove MDMs. I think, it's better to keep this new behavior with main itself.
A minor improvement (of keeping wait time after SDC service start/restart/stop, and retries to fetch SDC id/guid) is applicable for the old behavior as well, so can go in 4.20 branch. I've raised a separate PR for it here: #11099

sureshanaparti · 2025-06-27T14:27:56Z

@blueorangutan test

…/unprepare, validate Storage Pool can be created in Agent. - Implemented validation to fail Host disconnect from Storage Pool if there are Volumes attached and SDC client MDM removal requires scini service to be restarted - Implemented Storage Pool validation by checking whether MDM addresses from configuration file and from memory (using CLI) matches, otherwise file ModifyStoragePool command. - Introduced configuration key to apply timeout after making MDM changes for ScaleIO: powerflex.mdm.change.apply.timeout.ms (default 1000ms) - Implemented logic to apply timeout after making MDM changes for ScaleIO in prepare and unprepare logic - Added detection of MDM removal support via CLI - If MDM removal support via CLI supported then use CLI, fall back to edit drv_cfg.txt and restart scini instead

… pool when SDC client restart required and there are volumes attached to the Host

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

…tack/storage/datastore/util/ScaleIOUtil.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

… validation

…y_guid, --rescan in drv_cfg cmd as it is not supported. --file parameter is supported with --add_mdm, --mod_mdm_ip, --remove_mdm, --set_guid, --set_mdm_password, --reset_mdm_password

…h SDC id/guid and some code improvements

…cs) to wait after SDC service start/restart/stop

…other connected pools (with same SDC Id, i.e pools of same PowerFlex storage cluster)

sureshanaparti · 2025-07-03T12:43:02Z

@blueorangutan package

blueorangutan · 2025-07-03T12:44:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-03T13:56:22Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✖️ debian ✔️ suse15. SL-JID 14008

sureshanaparti · 2025-07-03T15:28:24Z

@blueorangutan package

blueorangutan · 2025-07-03T15:30:03Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-03T17:22:06Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14013

…000 ms

sureshanaparti · 2025-07-07T15:13:02Z

@blueorangutan package

blueorangutan · 2025-07-07T15:14:04Z

@sureshanaparti a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2025-07-07T17:03:03Z

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ debian ✔️ suse15. SL-JID 14072

boring-cyborg bot added component:kvm component:primary-storage component:storage labels Jun 17, 2025

DaanHoogland added this to the 4.21.0 milestone Jun 17, 2025

sureshanaparti requested review from harikrishna-patnala and Copilot June 17, 2025 07:55

Copilot AI reviewed Jun 17, 2025

View reviewed changes

utils/src/main/java/com/cloud/utils/script/Script.java Outdated Show resolved Hide resolved

harikrishna-patnala approved these changes Jun 19, 2025

View reviewed changes

sureshanaparti commented Jun 25, 2025

View reviewed changes

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java Outdated Show resolved Hide resolved

sureshanaparti commented Jun 25, 2025

View reviewed changes

...leio/src/main/java/org/apache/cloudstack/storage/datastore/provider/ScaleIOHostListener.java Outdated Show resolved Hide resolved

sureshanaparti changed the title ~~PowerFlex/ScaleIO MDM and host SDC connection enhancements~~ PowerFlex/ScaleIO - MDM and host SDC connection enhancements Jun 27, 2025

mprokopchuk and others added 17 commits July 3, 2025 15:48

Add feature flag to enable/disable blocking unprepare ScaleIO storage…

b13712c

… pool when SDC client restart required and there are volumes attached to the Host

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

5674f56

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

b7b40d9

…tack/storage/datastore/manager/ScaleIOSDCManager.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

Update plugins/storage/volume/scaleio/src/main/java/org/apache/clouds…

12eea4a

…tack/storage/datastore/util/ScaleIOUtil.java Co-authored-by: Suresh Kumar Anaparti <[email protected]>

- Addressed code review comments: Added checking mdm removal response…

4636ae9

… validation

Updated scope to Zone for the new powerflex settings

b88048b

Get powerflex storage config settings from zone, and some fixes

018cf17

Removed --file parameter used with --query_mdms, --query_vols, --quer…

434862b

…y_guid, --rescan in drv_cfg cmd as it is not supported. --file parameter is supported with --add_mdm, --mod_mdm_ip, --remove_mdm, --set_guid, --set_mdm_password, --reset_mdm_password

rebase fixes

8cdefca

typo fix

789aea6

review fix

ad41f3f

Added wait time after SDC service start/restart/stop, retries to fetc…

44af31c

…h SDC id/guid and some code improvements

Apply suggestions from code review

ba8a6bd

Added agent property 'powerflex.sdc.service.wait' for the time (in se…

67c47c2

…cs) to wait after SDC service start/restart/stop

Allow unprepare sdc when there are no volumes mapped on the host for …

778a873

…other connected pools (with same SDC Id, i.e pools of same PowerFlex storage cluster)

Fix wait time for remove PowerFlex MDMs, and some code improvements

9befbd3

sureshanaparti force-pushed the scaleio_mdm_enhancements branch from 38db39e to 9befbd3 Compare July 3, 2025 12:41

boring-cyborg bot added the component:agent label Jul 3, 2025

sureshanaparti added 2 commits July 7, 2025 16:52

Change default value of config 'powerflex.mdm.change.apply.wait' to 3…

21d5813

…000 ms

Fix sdc usage cmd (drv_cfg) result, and log command result

ef3c3f1

PowerFlex/ScaleIO - MDM and host SDC connection enhancements #11047

Are you sure you want to change the base?

PowerFlex/ScaleIO - MDM and host SDC connection enhancements #11047

Conversation

sureshanaparti commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

Uh oh!

sureshanaparti commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

codecov bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

sureshanaparti commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

blueorangutan commented Jun 17, 2025

Uh oh!

harikrishna-patnala left a comment

Choose a reason for hiding this comment

Uh oh!

sureshanaparti commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

blueorangutan commented Jun 19, 2025

Uh oh!

harikrishna-patnala commented Jun 23, 2025

Uh oh!

blueorangutan commented Jun 23, 2025

Uh oh!

blueorangutan commented Jun 24, 2025

Uh oh!

Uh oh!

Uh oh!

sureshanaparti commented Jun 25, 2025

Uh oh!

blueorangutan commented Jun 25, 2025

Uh oh!

blueorangutan commented Jun 25, 2025

Uh oh!

rohityadavcloud commented Jun 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sureshanaparti commented Jun 27, 2025

Uh oh!

sureshanaparti commented Jun 27, 2025

Uh oh!

sureshanaparti commented Jul 3, 2025

Uh oh!

blueorangutan commented Jul 3, 2025

Uh oh!

blueorangutan commented Jul 3, 2025

Uh oh!

sureshanaparti commented Jul 3, 2025

Uh oh!

blueorangutan commented Jul 3, 2025

Uh oh!

blueorangutan commented Jul 3, 2025

Uh oh!

sureshanaparti commented Jul 7, 2025

sureshanaparti commented Jun 17, 2025 •

edited

Loading

codecov bot commented Jun 17, 2025 •

edited

Loading

rohityadavcloud commented Jun 26, 2025 •

edited

Loading