Article Number: 000200962 📠 Print

ECS: Solution to address CVE-2022-31231 security vulnerability on

3.5.x/3.6.x
Summary: Addresses an Improper Access Control in the Identity and Access Management (IAM) module. A
remote unauthenticated attacker could potentially exploit this vulnerability, leading to gain read access to
unauthorized data. This affects all ECS 3.5.x.x and ECS 3.6.x.x versions.

Audience Level: Customer

Article Content

Symptoms

CVE ID: CVE-2022-31231

Severity: Medium

Cause

Improper Access Control in the Identity and Access Management (IAM) module.

Resolution

Who should run this procedure?

Dell requests this procedure of upgrading xDoctor and installation of the patch be done by Customers. This is the quickest and safest
method as it avoids prolonged exposure to this vulnerability. All the steps are detailed in this KB. There is also a video guide which
can be followed to accompany this KB which resides at below link.

Impact of procedure:
Expect possible I/O timeouts while dataheadsvc services are restarted node by node. Applications should be accessing the cluster via a
Load balancer and must be able to handle the I/O timeout. A maintenance window is advised when performing this procedure.

CAS only buckets Exception:

If all buckets on a system are exclusively CAS highlighted below then it is not affected by this security vulnerability. It is therefore not
necessary to apply the patch and this KB does not have to be followed.

Command: svc_bucket list

Example:
 admin@ecs-n1:~> svc_bucket list
svc_bucket v1.0.33 (svc_tools v2.5.1) Started 2022-07-08 08:49:11

Bucket Temp
Replication Owner Owner
API FS Versioning Failed
Bucket Name Namespace Group User VDC
Type Enabled Enabled (TSO)

cas_bucket region_ns RG1 casuser VDC1

CAS false Disabled False
cas_bu region_ns RG1 cas_obj VDC1
CAS false Disabled False
test region_ns RG1 test1 VDC1
CAS false Disabled False
test_cas region_ns RG1 test_cas VDC1
CAS false Disabled False
test_bkt_cas region_ns RG1 user_test VDC1
CAS false Disabled False
Friday_cas region_ns RG1 Friday_cas VDC1
CAS false Disabled False

Time taken for the activity (Approximately):

A 60 second delay is set by default per node between service restarts. The number of nodes in a Virtual Data Center (VDC) multiplied by
60 seconds + 30 minutes for preparation, service stabilization, and post checks needed.

Examples:
An 48 node VDC system can take approximately 80 mins:
60 seconds X 48 (Number of VDC nodes) + 30 minutes (preparation) = 80 mins approx.

An 8 node VDC system can take approximately 40 mins:

60 seconds X 8 (Number of VDC nodes) + 30 minutes (preparation) = 40 mins approx.

Frequently Asked Questions (FAQ):

Q: Is the patch part of the xDoctor release?
A: The patch install script is part of xDoctor release 4.8-84 and higher. Instructions for the download of xDoctor and execution of patch
install are in the resolution steps.

Q: Can I update multiple VDCs in parallel?

A: No, patch 1 VDC at a time.

Q: If I upgrade ECS after running this procedure, do I rerun the procedure post upgrade?
A: No, if upgrading to a code version specified in DSA-2022-153 which has the permanent fix. Yes, if upgrading to a code
version not specified in this same DSA.

Q: Does the patch need to be reapplied on a system where it was previously installed after a node replacement, reimage, or expansion?
A: No, if the VDC is at the code version that is specified in DSA-2022-153 which has the permanent fix. Yes, if doing any of these actions
against a VDC running a code version not specified in this same DSA. Where the patch is required for these scenarios, the Dell engineer
in question will contact to inform that update is required.

Q: What if I'm only using legacy users and not using IAM?
A: Customers need to apply the patch regardless if using only legacy users and not IAM.

Q: What user should you be logged in as to execute all commands in this KB?
A: admin

Q: Does svc_patch have to be run on all racks or with a specialized MACHINES file where multiple racks in a VDC?
A: No, it autodetects if multiple racks exist and patches all nodes on all racks on that VDC.

Q: I notice the target xDoctor release is no longer 4.8-84.0. Why?

A: xDoctor releases occur frequently so it is always recommended to upgrade to highest released version. If however you have
previously ran the fix using 4.8-84.0 then system is fully protected against the vulnerability and does not need to be rerun.

Resolution Summary:

1. Upgrade your ECS xDoctor software to version 4.8-84.0 or later.

2. Run Prechecks.
3. Apply the system patch with svc_patch tool included with xDoctor.
4. Confirm that the fix has been applied.
5. Troubleshooting.

Resolution Steps:
1. Upgrade your ECS xDoctor software to latest Version available.

a. Check xDoctor version running on your system. If the version is 4.8-84.0 or later, move to step 2 "Run Prechecks". If not,
proceed with the steps below.

Command:
# sudo xdoctor --version

Example:
admin@node1:~> sudo xdoctor --version
4.8-84.0

b. Sign to the Dell Support Site, connect directly to this download link, search for xDoctor using the keyword search bar, and
click on xDoctor 4.8-84.0 RPM link to download. If you wish to view the Release notes, follow the Release Notes, select
Manuals and documents from the sidebar from where they should be available for download.
c. Once the RPM is downloaded, use any remote SCP program to upload the file to the /home/admin directory on the first ECS
node.
d. Once the upload is complete, SSH to the first node of the ECS system using admin.
e. Upgrade xDoctor on all the nodes with the newly distributed version.

Command:
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-84.0.noarch.rpm

Example:
admin@ecs-n1:~> sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-84.0.noarch.rpm
2022-07-04 07:41:49,209: xDoctor_4.8-83.0 - INFO : xDoctor Upgrader Instance (1:SFTP_ONLY)
2022-07-04 07:41:49,210: xDoctor_4.8-83.0 - INFO : Local Upgrade (/home/admin/xDoctor4ECS-4.8-
84.0.noarch.rpm)
2022-07-04 07:41:49,226: xDoctor_4.8-83.0 - INFO : Current Installed xDoctor version is 4.8-83.0
2022-07-04 07:41:49,242: xDoctor_4.8-83.0 - INFO : Requested package version is 4.8-84.0
2022-07-04 07:41:49,242: xDoctor_4.8-83.0 - INFO : Updating xDoctor RPM Package (RPM)
2022-07-04 07:41:49,293: xDoctor_4.8-83.0 - INFO : - Distribute package
2022-07-04 07:41:50,759: xDoctor_4.8-83.0 - INFO : - Install new rpm package
2022-07-04 07:42:04,401: xDoctor_4.8-83.0 - INFO : xDoctor successfully updated to version 4.8-84.0

f. If the environment is a multi-rack VDC, the new xDoctor package must be installed on the first node of each rack. To identify
these rack primaries, run the below command. In this instance, there are four racks and therefore four rack primaries
highlighted

1. Find the rack primary nodes

Command:
# svc_exec -m "ip address show private.4 |grep -w inet"

Example:
 admin@ecsnode1~> svc_exec -m "ip address show private.4 |grep -w inet"
svc_exec v1.0.2 (svc_tools v2.1.0) Started 2021-12-20 14:03:33

Output from node: r1n1 retval: 0

inet 169.254.1.1/16 brd 169.254.255.255 scope global private.4

Output from node: r2n1 retval: 0

inet 169.254.2.1/16 brd 169.254.255.255 scope global private.4

Output from node: r3n1 retval: 0

inet 169.254.3.1/16 brd 169.254.255.255 scope global private.4

Output from node: r4n1 retval: 0

inet 169.254.4.1/16 brd 169.254.255.255 scope global private.4
2. Copy the package from the first node of the system (R1N1) to the other rack primaries per below:

Example:
admin@ecs-n1: scp xDoctor4ECS-4.8-84.0.noarch.rpm 169.254.2.1:/home/admin/
xDoctor4ECS-4.8-84.0.noarch.rpm
100% 32MB 31.9MB/s 00:00
admin@ecsnode1~> scp xDoctor4ECS-4.8-84.0.noarch.rpm 169.254.3.1:/home/admin/
xDoctor4ECS-4.8-84.0.noarch.rpm
100% 32MB 31.9MB/s 00:00
admin@ecsnode1~> scp xDoctor4ECS-4.8-784.0.noarch.rpm 169.254.4.1:/home/admin/
xDoctor4ECS-4.8-84.0.noarch.rpm
100% 32MB 31.9MB/s 00:00
admin@ecsnode1~>

3. Per step 1 above, run the same xDoctor install command on each of the above rack primaries identified previously.

Command:
# sudo xdoctor --upgrade --local=/home/admin/xDoctor4ECS-4.8-84.0.noarch.rpm

2. Run Prechecks

a. Use svc_dt command to check if DTs are stable. DTs are stable if "Unready #" column shows 0. If yes, go to the next
check. If no, wait 15 minutes and check again. If DTs have not stabilized, open a service request with ECS support team.

Command:
# svc_dt check -b

Example:
 admin@ecs-n1: svc_dt check -b

svc_dt v1.0.27 (svc_tools v2.4.1) Started 2022-06-14 11:34:26

Date Total DT Unknown # Unready # RIS Fail # Dump Fail #

Check type Time since check Check successful

2022-06-14 11:34:09 1920 0 0 0 0

AutoCheck 0m 17s True
2022-06-14 11:32:59 1920 0 0 0 0
AutoCheck 1m 27s True
2022-06-14 11:31:48 1920 0 0 0 0
AutoCheck 2m 38s True
2022-06-14 11:30:38 1920 0 0 0 0
AutoCheck 3m 48s True
2022-06-14 11:29:28 1920 0 0 0 0
AutoCheck 4m 58s True
2022-06-14 11:28:18 1920 0 0 0 0
AutoCheck 6m 8s True
2022-06-14 11:27:07 1920 0 0 0 0
AutoCheck 7m 19s True
2022-06-14 11:25:57 1920 0 0 0 0
AutoCheck 8m 29s True
2022-06-14 11:24:47 1920 0 0 0 0
AutoCheck 9m 39s True
2022-06-14 11:23:37 1920 0 0 0 0
AutoCheck 10m 49s True
b. Use svc_patch command to validate that all nodes are online. If yes, go to the next step. If no, investigate the reason, bring it
back online, and run the check again. If a node cannot be brought online, open a service request with ECS support team to
investigate.

Command:
# /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status

Example:
admin@ecs-n1:~> /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online DONE
Checking Installed Patches and Dependencies DONE

Patches/releases currently installed:

n/a (Base release)

Patches that need to be installed:

CVE-2022-31231_iam-fix (PatchID: 3525)

Files that need to be installed:

/opt/storageos/conf/iam.object.properties (from CVE-2022-31231_iam-fix)
/opt/storageos/lib/storageos-iam.jar (from CVE-2022-31231_iam-fix)

The following services need to be restarted:

dataheadsvc

3. Apply the system patch with svc_patch tool included with xDoctor.

a. Run svc_patch command, type "y" and press "Enter" key when prompted to install the patch. The command can run on any
ECS node.

Commands:
# screen -S patchinstall
# unset TMOUT
# /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch install

Example:
Note: There is a prompt to proceed in the output below.
 admin@ecs-n1:~> screen -S patchinstall
admin@ecs-n1:~> unset TMOUT
admin@ecs-n1:~> /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch install
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online DONE
Checking Installed Patches and Dependencies DONE

Patches/releases currently installed:

n/a (Base release)

Patches that will be installed:

CVE-2022-31231_iam-fix (PatchID: 3525)

Files that will be installed:

/opt/storageos/conf/iam.object.properties (from CVE-2022-31231_iam-fix)
/opt/storageos/lib/storageos-iam.jar (from CVE-2022-31231_iam-fix)

The following services will be restarted:

dataheadsvc

Patch Type: Standalone

Number of nodes: 5
Number of seconds to wait between restarting node services: 60
Check DT status between node service restarts: false

Do you wish to continue (y/n)?y

Distributing files to node 169.254.1.1

Distributing patch installer to node '169.254.1.1'
Distributing files to node 169.254.1.2
Distributing patch installer to node '169.254.1.2'
Distributing files to node 169.254.1.3
Distributing patch installer to node '169.254.1.3'
Distributing files to node 169.254.1.4
Distributing patch installer to node '169.254.1.4'
Distributing files to node 169.254.1.5
Distributing patch installer to node '169.254.1.5'

Restarting services on 169.254.1.1

Restarting dataheadsvc
Waiting 60 seconds for services to stabilize...DONE
Restarting services on 169.254.1.2
Restarting dataheadsvc
Waiting 60 seconds for services to stabilize...DONE
Restarting services on 169.254.1.3
Restarting dataheadsvc
Waiting 60 seconds for services to stabilize...DONE
Restarting services on 169.254.1.4
Restarting dataheadsvc
Waiting 60 seconds for services to stabilize...DONE
Restarting services on 169.254.1.5
Restarting dataheadsvc
Waiting 60 seconds for services to stabilize...DONE

Patching complete.
b. Exit the screen session when patching completed per the above output.

Example:
 admin@node1:/> exit
logout

[screen is terminating]
admin@node1:/>
Note:
If you accidently close the PuTTY session while execution is in progress, you can reattach by logging back into the same
node and run the below command:

Command:
# screen -ls
admin@node 1:~> screen -ls
There is a screen on:
113275.pts-0.ecs-n3 (Detached)
1 Socket in /var/run/uscreens/S-admin.

Reattach to Detached session from previous output

admin@node1:~> screen -r 113277.pts-0.ecs-n3

4. Confirm that the fix has been applied.

a. The output below is from a system where the fix has been applied.

Command:
# /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status

Example:
admin@ecs-n1:~> /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online DONE
Checking Installed Patches and Dependencies DONE

Patches/releases currently installed:

CVE-2022-31231_iam-fix (PatchID: 3525) Fix for ECS iam vulnerability
CVE-2022-31231
n/a (Base release)

Patches that need to be installed:

No files need to be installed.

The following services need to be restarted:

No services need to be restarted.

b. The output below is from a system where the fix has not been applied.

Example:
 admin@ecs-n1:~> /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online DONE
Checking Installed Patches and Dependencies DONE

Patches/releases currently installed:

n/a (Base release)

Patches that need to be installed:

CVE-2022-31231_iam-fix (PatchID: 3525)

Files that need to be installed:

/opt/storageos/conf/iam.object.properties (from CVE-2022-31231_iam-fix)
/opt/storageos/lib/storageos-iam.jar (from CVE-2022-31231_iam-fix)

The following services need to be restarted:

dataheadsvc

Troubleshooting
1. Patch reports below error while doing precheck. In this scenario contact remote support who will provided customer Isolated
Patch for specific environment

Example:
admin@ecs-n1 /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online DONE
Checking Installed Patches and Dependencies FAILED
Fatal: Currently installed version of storageos-iam.jar is unknown.
This likely means that a custom Isolated Patch is installed.
Please contact your next level of support for further steps, and
include this information
Detected md5sum: 6ec26421d426365ecb2a63d8e0f8ee4f

2. Failed to add the host to the list of known hosts while applying patch.

Example:
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online FAILED

ERROR: Could not execute commands on the object-main container on 169.254.x.x

Output was 'Failed to add the host to the list of known hosts (/home/admin/.ssh/known_hosts).
:patchtest:'

Patching is unable to continue with unreachable nodes. To proceed:

- Resolve problems accessing node(s) from this one.
- Manually pass a MACHINES file containing the list of working nodes to patch (not recommended).
- Contact your next level of support for other options or assistance.

Resolution:
The reason could be user of file /home/admin/.ssh/known_hosts was root which should be admin by default.
 Example:
admin@node1:~> ls -l /home/admin/.ssh/known_hosts
-rw------- 1 root root 1802 Jul 23 2019 /home/admin/.ssh/known_hosts
admin@ecs:~>

To fix the issue from another PuTTY session, log on to the reported node or nodes and change the user to admin on the nodes
where it is present as root user using below command on all the reported nodes:

Command:
# sudo chown admin:users /home/admin/.ssh/known_hosts

Example:
admin@node1:~> sudo chown admin:users /home/admin/.ssh/known_hosts

Now rerun the the svc_patch command again and it should pass
admin@node1:~> /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch install

3. Could not execute commands on the object-main container on 169.254.x.x due to incorrect host key in
/home/admin/.ssh/known_hosts.

Example:
svc_patch Version 2.9.2

Verifying patch bundle consistency DONE

Detecting nodes in current VDC DONE
Reading in patch details (1 of 2) DONE
Reading in patch details (2 of 2) DONE
Validating nodes are online FAILED

ERROR: Could not execute commands on the object-main container on 169.254.x.x

Output was '@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:RcwOsFj7zPA5p5kSeYovF4UlZTm125nLVeCL1zCqOzc.
Please contact your system administrator.
Add correct host key in /home/admin/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/admin/.ssh/known_hosts:14
You can use following command to remove the offending key:
ssh-keygen -R 169.254.x.x -f /home/admin/.ssh/known_hosts
Password authentication is disabled to avoid man-in-the-middle attacks.
Keyboard-interactive authentication is disabled to avoid man-in-the-middle attacks.
:patchtest:'

Patching is unable to continue with unreachable nodes. To proceed:

- Resolve problems accessing node(s) from this one.
- Manually pass a MACHINES file containing the list of working nodes to patch (not recommended).
- Contact your next level of support for other options or assistance.

Resolution:
Contact ECS support for a resolution.

4. When using xDoctor version 4.8-85.0 release in prechecks or applying this patch you may get an alert outlining the md5sum did
not match for svc_base.py:
 # /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/svc_patch status
svc_patch Version 2.9.3

Verifying patch bundle consistency FAILED

Patch bundle consistency check failed - md5sums for one or more files
in the patch bundle were invalid, or files were not found.

svc_patch will attempt to validate files in the patch using MD5SUMS.bundle, which
is bundled with the patch.

Output from md5sum was:

./lib/libs/svc_base.py: FAILED
md5sum: WARNING: 1 computed checksum did NOT match

Resolution:
Run the below command prior to applying the patch to update the md5sum:
# sudo sed -i '/svc_base.py/d' /opt/emc/xdoctor/patches/CVE-2022-31231_iam-fix/MD5SUMS.bundle
# sudo sed -i '/MD5SUMS.bundle/d' /opt/emc/xdoctor/.xdr_chksum

Article Properties

Last Published Date

10 Nov 2022

Version
24

Article Type
Solution