CYBERARK UNIVERSITY

Vault Availability
Cluster Vault

CyberArk Training
1
OBJECTIVES

By the end of this lesson you will be able to:

• Describe the different solutions for Vault availability.

• Describe the strengths and limitations of each model.

• Deploy High Availability Cluster

2
VAULT AVAILABILITY OVERVIEW

3
VAULT AVAILABILITY SOLUTIONS

Replicate
COLD • Secure replication of encrypted data to a remote Windows server for tape
backup to an off-site facility

Disaster Recovery (DR)

WARM • One way replication of vault data to a standby Vault server

High Availability (HA)

HOT • Cluster Vault – Two Vault servers using Clustering Services
• Distributed Vaults – Multiple Vault servers providing services at the same time

4
DISASTER RECOVERY

• The Disaster Recovery

DC2
(DR) Vault is a
replication/failover solution
CPM/PVWA/
designed to create a PSM…

stand-by copy of a
DR
Production Vault on a Replication

remote and dedicated

machine Replication

CPM/PVWA/
• The DR-Vault can be PSM… Vault
DR
DC3
activated in the case of a

Replication
Disaster Recovery
situation either DC1
CPM/PVWA/
automatically or manually PSM…

DR

5
 DISTRIBUTED VAULTS
• The Distributed Vaults (DV)
solution spreads the load DC4
from a single primary Vault
(Master) to multiple Satellite CPM/PVWA/
Vaults PSM…
AIM CPs Backend Processes

• The Satellite Vaults are

spread throughout the DC1
Satellite
Vault
deployment to provide read Replication
requests from clients
throughout the organization AIM CPs
Replication

• If a Satellite Vault is CPM/PVWA/ Satellite

unavailable, clients that have PSM…
Primary Vault DC3
Vault
been working with this
Satellite Vault will reconnect

Replication
to another Vault, Satellite or
Master CPM/PVWA/
PSM…
• Since PAS version 11.3 up to DC2
5 satellite vaults can be
deployed Satellite Vault
(Primary Candidate)

6
DISTRIBUTED VAULTS ACTIVE-ACTIVE SERVICES

• CyberArk extended the PAS solution to support

active/active architectures with multiple
Enterprise Password Vaults

• Password retrieval and Session Management,

will be available in the event of an outage,
eliminating data loss

• Once connectivity is resumed, all audits and

session related information will be
synchronized back to the Primary Vault

• For details on implementation contact your

Account Representative

7
VAULT CLUSTER

• The Vault is installed as a high-availability

cluster of servers which provide access to the VIP

accounts in the Vault. In this implementation, Cluster Vault (Passive Cluster Vault (Active Node)
there is always one Server that is on standby Node)
IP Public Network IP + VP
in case the other Server in the cluster fails IP Private Network IP

• To all other CyberArk components, the two

PARAGNT DB V LC ENE DB V LC ENE PARAGNT

Vault Servers in the cluster can be viewed as a CVM CVM

single system, which allows high availability of

the Vault services and allows for the loss of
one Vault server without service disruption
Quorum

Shared Storage
Data + Metadata

8
CLUSTER VAULT ARCHITECTURE

9
HIGH AVAILABILITY ARCHITECTURE

• Two identical vault servers

• Dedicated SAN and PVWA CPM PSM

Cluster Shared Storage

Public Network

Vault Private Network Vault

Storage Network

Shared Storage

10
CYBERARK CLUSTER VAULT MANAGER (CVM)

• New service monitoring the CyberArk Digital

Cluster Vault resources and connections to
other CyberArk Digital Cluster Vault
Node 2 Node 1
components.
Shared Storage
• Active Node: CVM will monitor the status of
local resources: DB V LC ENE CV DB V LC ENE CV

• PrivateArk Server Monitoring Node 1 Monitoring

• Logic Container Quorum Disk

IP Private Network IP
• Database
• ENE (optional) IP Public Network IP + VIP
• PARAgent (optional)
• The active CVM will also monitor the status
of the remote passive CVM. Public Virtual IP

• Passive Node: CVM will monitor (via private

network) the status of CVM in the active node.

11
VIRTUAL IP

• The Cluster Vault must have only one IP

10.10.10.10
exposed for clients – Virtual IP.

• The Cluster Vault will allocate the VIP on the

active node during start up. The CVM will Node A Node B
monitor the VIP to ensure there are no
duplicates (v9.8).
10.10.1.1 10.10.1.2
• During failover/switchover, the CVM will switch
the VIP to the other node.
• In order to prevent possible problems, each
node should have only one single static IP.

12
SHARED STORAGE

• The metadata (database) and data (external files) will be stored

on a shared storage disk.

• Both nodes are connected to the shared storage but only the
active node is in “online status” and can read/write from/to
the disk.

13
QUORUM DISK

• In order to prevent corruption and

communication errors CyberArk employs the
Quorum mechanism.

• The Quorum uses a separate disk on the Active Is Alive? Passive

shared storage. node node

• Quorum disk will always stay offline during

normal Cluster Vault operation (except during
installation) but remain reserved for the active
node (v9.8).
Storage:

14
DETECTING A FAILURE

• Failover is triggered by failure of:

• Vault services
• Storage availability
• Virtual IP availability Check
Active Passive
• Loss of Quorum ownership node node

• The Cluster Vault service identifies a failure in

one of the resources.
• The Cluster Vault service will attempt to restart
a failed service once before going into failover Storage:
mode (v9.8).

15
FAILOVER PROCESS

• The Cluster Vault service on the Active node

changes its status to “Failover” mode and
shuts down all resources.

• The Cluster Vault service on the Passive node Active node: Check Passive node:
Changes to
will then reserve the shared resources, such as Failover then
Changes to
the VIP, Shared Storage and Quorum Disk. Active Mode
Passive Mode

• Once the Shared Storage is online, the

Passive Node has now been promoted to the
Active Node and can start the services and
provide Vault services. Storage:

• The Cluster Vault service on the former active

node will switch its role to Passive, and will
start monitoring the new active node.
Quorum

16
CLUSTER VAULT MANAGEMENT

17
CLUSTER VAULT MANAGEMENT UTILITY – ACTIVE NODE

• The new Cluster Vault is

managed and controlled by
the Cluster Vault
Management Utility.
• Before restarting a Vault
machine that is part of a
cluster, it is highly
recommended to stop the
node from the Management
Utility in order to make sure
all resources shut down
properly.
• The graphic to the right
illustrates how a CVM utility
should look on the Active
Node on the cluster.

18
CLUSTER VAULT MANAGEMENT UTILITY – STANDBY NODE

• This is the CVM utility

running on the standby
node

• Note that the local node is

always shown on the left,
regardless of whether it is
active or passive

• Shared Storage status is

reported at the bottom. In
this example, the Quorum
disk status is Released,
and the Storage drive is
Offline appropriately for
the Passive Node.

19
MONITORED SERVICES

• Monitored services can be

configured by an
Administrator.

• Using the CVM the

Administrator can select the
services to be monitored by
the Cluster Vault Manager.

• Services not monitored will

be ignored and will not
trigger a cluster failover.

20
SIMULATING FAILOVER

• To Perform a switchover
test, open the CVM on the
Active Node of the cluster:

• Click the Switchover

button shown highlighted.

• Click Continue to confirm

the message.

• The operation is complete

when the node status is
updated.

21
CYBERARK DIGITAL CLUSTER VAULT
SERVER INSTALLATION
(PREPARATION AND REQUIREMENTS)

22
PREPARE THE SERVERS

• The Vault machines must meet the recommended

system requirements

• Supported on CyberArk EPV v9.7 on MS Windows 2012

• The two Cluster Vault Nodes must be connected directly
via a private network or cross-over cable
• It is highly recommended that both nodes have identical
specifications including memory and processor
• The clocks on both cluster nodes must be synchronized

23
STORAGE PREREQUISITES
• Shared storage must support Persistent Reservation

• It is recommended to use an enterprise-grade fiber-channel

SAN solution
• iSCSi network storage is not recommended for a production
implementation.
• If iSCSI is used in a non-production environment then a
Windows update (KB2955164) should be installed in order
to ensure database stability.
• Using iSCSi also requires a FW exception to dbparm.ini
during installation.

24
PREPARE THE STORAGE

• Prepare the shared storage with two drives.

• One drive is for the Vault data, and the other drive is for
the Quorum Disk.

• Drive letters for the Quorum and Storage disks must be

identical on both nodes.

• During EPV Cluster Vault installation, ensure that the

shared storage resources are online for ONLY the
node currently being installed. After the EPV Cluster
Vault is successfully installed, the CVM will manage
the Shared Storage.

25
 CLUSTER INSTALLATION
(INSTALL THE FIRST NODE)

26
INSTALL THE FIRST NODE – VAULT INSTALLATION MODE

Launch the setup.exe and

choose Cluster-Node Vault
installation

27
INSTALL THE FIRST NODE – SAFES LOCATION

Choose the location on the

shared storage to store the
safes

28
INSTALL THE FIRST NODE – OPERATOR CD PATH

• Copy the encryption keys

from the operator CD to a
folder on the local drive

• Select the folder on the

local drive as the
Operator CD path

• Complete the installation,

but do not reboot
immediately

29
 INSTALL THE FIRST NODE – CONFIGURE STORAGE

• In an Administrators Command
Window, navigate to the
PrivateArk\Server\ClusterVault
directory.

• Use the following command line

to set the Quorum and Shared
Storage drive letters:

StorageManager.exe –qE -sF

• -q sets drive letter for quorum
• -s sets drive letter for shared
storage

30
 INSTALL THE FIRST NODE – CONFIGURE CLUSTERVAULT.INI

Set the names and IP

addresses for the local and
peer node in ClusterVault.ini
• Logical Names
• Virtual IP
• Peer and Local Public and
Private IP addresses
• located in C:\Program Files
(x86)\PrivateArk\Server\Cluster
Vault\

• The information defined in the

ClusterVault.ini file, is displayed
by the Cluster Vault
Management utility or CVM.

31
INSTALL THE FIRST NODE – REBOOT

• Restart the first node and

verify that all resources
have been started
successfully. The following
message will appear in the
ClusterVaultConsole.log:

CVMCS087I All the

resources are running
successfully

• Launch Cluster Vault

Management, check that
node is showing as
“Active”, shared storage
as “Online” and Quorum
as “Reserved”.

32
PREPARING FOR VAULT INSTALLATION
ON SECOND NODE

33
 COPY ENCRYPTION KEYS TO SECOND NODE
• Use the same set of Operator
Keys that you used to install
the first node of the Cluster
Vault.
• Copy the additional keys
listed here, that were
generated during the
installation of the first node to
the same location in the
second node. These keys will
be created in the folder
containing the original
Operator Keys.
• Backup.key
• VaultUser.pass
• ReplicationUser.pass
• VaultEmergency.pass

34
STOP SERVICES ON FIRST NODE

• Before starting the

installation of the second
node of the Cluster Vault,
we need to stop all
services on the first node.

• Log on to the first node,

and launch Cluster Vault
Management. Select the
stop symbol that is
highlighted in the graphic.

35
 SET SHARED DISKS TO OFFLINE ON FIRST NODE
• Use the Disk Management
utility to verify the shared
disks are offline on the first
node.

• Make sure that there are no

open files or folders on the
shared storage.

36
BRING SHARED DISKS ONLINE ON SECOND NODE

• Use the Disk Management

utility to bring the Shared
Disks online on the
second Node.

• Ensure that the drive

letters for the Quorum and
Storage disks are
identical in both nodes

37
 CLUSTER INSTALLATION
(INSTALL THE SECOND NODE)

38
INSTALL THE SECOND NODE – SAFES LOCATION

Install The Vault on the

Second Node.

Make sure you select:

• “Cluster-Node Vault
installation” as the
installation mode
• the same drive letter and
folder on the shared storage
for the Safes location.

39
INSTALL THE SECOND NODE – VAULTID

The Vault-id parameter must

be consistent for both cluster
nodes.

• Open DBParm.ini on the

first node

• Copy the Vault-id

parameter from the first
node to DBParm.ini on the
second node

40
INSTALL THE SECOND NODE – SERVER-ID

The server-id parameter

must be consistent for both
cluster nodes.

• Open my.ini on the first

node in the Database
subdirectory

• Copy the Server-id from

the first node to the
second node

41
 INSTALL THE SECOND NODE – CONFIGURE STORAGE
The disk identifiers must be
recorded in the ClusterVault.ini
file in the StorageIdentifier and
QuorumDiskIdentifier
parameters.

• Use the following command to

set the Quorum and Shared
Storage drive letters:

StorageManager.exe –qE –sF

• Use the same drive letters as

on the first node

42
INSTALL THE SECOND NODE – CONFIGURE CLUSTERVAULT.INI

• Set the names and IP

addresses for the local
and peer nodes in
ClusterVault.ini
• Logical Names
• Virtual IP
• Peer and Local Public
and Private IP addresses

43
INSTALL THE SECOND NODE – REBOOT

• Restart the second node

and verify that all resources
have been started
successfully. The following
message should appear in
the
ClusterVaultConsole.log:

CVMCS087I All the

resources are running
successfully

• After the Second node has

started successfully and is
active, start the first node in
Passive mode and then
trigger a switchover to test
the cluster failover process.

44
CLUSTER VAULT LOGS

45
LOGS

• ClusterVaultConsole.log
• Cluster Vault log file

• ClusterVaultTrace.log
• Cluster Vault trace file

• Setting the debug level for

the Cluster Vault can be
set dynamically with no
restart needed.

46
QUIZ
1. What are the 3 main types of Vault Availability?
• Cold = Replicate backup
• Warm = Disaster Recovery
• Hot = High Availability and Distributed Vaults

2. What is the Quorum Disk used for in a HA Cluster Vault architecture?

• The Quorum Mechanism is based on a voting algorithm. When starting up, each node in the cluster has a
vote when accessing the Shared Resources, and the Quorum disk provides the tie breaking vote.

3. What does the CVM or Cluster Vault Management Utility provide?

• The CVM monitors shared services, disk storage, the Virtual IP and the status of the other node of the
cluster.
• An Administrator can initiate a failover from the Active to the Passive Node from the CVM.
• Monitoring of services by the CVM can be selectively turned off, if necessary.

4. After installing NodeA, what information needs to be copied manually to NodeB?

• Copy the additional keys, i.e., Backup.key, VaultUser.pass, ReplicationUser.pass, VaultEmergency.pass.
• Copy the Server-id from my.ini.
• Copy the Vault-id parameter.

47
THANK YOU

48