Architecture and Design for a Virtual

Infrastructure Workload Domain

09 FEB 2021
VMware Validated Design 6.2
VMware Cloud Foundation 4.2
Architecture and Design for a Virtual Infrastructure Workload Domain

You can find the most up-to-date technical documentation on the VMware website at:

https://docs.vmware.com/

VMware, Inc.
3401 Hillview Ave.
Palo Alto, CA 94304
www.vmware.com

©
Copyright 2020-2021 VMware, Inc. All rights reserved. Copyright and trademark information.

VMware, Inc. 2
 Contents

About Architecture and Design for a Virtual Infrastructure Workload Domain 4

1 Architecture Overview for a Virtual Infrastructure Workload Domain 6

2 Detailed Design for a Virtual Infrastructure Workload Domain 10

Physical Infrastructure Design for a Virtual Infrastructure Workload Domain 10
Availability Zones and Regions for a Virtual Infrastructure Workload Domain 11
Workload Domains and Racks for a Virtual Infrastructure Workload Domain 12
Virtual Infrastructure Design for a Virtual Infrastructure Workload Domain 17
ESXi Detailed Design for a Virtual Infrastructure Workload Domain 18
vCenter Server Design for a Virtual Infrastructure Workload Domain 30
vSphere Networking Design for a Virtual Infrastructure Workload Domain 53
Software-Defined Networking Design for a Virtual Infrastructure Workload Domain 64
Shared Storage Design for a Virtual Infrastructure Workload Domain 133
vSphere with Tanzu Detailed Design for a Virtual Infrastructure Workload Domain 156

VMware, Inc. 3
About Architecture and Design for a Virtual
Infrastructure Workload Domain

The Architecture and Design for a Virtual Infrastructure Workload Domain document contains
a validated design model for a virtual infrastructure workload domain in a single-region or a
dual-region Software-Defined Data Center (SDDC).

Chapter 1 Architecture Overview for a Virtual Infrastructure Workload Domain discusses the
building blocks and the main principles of a workload domain. Chapter 2 Detailed Design for a
Virtual Infrastructure Workload Domain provides the available design options according to the
design objectives, and a set of design decisions to justify selecting the path for building each
component.

Intended Audience
Architecture and Design for a Virtual Infrastructure Workload Domain document is intended for
cloud architects who are familiar with and want to use VMware software to deploy and manage
a virtual infrastructure workload domain in the SDDC that meets multiple requirements. The
document provides guidance for capacity, scalability, backup and restore, and extensibility for
disaster recovery support.

Supported VMware Cloud Foundation Version

Architecture and Design for a Virtual Infrastructure Workload Domain is compatible with VMware
Cloud Foundation 4.2.

Required VMware Software

Architecture and Design for a Virtual Infrastructure Workload Domain is compliant and validated
with certain product versions. See VMware Validated Design Release Notes.

Before You Apply This Guidance

If you plan to deploy an SDDC by following the prescriptive path of VMware Validated Design,
to apply Architecture and Design for a Virtual Infrastructure Workload Domain, you must be
acquainted with the following guidance:

n Introducing VMware Validated Design. See Guided Documentation Map of VMware Validated
Design.

VMware, Inc. 4
Architecture and Design for a Virtual Infrastructure Workload Domain

n Architecture and Design for the Management Domain.

If you plan to deploy an SDDC by following the VMware Cloud Foundation documentation, you
must be acquainted with the following guidance:

n Introducing VMware Cloud Foundation. See the VMware Cloud Foundation documentation
page.

n Optionally, VMware Cloud Foundation Operations and Administration Guide.

n Architecture and Design for the Management Domain.

VMware, Inc. 5
Architecture Overview for a
Virtual Infrastructure Workload
Domain
1
By implementing this design for the SDDC, an IT organization can automate the provisioning
of common, repeatable requests for IT services and respond to business needs with agility and
predictability. This SDDC design provides an IT solution with features across many areas such as
operations management, cloud management, business continuity, and security and compliance.

Figure 1-1. Architecture Overview of the SDDC Workload Domain in a Region

Another Solution Add-On

Cloud Operations and Automation Solution Add-on

Cross-Region vRealize vRealize vRealize vRealize

Workspace Suite Lifecycle Operations Log Insight Automation
ONE Access Manager Manager

Management Domain Workload Domain Workload Domain

Region-
SDDC Specific vSphere with Tanzu vSphere with Tanzu
Manager Workspace
ONE Access
NSX-T NSX-T
(1:1 or 1:N) (1:1 or 1:N)
NSX-T
vCenter Server vCenter Server
vCenter Server
Principal Storage Principal Storage
Principal Storage (vSAN, vVols, NFS, or VMFS on FC) (vSAN, vVols, NFS, or VMFS on FC)
(vSAN)

ESXi ESXi ESXi

Consolidated SDDC
Architecture

Standard SDDC Architecture

VMware, Inc. 6
Architecture and Design for a Virtual Infrastructure Workload Domain

Workload Domain Architecture

The workload domain forms an additional building block of the SDDC to the management domain
and consists of components from the physical infrastructure, virtual infrastructure, and security
and compliance layers. The virtual infrastructure layer controls the access to the underlying
physical infrastructure layer, it controls and allocates resources to workloads running in the
workload domain. The security and compliance layer provides role-based access controls and
integration with the corporate identity provider.

Table 1-1. Initial Component Configuration of the Workload Domain in Each Region

Component Services

ESXi Virtual infrastructure for running the SDDC management

components and tenant workloads. See ESXi Detailed
Design for a Virtual Infrastructure Workload Domain.

vCenter Server Centralized and extensible management of workload ESXi

hosts and management appliances. See vCenter Server
Design for a Virtual Infrastructure Workload Domain.

NSX-T Logical switching, dynamic routing, and load balancing for

the SDDC components. See Software-Defined Networking
Design for a Virtual Infrastructure Workload Domain.

vSAN Principal software-defined storage for all SDDC

components. See Shared Storage Design for a Virtual
Infrastructure Workload Domain.

Availability Zones and Regions

The SDDC design consists of one, two, or more regions. Each region includes at least one
management domain and one or more workload domains. vSphere clusters in a region can use
two availability zones.

Figure 1-2. Availability Zones and Regions

Region A: SFO Region B: LAX

Availability Availability
Zone 1 Zone 1
Future
Availability
Zone
Future
Availability Availability
Zone 2 Zone

This design starts with a single region and expands to two regions, with the option to use one or
two availability zones in the first region (Region A) and single availability zone in the second region
(Region B). You can apply the design for two availability zones to deploy multiple availability
zones.

VMware, Inc. 7
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 1-3. Component Location in a Single Availability Zone

Tenant Workloads NSX-T Edges

APP APP APP APP

OS OS OS OS

vSAN

vSphere Distributed Switch with NSX-T

Workload Domain
vCenter Server

Shared Edge and Workload Cluster

ESXi ESXi ESXi ESXi

Single Availability Zone

SDDC Architecture

VMware, Inc. 8
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 1-4. Component Location in Multiple Availability Zones

Tenant Workloads NSX-T Edges

APP APP APP APP

OS OS OS OS

vSAN

vSphere Distributed Switch with NSX-T

Workload Domain
vCenter Server

Shared Edge and Workload Cluster

Availability Availability
Zone1 Zone2

ESXi ESXi

ESXi ESXi

ESXi ESXi

ESXi ESXi

Dual Availability Zone

SDDC Architecture

Security and Compliance

This design provides role-based access control through the integration of an identity and access
management solution which integrates with Microsoft Active Directory.

VMware, Inc. 9
Detailed Design for a Virtual
Infrastructure Workload Domain 2
The detailed design considers physical infrastructure, virtual infrastructure, operations
management, and security and compliance design components. It includes numbered design
decisions and the justification and implications of each decision.

This design provides two design options for availability zone design. In certain areas,
configurations and design decision alternatives are specific to either a single availability zone and
to a multiple availability zone design.

This chapter includes the following topics:

n Physical Infrastructure Design for a Virtual Infrastructure Workload Domain

n Virtual Infrastructure Design for a Virtual Infrastructure Workload Domain

Physical Infrastructure Design for a Virtual Infrastructure

Workload Domain
The physical infrastructure design includes design decisions for availability zones, regions,
clusters, racks, and physical network connectivity.

VMware, Inc. 10
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-1. Physical Infrastructure in the SDDC

Cloud Service Catalog Cloud Business Security and

Automation Operations Continuity Compliance
Self-Service Portal

Orchestration

Virtual Hypervisor
Infrastructure Identity and Access
Monitoring Fault Tolerance
Pools of Resources Management

Virtualization Control
Backup &
Logging Industry Regulations
Restore
Physical Compute
Infrastructure
Storage
Life Cycle Security Policies
Network Management

n Availability Zones and Regions for a Virtual Infrastructure Workload Domain

Availability zones and regions have different purposes. Availability zones protect against
failures of individual hosts. You can consider regions to place workloads closer to your
customers, comply with data privacy laws and restrictions, and support disaster recovery
solutions for the entire SDDC.

n Workload Domains and Racks for a Virtual Infrastructure Workload Domain

Availability Zones and Regions for a Virtual Infrastructure Workload

Domain
Availability zones and regions have different purposes. Availability zones protect against failures
of individual hosts. You can consider regions to place workloads closer to your customers, comply
with data privacy laws and restrictions, and support disaster recovery solutions for the entire
SDDC.

This design uses a protected region for tenant workloads with one or two availability zones and
recovery region with a single availability zone. You can place workloads in each availability zone
and region. Usually, multiple availability zones form a single region.

Availability zones

An availability zone is the fault domain of the SDDC. Multiple availability zones can provide
continuous availability of an SDDC, minimize down time of services and improve SLAs.

Regions

Regions provide disaster recovery across different SDDC instances or a location that is closer
to your customers. Each region is a separate SDDC instance. The regions have a similar
physical layer and virtual infrastructure designs but different naming.

VMware, Inc. 11
Architecture and Design for a Virtual Infrastructure Workload Domain

Regions are geographically separate, but latency between them must be 150 ms or lower.

The identifiers follow United Nations Code for Trade and Transport Locations (UN/LOCODE) and
also contain a numeric instance ID.

Table 2-1. Availability Zones and Regions in the SDDC

Region Identifier and Region-Specific Domain
Region Availability Zone Name Region Description

SFO SFO01 sfo.rainpole.io Availability Zone 1 in San Francisco, CA, USA

based data center

SFO SFO02 sfo.rainpole.io Availability Zone 2 in San Francisco, CA, USA

based data center

LAX LAX01 lax.rainpole.io Los Angeles, CA, USA based data center

Table 2-2. Design Decisions on Availability Zones and Regions

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-PHY-001 In Region SFO, that is Supports all tenant n Using a single

Region A, deploy one or workloads and NSX-T Edge availability zone results
two availability zones to nodes for a region. in limited redundancy of
support tenant workloads the overall solution.
and NSX-T Edge nodes. n A single availability
zone can become
a single point of
failure and prevent
high-availability design
solutions in a region.

Supports stretched clusters Implementing two

and application-aware availability zones increases
failover for high availability the solution footprint
between two physical and can complicate the
locations. operational procedures.

SDDC-WLD-PHY-002 For a dual-region SDDC, in Supports all tenant n Using a single

Region LAX, that is Region workloads and NSX-T Edge availability zone results
B, deploy one availability nodes for a region. in limited redundancy of
zone. You can later add another the overall solution.
availability zone to extend n A single availability
and scale out the tenant zone can become
capabilities of the SDDC. a single point of
failure and prevent
high-availability design
solutions in a region.

Workload Domains and Racks for a Virtual Infrastructure Workload

Domain
The SDDC functionality is distributed across multiple workload domains and vSphere clusters.
Each workload domain is a logical abstraction of private cloud capacity and consists of one or
more clusters. Each cluster can exist vertically in a single rack or be spanned horizontally across

VMware, Inc. 12
Architecture and Design for a Virtual Infrastructure Workload Domain

multiple racks. You determine the total number of racks for each cluster type according to your
scalability needs.

Workload domains are provisioned automatically by SDDC Manager and administered and
patched independently.

Workload Domain to Rack Mapping

The relationship between workload domains and data center racks is not one-to-one. While a
workload domain is an atomic unit of repeatable building blocks, a rack is a unit of size. Because
workload domains can have different sizes, you map workload domains to data center racks
according to the use case.

When using a Layer 3 network fabric, the clusters in the management domain cannot span
racks. Management appliances and virtual machines rely on VLAN-backed networks. The physical
network configuration terminates Layer 2 networks in each rack at the Top of the Rack (ToR)
switch. Therefore, you cannot migrate a virtual machine to a different rack because the IP subnet
is available only in the rack where the virtual machine currently runs.

Table 2-3. Workload Domain to Rack Configuration Options

Workload Domain to Rack Configuration Description

One Workload Domain in One Rack One workload domain can occupy exactly one rack.

Multiple Workload Domains in One Rack Two or more workload domains can occupy a single rack,
for example, one management workload domain and one
virtual infrastructure workload domain can be deployed to
a single rack.

Single Workload Domain Across Multiple Racks A single workload domain can stretch across multiple
adjacent racks. For example, a virtual infrastructure
workload domain that has more ESXi hosts than a single
rack can support.

Stretched Workload Domain Across Availability Zones A cluster in a single workload domain can span across
two availability zones by using VMware vSAN™ stretched
clustering.

VMware, Inc. 13
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-4. Design Decisions on Clusters and Racks

Decision ID Design Decision Design Justification Design Implications

SDDC-WLD-PHY-003 Use two separate power Redundant power feeds All equipment used must
feeds for each rack. increase availability by support two separate
ensuring that failure of a power feeds. The
power feed does not bring equipment must keep
down all equipment in a running if one power feed
rack. fails.
Combined with redundant If the equipment of an
network connections to entire rack fails, the cause,
a rack and in a rack, such as flooding or an
redundant power feeds earthquake, also affects
prevent a failure of the neighboring racks. Use a
equipment in an entire rack. second region to reduce
downtime associated with
such an event.

Clusters and Racks in a Single Availability Zone

Figure 2-2. SDDC Cluster Architecture for a Single Availability Zone

External
connection

ToR ToR ToR ToR ToR ToR

Switch Switch Switch Switch Switch Switch

Shared edge and

workload cluster
(4 ESXi hosts)

Management
cluster
(4 ESXi hosts)

Workload cluster
(19 ESXi host each)

VMware, Inc. 14
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-5. Design Decisions on Clusters and Racks for Single Availability Zone

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-PHY-004 Mount the compute Mounting the compute The data centers must
resources (minimum of 4 resources for the workload have sufficient power and
ESXi hosts) for the shared domain cluster together can cooling to operate the
edge and workload cluster ease physical data center server equipment according
together in a single rack. design, deployment, and to the selected vendor and
troubleshooting. products.
You only need to provide If the equipment in the
on-ramp and off-ramp entire rack fails, to reduce
connectivity to physical downtime associated with
networks (for example, such an event, you must
north-south Layer 3 routing have a second availability
for NSX -T) to a single rack. zone or region.
NSX-T Edge resources
require external
connectivity to physical
network devices. Placing
NSX-T Edge resources in
the same rack minimizes
VLAN spread.

VMware, Inc. 15
Architecture and Design for a Virtual Infrastructure Workload Domain

Two Availability Zones

Figure 2-3. SDDC Cluster Architecture for Two Availability Zones

External External External

connection connection connection

ToR ToR ToR ToR ToR ToR

Switch Switch Switch Switch Switch Switch

Stretched Stretched Management

management cluster management cluster cluster
Availability Zone 1 Availability Zone 2 (4 ESXi hosts)
(4 ESXi hosts) (4 ESXi hosts)

Stretched shared Stretched shared

edge and edge and Еdge and
workload cluster workload cluster workload cluster
Availability Zone 1 Availability Zone 2 (4 ESXi hosts)
(4 ESXi hosts) (4 ESXi hosts)

Availability Zone 1 Availability Zone 2

Region A Region B

Table 2-6. Design Decisions on Clusters and Racks for Two Availability Zones

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-PHY-005 In each availability zone, Mounting the compute The data centers must
mount the compute resources for the shared have sufficient power and
resources (minimum of 4 edge and workload cooling to operate the
ESXi hosts) for the shared cluster together can server equipment according
edge and workload cluster ease physical data center to the selected vendor and
together in a single rack. design, deployment, and products.
troubleshooting. If the equipment in the
You only need to provide entire rack fails, to reduce
on-ramp and off-ramp downtime associated with
connectivity to physical such an event, you must
networks (for example, have a second availability
north-south Layer 3 routing zone or region.
for NSX -T) to a single rack.
NSX-T Edge appliances
require external
connectivity to physical
network devices. Placing
NSX-T Edge appliances in
the same rack minimizes
VLANs that must be
created in AZ2.

VMware, Inc. 16
Architecture and Design for a Virtual Infrastructure Workload Domain

Virtual Infrastructure Design for a Virtual Infrastructure

Workload Domain
The virtual infrastructure design includes the components that make up the virtual infrastructure
layer for providing software-defined storage, networking, and compute.

These components include the software products that provide the virtualization platform
hypervisor, virtualization management, storage virtualization, and network virtualization. The
VMware products in this layer are vSphere, vSAN, and NSX-T Data Center.

Figure 2-4. Virtual Infrastructure in the SDDC

Cloud Service Catalog Cloud Business Security and

Automation Operations Continuity Compliance
Self-Service Portal

Orchestration

Virtual Hypervisor
Infrastructure Identity and Access
Monitoring Fault Tolerance
Management
Pools of Resources

Virtualization Control
Backup &
Logging Industry Regulations
Restore
Physical Compute
Infrastructure
Storage
Life Cycle Security Policies
Network Management

For an overview of the setup of the virtual infrastructure layer in the workload domain, see
Chapter 1 Architecture Overview for a Virtual Infrastructure Workload Domain.

n ESXi Detailed Design for a Virtual Infrastructure Workload Domain

The compute layer of the virtual infrastructure is implemented by ESXi, a bare-metal
hypervisor that installs directly onto your physical server. With direct access and control of
underlying resources, ESXi logically partitions hardware to consolidate applications and cut
costs.

n vCenter Server Design for a Virtual Infrastructure Workload Domain

The vCenter Server design includes determining the number of vCenter Server instances in
the workload domain, their size, networking configuration, cluster layout, redundancy, and
security configuration.

n vSphere Networking Design for a Virtual Infrastructure Workload Domain

The network design prevents unauthorized access and provides timely access to business
data. This design uses vSphere Distributed Switch and VMware NSX-T Data Center for virtual
networking.

VMware, Inc. 17
Architecture and Design for a Virtual Infrastructure Workload Domain

n Software-Defined Networking Design for a Virtual Infrastructure Workload Domain

In this design, you use NSX-T Data Center to provide network connectivity for tenant
workloads by using virtual network segments and routing. You also create constructs for
region-specific and cross-region solutions. These constructs isolate the solutions from the
rest of the network, providing routing to the data center and load balancing.

n Shared Storage Design for a Virtual Infrastructure Workload Domain

n vSphere with Tanzu Detailed Design for a Virtual Infrastructure Workload Domain
vSphere with Tanzu transforms vSphere to a platform for running Kubernetes workloads
natively on the hypervisor layer. When enabled on a vSphere cluster, vSphere with Tanzu
provides the capability to run Kubernetes workloads directly on ESXi hosts and to create
upstream Kubernetes clusters in dedicated resource pools.

ESXi Detailed Design for a Virtual Infrastructure Workload Domain

The compute layer of the virtual infrastructure is implemented by ESXi, a bare-metal hypervisor
that installs directly onto your physical server. With direct access and control of underlying
resources, ESXi logically partitions hardware to consolidate applications and cut costs.

n Logical Design for ESXi for a Virtual Infrastructure Workload Domain

The logical design provides a high-level overview of the design for ESXi.

n Deployment Specification for ESXi for a Virtual Infrastructure Workload Domain

You determine the size of the compute resources, start-up configuration, and patching
and upgrade support for the ESXi hosts for the workload domain according to the design
objectives and aggregated requirements of the components of the SDDC.

n Network Design for ESXi for a Virtual Infrastructure Workload Domain

In the network design for the ESXi hosts in a workload domain, you place the hosts on a
VLAN for traffic segmentation and decide on the IP addressing scheme and name resolution
for optimal support for tenant workloads and maintenance of the hosts.

n Information Security and Access Design for ESXi for a Virtual Infrastructure Workload Domain
You design authentication access, controls, and certificate management for ESXi according to
industry standards and the requirements of your organization.

Logical Design for ESXi for a Virtual Infrastructure Workload Domain

The logical design provides a high-level overview of the design for ESXi.

To provide the foundational component of the virtual infrastructure each ESXi host consists of the
following elements:

n CPU and memory

n Storage devices

n Out of band management interfaces

n Network interfaces

VMware, Inc. 18
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-5. ESXi Logical Design

ESXi Host

Compute

CPU Memory

Storage

Non-Local Storage Local Storage

(SAN/NAS) (vSAN)

Network

NIC 1 and NIC 2 Out of Band

Uplinks Management Uplink

Deployment Specification for ESXi for a Virtual Infrastructure Workload Domain

You determine the size of the compute resources, start-up configuration, and patching and
upgrade support for the ESXi hosts for the workload domain according to the design objectives
and aggregated requirements of the components of the SDDC.

n Sizing Compute Resources for ESXi for a Virtual Infrastructure Workload Domain
You size the compute resources of the ESXi hosts in the workload domain according to
the system requirements of the management components and the requirements for tenant
workloads according to the design objectives.

n Host Boot Device and Scratch Partition Design for a Virtual Infrastructure Workload Domain
Determining the boot device type and size for each ESXi host in the workload domain is
important for the creation of system storage volumes. You also plan the location of the
scratch partition according to the selected boot device type so that system log information is
available even if a storage failure occurs.

n Virtual Machine Swap Design for ESXi for a Virtual Infrastructure Workload Domain
When you decide on the placement of the VMkernel swap file of the virtual machines running
in a virtual infrastructure workload domain, consider the configuration efforts and traffic
related to transferring virtual machine system data across the data center.

VMware, Inc. 19
Architecture and Design for a Virtual Infrastructure Workload Domain

n Lifecycle Management Design for ESXi for a Virtual Infrastructure Workload Domain
When you decide on a life cycle management approach for the ESXi software, you consider
the effort and time required for preparing the environment and performing the patch,
upgrade, or update operation.

Sizing Compute Resources for ESXi for a Virtual Infrastructure Workload Domain
You size the compute resources of the ESXi hosts in the workload domain according to the
system requirements of the management components and the requirements for tenant workloads
according to the design objectives.

For a dual-region SDDC, to accommodate the components of NSX-T Federation, that is used for
central networking across the first and second regions, you must allocate more compute resources
on the ESXi hosts in both regions. If you have configured the ESXi hosts in Region A for use in a
single-region environment, you must increase their number of CPUs and memory size.
ESXi Server Hardware
The configuration and assembly process for each physical server designated to run ESXi should be
standardized, with all components installed in a consistent manner. Standardization of the physical
configuration of ESXi hosts removes variability, resulting in infrastructure that is more easily
managed and supported. ESXi hosts should be deployed with identical configuration across all
cluster members, including storage and networking configurations. For example, consistent PCIe
card installation, especially for network controllers, is essential for accurate mapping of physical
network controllers to virtual network resources. You ensure an even distribution of resources
available to virtual machines across ESXi hosts in a cluster by standardizing components within
each physical server.

n An average-size virtual machine has two virtual CPUs with 4 GB of RAM.

n A typical spec 2U ESXi host can run 60 average-size virtual machines.

This design uses vSAN ReadyNodes as the fundamental building block for the principal storage
system in the workload domain. Select all ESXi host hardware, including CPUs, according to
VMware Compatibility Guide and aligned to the ESXi version specified by this design. The sizing
of physical servers that run ESXi requires special considerations when you use vSAN storage. For
information about the models of physical servers that are vSAN-ready, see vSAN Compatibility
Guide for vSAN ReadyNodes. If you are not using vSAN ReadyNodes, your CPU, Disks and
I/O modules must be listed on the VMware Compatibility Guide under CPU Series and vSAN
Compatibility List aligned to the ESXi version specified by this design.

VMware, Inc. 20
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-7. Design Decisions on Server Hardware for ESXi

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-001 Use vSAN ReadyNodes Your SDDC is fully Hardware choices might be
with vSAN storage for each compatible with vSAN at limited.
ESXi host in the shared deployment.
edge and workload cluster.

SDDC-WLD-VI-ESXi-002 Allocate hosts with uniform A balanced cluster has You must apply vendor
configuration across the these advantages: sourcing, budgeting,
shared edge and workload n Predictable and procurement
cluster. performance even considerations for uniform
during hardware server nodes, on a per
failures cluster basis.

n Minimal impact
of resync or
rebuild operations on
performance

ESXi Host Memory

When sizing memory for the ESXi hosts in the workload domain, consider certain requirements.

n Requirements for the workloads that are running in the cluster

When sizing memory for the hosts in a cluster, to reserve the resources of one host for failover
or maintenance, set the admission control setting to N+1, which reserves the resources of one
host for failover or maintenance.

n Number of vSAN disk groups and disks on an ESXi host

To support the maximum number of disk groups, you must provide 32 GB of RAM. For
more information about disk groups, including design and sizing guidance, see Administering
VMware vSAN from the vSphere documentation.

Table 2-8. Design Decisions on Host Memory for ESXi

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-003 Install each ESXi host in the The large-sized NSX-T In a four-node cluster, only
shared edge and workload Edge appliances in this 768 GB is available for use
cluster with a minimum of vSphere Cluster require a due to the n+1 vSphere HA
256 GB RAM. total of 64 GB RAM. setting.
The remaining RAM
is available for tenant
workloads.

Host Boot Device and Scratch Partition Design for a Virtual Infrastructure Workload Domain
Determining the boot device type and size for each ESXi host in the workload domain is important
for the creation of system storage volumes. You also plan the location of the scratch partition
according to the selected boot device type so that system log information is available even if a
storage failure occurs.

VMware, Inc. 21
Architecture and Design for a Virtual Infrastructure Workload Domain

ESXi requires a boot disk of at least 8 GB for USB or SD devices, and 32 GB for other device types
such as HDD, SSD, or NVMe. A boot device must not be shared between ESXi hosts.

ESXi can boot from a disk larger than 2 TB if the system firmware and the firmware on any add-in
card support it. See vendor documentation.

The ESXi system storage volumes occupy up to 128 GB of disk space. A local VMFS datastore is
only created if the local disk device has at least 4 GB additional free space. To share a boot device
with a local VMFS datastore, you must use a local disk of 240 GB or larger. If a local disk cannot
be found, ESXi operates in a mode with limitations, where certain functionality is disabled, and the
scratch partition is on the RAM disk of the ESXi host, linked to the /tmp folder. As a result, log
information and stack traces are lost on host reboot. You can reconfigure the scratch partition to
use a separate disk or LUN.

An ESXi installation process on a USB or SD device does not configure a default scratch partition
on the local disk. Place the scratch partition on a shared datastore and configure remote syslog
logging for the ESXi host.

Table 2-9. Design Decisions for Host Boot Device and Scratch Partition of ESXi

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-004 Install and configure all Provides hosts with large When you use SATA-DOM
ESXi hosts in the shared memory, that is, greater or SD devices, scratch
edge and workload cluster than 512 GB, with enough partition and ESXi logs
to boot using a 32-GB space for the core dump are not retained locally.
device or greater. partition while using vSAN. Configure the scratch
partition of each ESXi host
on supplemental storage.

SDDC-WLD-VI-ESXi-005 Use the default n If a failure in the vSAN When you use SATA-DOM
configuration for the cluster occurs, the ESXi or SD devices, scratch
scratch partition on all ESXi hosts remain responsive partition and ESXi logs
hosts in the shared edge and log information is are not retained locally.
and workload cluster still accessible. Configure the scratch
n It's not possible to use partition of each ESXi host
vSAN datastore for the on supplemental storage.
scratch partition.

Virtual Machine Swap Design for ESXi for a Virtual Infrastructure Workload Domain
When you decide on the placement of the VMkernel swap file of the virtual machines running in
a virtual infrastructure workload domain, consider the configuration efforts and traffic related to
transferring virtual machine system data across the data center.

When a virtual machine is powered on, the system creates a VMkernel swap file to serve as a
backing store for the contents of the virtual machine's RAM. By default, the swap file is stored in
the same location as the configuration file of the virtual machine. The co-location simplifies the
configuration. However, it can lead to generating additional replication traffic that is not needed.

VMware, Inc. 22
Architecture and Design for a Virtual Infrastructure Workload Domain

You can reduce the amount of traffic that is replicated by changing the swap file location on the
ESXi host. However, the completion of vSphere vMotion operations might take longer when the
swap file must be recreated because pages swapped to a local swap file on the source host must
be transferred across the network to the destination host.

Table 2-10. Design Decisions for Virtual Machine Swap Configuration of ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-006 For tenant workloads Simplifies the configuration Increases the amount of
running in the shared edge process. on-disk storage required
and workload cluster, save to host the entire virtual
the virtual machine swap machine state.
file at the default location.

Lifecycle Management Design for ESXi for a Virtual Infrastructure Workload Domain
When you decide on a life cycle management approach for the ESXi software, you consider the
effort and time required for preparing the environment and performing the patch, upgrade, or
update operation.

Life cycle management of ESXi is the process of performing patch updates or upgrades to
the underlying ESXi operating system. In a typical ESXi environment, you perform life cycle
management is by using vSphere Lifecycle Manager that is running in vCenter Server. When
implementing a solution with VMware Cloud Foundation, you use SDDC Manager for life cycle
management where additional components are included as part of the life cycle management
process.

Table 2-11. Design Decisions for Lifecycle Management of ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-007 Use SDDC Manager to SDDC Manager has a The operations team must
perform the life cycle greater awareness of the understand and be aware of
management of ESXi hosts full SDDC solution and the impact of performing a
in the shared edge and therefore handles the patch patch, upgrade, or upgrade
workload cluster. update or upgrade of the by using SDDC Manager.
workload domain as a
single process.
Directly performing life
cycle management tasks on
an ESXi host or through
vCenter Server has the
potential to cause issues
within SDDC Manager.

Network Design for ESXi for a Virtual Infrastructure Workload Domain

In the network design for the ESXi hosts in a workload domain, you place the hosts on a VLAN
for traffic segmentation and decide on the IP addressing scheme and name resolution for optimal
support for tenant workloads and maintenance of the hosts.

VMware, Inc. 23
Architecture and Design for a Virtual Infrastructure Workload Domain

Network Segments
To perform system functions in a virtual infrastructure in addition to providing network
connectivity to the virtual machines, the ESXi hosts in the workload domain are connected to
several dedicated networks.

Management network

Carries traffic for management of the ESXi hosts and communication to and from vCenter
Server. In addition, on this network, the hosts exchange heartbeat messages when vSphere
HA is enabled in non-vSAN clusters.

When using multiple availability zones, you use this network for vSAN witness traffic. See
vSAN Witness Network Design.

vSphere vMotion network

Carries traffic for relocating virtual machines between ESXi hosts with zero downtime.

vSAN network

Carries the communication between ESXi hosts in the cluster to implement a vSAN-based
shared storage. In addition, on this network, the hosts exchange heartbeat messages when
vSphere HA is enabled in vSAN clusters.

Underlay Transport Network

Carries overlay traffic between the management components in the management domain
and traffic for software-defined network services such as load balancing and dynamic routing
(East-West traffic).

Uplink Networks

Carry traffic for communication between software-defined overlay networks and the external
network (North-South traffic). In addition, on these networks, routing control packets are
exchanged to establish required routing adjacencies and peerings.

VMware, Inc. 24
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-12. Design Decisions on Network Segments for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-008 Place the ESXi hosts in n Physical VLAN security A new VLAN and a
the shared edge and separation between new subnet are required
workload cluster on a new VI Workload Domain for VI Workload Domain
VLAN-backed management ESXi hosts and management network.
network segment dedicated other management
for VI Workload Domain. components in
Management Domain is
achieved.
n Reduces the number
of VLANs needed as
a single VLAN can
be allocated for both
the ESXi hosts and
NSX-T Edge nodes in
the shared edge and
workload cluster.

IP Addressing
The IP address of the management interface of each ESXi host can be assigned by using DHCP
or statically. Assign static IP addresses and host names for ESXi management interfaces across all
ESXi hosts in the workload domain.

Table 2-13. Design Decisions on IP Address Scheme for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-009 Allocate statically assigned Ensures stability across the Requires precise IP address
IP addresses and host SDDC and makes it simpler management.
names across all ESXi hosts to maintain and makes it
in the shared edge and easier to track.
workload cluster.

Name Resolution
The hostname of each ESXi host in the workload domain is allocated to a specific domain for name
resolutionaccording to the region the host is in.

The IP address and host name of each ESXi host are associated with a fully qualified name ending
with a region-specific suffix, such as, sfo.rainpole.io for the first region of the SDDC, Region A,
and lax.rainpole.io for an additional region, Region B.

VMware, Inc. 25
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-14. Design Decisions on Name Resolution for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-010 Configure forward and All ESXi hosts are You must provide DNS
reverse DNS records for accessible by using a fully records for each ESXi host.
each ESXi host in the qualified domain name
shared edge and workload instead of by using IP
cluster, assigning the addresses only.
records to the child domain
in each region.

Time Synchronization
Time synchronization provided by the Network Time Protocol (NTP) is important to ensure that all
components in the SDDC are synchronized to the same time source. For example, if the clocks on
the physical machines in your vSphere network are not synchronized, SSL certificates and SAML
Tokens, which are time-sensitive, might not be recognized as valid in communications between
network machines. Time inconsistencies in vSphere can cause first-boot to fail at different services
depending on where in the environment time is not accurate and when the time is synchronized.

Table 2-15. Design Decisions on Time Synchronization for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-011 Configure time Ensures consistent time An operational NTP service

synchronization by using an across all devices in the must be available in the
internal NTP time source environment, which can environment.
across all ESXi hosts in the be critical for proper root
shared edge and workload cause analysis and auditing.
cluster.

SDDC-WLD-VI-ESXi-012 Set the NTP service policy Ensures that the NTP None.
to Start and stop with service is available right
host across all ESXi hosts after you restart an ESXi
in the shared edge and host.
workload cluster.

Information Security and Access Design for ESXi for a Virtual Infrastructure
Workload Domain
You design authentication access, controls, and certificate management for ESXi according to
industry standards and the requirements of your organization.

Host Access
After installation, you add ESXi hosts to a vCenter Server system for host management.

Direct access to the host console is still available and most commonly used for troubleshooting
purposes. You can access ESXi hosts directly by using one of these four methods:

VMware, Inc. 26
Architecture and Design for a Virtual Infrastructure Workload Domain

Method for ESXi Host Access Description

Direct Console User Interface (DCUI) Graphical interface on the console. Provides basic
administrative controls and troubleshooting options.

ESXi Shell A Linux-style bash login to the ESXi console itself.

Secure Shell (SSH) Access Remote command-line console access.

VMware Host Client HTML5-based client that has a similar interface to the
vSphere Client but for managing individual ESXi hosts
only. You use the VMware Host Client for emergency
management when vCenter Server is temporarily
unavailable.

You can enable or disable each method. By default, the ESXi Shell and SSH are disabled to protect
the ESXi host. The Direct Console User Interface is disabled only if Strict Lockdown Mode is
enabled.

Table 2-16. Design Decisions on Host Access for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-013 Configure the SSH service Ensures that on an ESXi Might be in direct conflict
policy to Start and stop host reboot, the SSH with your corporate security
with host across all ESXi service is started ensuring policy.
hosts in the shared edge access from SDDC Manager
and workload cluster. is maintained.

SDDC-WLD-VI-ESXi-014 Configure the advanced Ensures that only critical Might be in direct conflict
setting messages appear in with your corporate security
UserVars.SuppressShellWa the VMware Host Client policy.
rning 1. and vSphere Client by
suppressing the warning
message about enabled
local and remote shell
access.

User Access
By default, you can log in to an ESXi host only by using root account. To have more accounts that
can access the ESXi hosts in the workload domain, you can add the hosts to an Active Directory
domain. After the ESXi host has been added to an Active Directory domain, you can grant access
by using Active Directory groups. Auditing logins to the ESXi hosts becomes easier too.

VMware, Inc. 27
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-17. Design Decisions on User Access for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-015 Join each ESXi host in Using Active Directory Adding ESXi hosts to
the domain to the Active membership provides the Active Directory
Directory domain of the greater flexibility in granting domain can add some
region in which the ESXi access to ESXi hosts. administrative overhead.
host resides. Ensuring that users log in
with a unique user account
provides greater visibility
for auditing.

SDDC-WLD-VI-ESXi-016 Change the default ESX Using an Active Directory Additional changes to
Admins group to an Active group is more secure the ESXi hosts advanced
Directory group ug-esxi- because it removes settings are required.
admins. a known administrative
access point.

SDDC-WLD-VI-ESXi-017 Add ESXi administrators Adding ESXi administrator Administration of direct

to the ug-esxi-admins accounts to the Active user access is controlled by
group in Active Directory Directory group provides using Active Directory.
following standard access these benefits.
procedures. n Direct control on
the access to the
ESXi hosts by using
Active Directory group
membership
n Separation of
management tasks
n More visibility for access
auditing

Password Management and Account Lockout Behavior

ESXi enforces password requirements for access from the Direct Console User Interface, the
ESXi Shell, SSH, or the VMware Host Client. By default, you have to include a mix of characters
from four-character classes: Lowercase letters, uppercase letters, numbers, and special characters
such as underscore or dash when you create a password. By default, required password length
between 7 and 40 characters. Passwords cannot contain a dictionary word or part of a dictionary
word.

Account locking is supported for access by using SSH and the vSphere Web Services SDK.
The Direct Console Interface and the ESXi Shell do not support account lockout. By default, a
maximum of five failed attempts is allowed before the account is locked. The account is unlocked
after 15 minutes by default.

This design applies a password policy according to security best practices and standards.

VMware, Inc. 28
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-18. Example Password Policy Specification

Password Setting Value

Minimum Length 1

Maximum lifetime 60 days

Remember old passwords 5

that is, remember that previous 5 passwords so they do
not get reused.

Allow similar passwords Deny

Complexity At least 1 uppercase, 1 lowercase, 1 number, and 1 special

character

Max failed login attempts 3

Time interval between failures 900 seconds

Unlock time 0

Table 2-19. Design Decisions on Passwords and Account Lockout Behavior for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-018 Configure a policy for ESXi Aligns with security best None.
host password and account practices or industry
lockout according to the standards with which
security best practices or your organization maintains
industry standards with compliance.
which your organization
maintains compliance.

Certificate Management
To establish a secure connection with VMware SDDC Manager and prevent man-in-the-middle
(MiTM) attacks, the Common Name (CN) attribute of the certificate of each ESXi host must be set
to the FQDN of the host.

By default, ESXi hosts are deployed with a self-signed certificate whose CN attribute is set to
localhost.localdomain. As a result, after you assign an FQDN to each ESXi host in the domain,
you must regenerate the host certificate.

Table 2-20. Design Decisions on Certificate Management for ESXi Hosts

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-ESXi-019 Regenerate ESXi hosts Establishes a secure You must manually

certificates after assigning connection with VMware regenerate the certificates
them an FQDN. Cloud Builder during of the ESXi hosts before
the deployment of the deployment of the
the management domain workload domain.
and prevents man-in-the-
middle (MiTM) attacks.

VMware, Inc. 29
Architecture and Design for a Virtual Infrastructure Workload Domain

vCenter Server Design for a Virtual Infrastructure Workload Domain

The vCenter Server design includes determining the number of vCenter Server instances in the
workload domain, their size, networking configuration, cluster layout, redundancy, and security
configuration.

By using vCenter Server, you manage your vSphere infrastructure from a centralized location.
It acts as a central administration point for ESXi hosts and their respective virtual machines.
Implemented within the same appliance is the Platform Services Controller which provides a set
of infrastructure services including vCenter Single Sign-On, License service, Lookup Service, and
VMware Certificate Authority (VMCA).

n Logical Design for vCenter Server for a Virtual Infrastructure Workload Domain
For the workload domain, you deploy one vCenter Server appliance that manages the ESXi
hosts that run NSX-T Edge appliances and tenant workloads.

n Deployment Specification of vCenter Server for a Virtual Infrastructure Workload Domain

You determine the size of the compute resources, high availability implementation, and
patching and upgrade support for the workload domain vCenter Server according to the
design objectives and aggregated requirements of the SDDC.

n Network Design for vCenter Server for a Virtual Infrastructure Workload Domain
In the network design for the workload domain vCenter Server, you place vCenter Server on a
VLAN for traffic segmentation and decide on the IP addressing scheme and name resolution
for optimal support for the SDDC management components, host management, and tenant
workloads distribution.

n vSphere Cluster Design for a Virtual Infrastructure Workload Domain

The vSphere Cluster design must consider the characteristics of the workloads deployed in
the workload domain.

n Information Security and Access Design for vCenter Server for a Virtual Infrastructure
Workload Domain
You design authentication access, controls, and certificate management for the workload
domain vCenter Server according to industry standards and the requirements of your
organization.

Logical Design for vCenter Server for a Virtual Infrastructure Workload Domain
For the workload domain, you deploy one vCenter Server appliance that manages the ESXi hosts
that run NSX-T Edge appliances and tenant workloads.

A workload domain consists of one vCenter Server instance with an embedded Platform Services
Controller according to the scale, number of virtual machines, and availability requirements for
your environment.

VMware, Inc. 30
Architecture and Design for a Virtual Infrastructure Workload Domain

vCenter Server is deployed as a preconfigured virtual appliance that is running the VMware
Photon™ operating system. vCenter Server is required for some advanced vSphere features,
such as vSphere High Availability (vSphere HA), vSphere Fault Tolerance, vSphere Distributed
Resource Scheduler (vSphere DRS), vSphere vMotion, and vSphere Storage vMotion.

Figure 2-6. Logical Design of vCenter Server in the Workload Domain

Region A Region B

Identity Source Identity Source

Active Directory Active Directory

Access Enhanced Linked Mode Enhanced Linked Mode Access

User Interface User Interface

Management Management
Domain vCenter Domain vCenter
API Server Server API

Workload Domain Workload Domain

vCenter Server vCenter Server
Solution and
Virtual User Authentication Virtual
Appliance Appliance

vCenter Single
Sign-On Domain

Supporting Infrastructure: Supporting Infrastructure:

Shared Storage, DNS, NTP Shared Storage, DNS, NTP

Virtual Infrastructure Virtual Infrastructure

ESXi ESXi

VMware, Inc. 31
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-21. vCenter Server Configuration

Single Region Multiple Availability Zones Multiple Regions

n One vCenter Server instance that n One vCenter Server instance that n In each region, one vCenter
is allocated to the workload is allocated to the workload Server instance that is allocated
domain and the SDDC tenant domain and the SDDC tenant to the workload domain and the
workloads. workloads. SDDC tenant workloads.
n A should-run VM-Host affinity rule n One vCenter Single Sign-On
in vSphere DRS specifies that the domain for the vCenter Server
vCenter Server appliance should instances in both regions.
run in the primary availability zone n In Region A, a should-run VM-
unless an outage in this zone Host affinity rule in vSphere
occurs. DRS specifies that the vCenter
Server appliance should run in the
primary availability zone unless an
outage in this zone occurs.

Deployment Specification of vCenter Server for a Virtual Infrastructure Workload

Domain
You determine the size of the compute resources, high availability implementation, and patching
and upgrade support for the workload domain vCenter Server according to the design objectives
and aggregated requirements of the SDDC.

n Deployment Model for vCenter Server for a Virtual Infrastructure Workload Domain
The number of vCenter Server instances in your SDDC is determined by the number
of workload domains. Each workload domain has a single vCenter Server instance. You
determine the amount of compute and storage resources for the vCenter Server instance
according to the scale of the environment, the plans for deployment of virtual infrastructure
workload domains, and the requirements for isolation of management workloads from tenant
workloads.

n Enhanced Linked Mode Design for a Virtual Infrastructure Workload Domain

Enhanced Linked Mode (ELM) improves the manageability of the environment. By using
vCenter Enhanced Linked Mode, you can log in to all vCenter Server instances across
the SDDC that are joined to the same vCenter Single Sign-on domain and access their
inventories. Enhanced Linked Mode also replicates all global permissions, licenses, policies,
and tags between the linked vCenter Server instances.

n High Availability Design for vCenter Server for a Virtual Infrastructure Workload Domain
Protecting the vCenter Server system is important because it is the central point of
management and monitoring for the SDDC. You protect vCenter Server according to the
maximum downtime tolerated and whether fail-over automation is required.

VMware, Inc. 32
Architecture and Design for a Virtual Infrastructure Workload Domain

n Life Cycle Management Design of vCenter Server for a Virtual Infrastructure Workload
Domain
You decide on the life cycle management of the vCenter Server appliance according to
amount of time and effort to perform a deployment, upgrade, or patch operation and to the
impact such an operation has on the solutions that are connected to that vCenter Server.

Deployment Model for vCenter Server for a Virtual Infrastructure Workload Domain
The number of vCenter Server instances in your SDDC is determined by the number of workload
domains. Each workload domain has a single vCenter Server instance. You determine the amount
of compute and storage resources for the vCenter Server instance according to the scale of
the environment, the plans for deployment of virtual infrastructure workload domains, and the
requirements for isolation of management workloads from tenant workloads.
Deployment Type
You allocate a vCenter Server instance for the workload domain in each region, using Enhanced
Linked Mode to connect, view, and search across all linked vCenter Server systems in the SDDC.

VMware, Inc. 33
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-22. Design Decisions on the Deployment Model for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-001 Deploy a dedicated vCenter n Isolates vCenter Server None

Server system in the failures to management
primary workload domain or tenant workloads.
availability zone of each n Isolates vCenter Server
region for use by the operations between
workload domain. management and
tenants.
n Supports a scalable
cluster design where
you might reuse
the management
components as more
tenant workloads are
added to the SDDC.
n Simplifies capacity
planning for tenant
workloads because
you do not consider
management workloads
for the Workload
domain vCenter Server.
n Improves the ability to
upgrade the vSphere
environment and
related components
by providing for
explicit separation of
maintenance windows:
n Management
workloads remain
available while you
are upgrading the
tenant workloads
n Tenant workloads
remain available
while you are
upgrading the
management nodes
n Supports clear
separation of roles
and responsibilities
to ensure that only
administrators with
proper authorization
can attend to
the management
workloads.
n Facilitates quicker
troubleshooting and
problem resolution.

VMware, Inc. 34
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-22. Design Decisions on the Deployment Model for a Workload Domain vCenter Server
(continued)

Decision ID Design Decision Design Justification Design Implication

n Simplifies disaster
recovery operations
by supporting a
clear demarcation
between recovery
of the management
components and
compute workloads.
n Provides isolation of
potential network issues
by introducing network
separation of the
clusters in the SDDC.

SDDC-WLD-VI-VC-002 When using two availability Ensures that, by default, None.

zones in Region A, add the the vCenter Server instance
vCenter Server instance to is powered on within the
the VM group of Availability primary availability zone
Zone 1 in the management hosts group.
cluster.

Sizing Compute and Storage Resources

When you deploy the vCenter Server appliance, you select to deploy an appliance that is suitable
for the size of your environment. The option that you select determines the number of CPUs and
the amount of memory for the appliance.

Table 2-23. Compute Resource Specifications of vCenter Server

vCenter Server Appliance
Size Management Capacity Number of vCPUs Memory

X-Large environment Up to 2,000 hosts or 24 56 GB

35,000 virtual machines

Large environment Up to 1,000 hosts or 16 37 GB

10,000 virtual machines

Medium environment Up to 400 hosts or 4,000 8 28 GB

virtual machines

Small environment Up to 100 hosts or 1,000 4 19 GB

virtual machines

Tiny environment Up to 10 hosts or 100 virtual 2 12 GB

machines

When you deploy the vCenter Server appliance, the ESXi host or cluster on which you deploy
the appliance must meet minimum storage requirements. You determine the required storage not
only according to the size of the environment and the storage size, but also according to the disk
provisioning mode.

VMware, Inc. 35
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-24. Storage Resource Specifications of vCenter Server

Management
Appliance Size Capacity Default Storage Size Large Storage Size X-Large Storage Size

X-Large environment Up to 2,000 hosts 1805 GB 1905 GB 3665 GB

or 35,000 virtual
machines

Large environment Up to 1,000 hosts 1065 GB 1765 GB 3525 GB

or 10,000 virtual
machines

Medium environment Up to 400 hosts 700 GB 1700 GB 3460 GB

or 4,000 virtual
machines

Small environment Up to 100 hosts 480 GB 1535 GB 3295 GB

or 1,000 virtual
machines

Tiny environment Up to 10 hosts or 100 415 GB 1490 GB 3245 GB

virtual machines

Table 2-25. Design Decisions on Sizing a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-003 Deploy an appliance for the A vCenter Server appliance If the size of the workload
workload domain vCenter of a medium-deployment domain grows, you might
Server of a medium size is typically sufficient to have to increase the
deployment size or larger. manage tenant workloads vCenter Server Appliance
that run in a workload size.
domain.

SDDC-WLD-VI-VC-004 Deploy the workload The default storage None.

domain vCenter Server with capacity assigned to a
the default storage size. medium-sized appliance is
sufficient.

Enhanced Linked Mode Design for a Virtual Infrastructure Workload Domain

Enhanced Linked Mode (ELM) improves the manageability of the environment. By using vCenter
Enhanced Linked Mode, you can log in to all vCenter Server instances across the SDDC that are
joined to the same vCenter Single Sign-on domain and access their inventories. Enhanced Linked
Mode also replicates all global permissions, licenses, policies, and tags between the linked vCenter
Server instances.

You can join up to 15 vCenter Server virtual appliance deployments in a single vSphere Single
Sign-On domain.

VMware, Inc. 36
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-26. Enhanced Linked Mode across Regions and in a Region

ELM within the
Design Component ELM across Regions Region Only Considerations

Manageability ↑↑ ↓ You join all vCenter Server instances,

across regions, in one vCenter
Single Sign-On domain to improve
manageability. You replicate all global
permissions, licenses, policies, and tags,
and you can view the inventories of all
vCenter Server instances.

Scalability ↓↓ ↑↑ You join vCenter Server instances

in every region to separate vCenter
Single Sign-On domains to improve
scalability. By using that approach you
can deploy a total of 15 domains per
region, compared to a total of 15
domains for both regions if you share a
vCenter Single Sign-On domain between
regions.

VMware Validated Design uses a shared vCenter Single Sign-On domain for both regions. You can
also use different vCenter Single Sign-On domains for your regions according to the manageability
and scalability requirements of your organization.

VMware, Inc. 37
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-27. Design Decisions on vCenter Enhanced Linked Mode for the Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-005 Join all vCenter Server When all vCenter Server n Only one vCenter Single
instances to a single instances are joined in to a Sign-On domain exists.
vCenter Single Sign-On single vCenter Single Sign- n The number of
domain. On domain, they can share linked vCenter Server
authentication and license instances in the
data across all components same vCenter Single
and regions. Sign-On domain is
limited to 15 instances.
Because each workload
domain can contain a
single vCenter Server
instance, one VMware
Cloud Foundation
instance can contain no
more than 15 workload
domains, including the
management domain.

SDDC-WLD-VI-VC-006 Create a ring topology for By default, one vCenter None

the Single Sign-On domain Server instance replicates
running in the vCenter only with another vCenter
Server instances. Server instance. This setup
creates a single point of
failure for replication. A ring
topology ensures that each
vCenter Server instance has
two replication partners and
removes any single point of
failure.

High Availability Design for vCenter Server for a Virtual Infrastructure Workload Domain
Protecting the vCenter Server system is important because it is the central point of management
and monitoring for the SDDC. You protect vCenter Server according to the maximum downtime
tolerated and whether fail-over automation is required.

You can use the following methods for protecting the vCenter Server Appliance:

Table 2-28. Methods for Protecting the vCenter Server Appliance

Redundancy Method Protects vCenter Server Appliance

Automated protection using vSphere HA Yes

Manual configuration and manual failover, for example, Yes

using a cold standby clone.

HA cluster with external load balancer Not Available

vCenter Server HA Yes

VMware, Inc. 38
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-29. Design Decisions on High Availability of a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-007 Protect the workload Supports the availability vCenter Server becomes
domain vCenter Server objectives for vCenter unavailable during a
appliance by using vSphere Server appliances vSphere HA failover.
HA. without requiring manual
intervention if a failure
occurs.

SDDC-WLD-VI-VC-008 In vSphere HA, set the vCenter Server is the If the restart priority for
restart priority policy for the management and control another virtual machines
vCenter Server appliance to plane for physical and is set to highest, the
high. virtual infrastructure. In connectivity delays for
a HA event, vCenter management components
should be available first will be longer.
before other management
components come online to
ensure the rest of the SDDC
management stack comes
up cleanly.

Life Cycle Management Design of vCenter Server for a Virtual Infrastructure Workload Domain
You decide on the life cycle management of the vCenter Server appliance according to amount of
time and effort to perform a deployment, upgrade, or patch operation and to the impact such an
operation has on the solutions that are connected to that vCenter Server.

Life cycle management of vCenter Server involves the process of performing patch updates or
upgrades to the vCenter Server appliance. In a typical vSphere environment, you perform life
cycle management manually. When you implement a solution by using VMware Cloud Foundation,
you use SDDC Manager for life cycle management where additional components are included as
part of the life cycle management process.

Table 2-30. Design Decisions on Life Cycle Management of a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-009 Use SDDC Manager to Because the deployment The operations team must
perform the lifecycle scope of SDDC Manager have clear understanding
management of the covers the full SDDC and awareness of
workload domain vCenter stack, SDDC Manager the impact of a
Server. performs patching, update, patch, segmentation, and
or upgrade of the workload upgrade, or upgrade
domain as a single process. operation by using SDDC
Directly performing lifecycle Manager.
management tasks through
vCenter Server has the
potential to cause issues
within SDDC Manager.

VMware, Inc. 39
Architecture and Design for a Virtual Infrastructure Workload Domain

Network Design for vCenter Server for a Virtual Infrastructure Workload Domain
In the network design for the workload domain vCenter Server, you place vCenter Server on
a VLAN for traffic segmentation and decide on the IP addressing scheme and name resolution
for optimal support for the SDDC management components, host management, and tenant
workloads distribution.

Network Segments
For secure access to the vSphere Client and vCenter Server APIs, in each region, the workload
domain vCenter Server is connected to the management VLAN network segment.

Figure 2-7. vCenter Server Network Design

Data Active Internet/

Center Directory Enterprise
User Network

Region A Region B
(SFO - San Francisco) (LAX - Los Angeles)

Physical Management Domain Workload Domain Workload Domain Management Domain Physical
Upstream vCenter Server vCenter Server vCenter Server vCenter Server Upstream
Router sfo-m01-vc01. sfo-w01-vc01. lax-w01-vc01. lax-w01-vc01. Router
sfo.rainpole.local sfo.rainpole.local lax.rainpole.local lax.rainpole.local

172.16.11.0/24 172.17.11.0/24
VLAN: sfo01-m01-cl01-vds01-pg-mgmt VLAN: lax-m01-cl01-vds01-pg-mgmt

NSX-T Tier-0 Gateway/ Tier-1 Gateway

Table 2-31. Design Decisions on Network Segment for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-010 Place the appliance of the Reduces the number of None.

workload domain vCenter VLANs needed as a single
Server on the management VLAN can be allocated
VLAN network segment for both vCenter Server
in each region, that and NSX-T for Data Center
is, sfo-m01-cl01-vds01-pg- management components.
mgmt for Region A
and lax-m01-cl01-vds01-
pg-mgmt for Region B.

VMware, Inc. 40
Architecture and Design for a Virtual Infrastructure Workload Domain

IP Addressing
You can assign the IP address of the workload domain vCenter Server by using DHCP or statically
according to network configuration in your environment.

Table 2-32. Design Decisions on IP Addressing Scheme for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-011 Allocate a statically Ensures stability across the Requires precise IP address
assigned IP address and SDDC, makes it simpler management.
host name to the appliance to maintain and track,
of the workload domain and to implement a DNS
vCenter Server. configuration.

Name Resolution
vCenter Server appliances must maintain network connections to the following components:

n Systems running vCenter Server add-on modules

n Each ESXi host

Table 2-33. Design Decisions on Name Resolution for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-012 Configure forward and The vCenter Server You must provide DNS
reverse DNS records for the appliance is accessible by records for the workload
appliance of the workload using a fully qualified domain vCenter Server
domain vCenter Server, domain name instead of by appliance in each region.
assigning the record to using IP addresses only.
the child domain in each
region.

Time Synchronization
Time synchronization provided by the Network Time Protocol (NTP) is important to ensure that all
components within the SDDC are synchronized to the same time source.

Table 2-34. Design Decisions on Time Synchronization for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-013 Configure time Prevents from failures in the n An operational NTP

synchronization using an deployment of the vCenter service must be
internal NTP time for the Server appliance on an ESXi available to the
appliance for the workload host if the host is not using environment.
domain vCenter Server. NTP. n All firewalls between
the vCenter Server
appliance and the NTP
servers must allow NTP
traffic on the required
network ports.

VMware, Inc. 41
Architecture and Design for a Virtual Infrastructure Workload Domain

vSphere Cluster Design for a Virtual Infrastructure Workload Domain

The vSphere Cluster design must consider the characteristics of the workloads deployed in the
workload domain.

When you design the cluster layout in vSphere, consider the following guidelines:

n Use fewer, larger ESXi hosts, or more, smaller ESXi hosts.

n A scale-up cluster has fewer, larger ESXi hosts.

n A scale-out cluster has more, smaller ESXi hosts.

n Compare the capital costs of purchasing fewer, larger ESXi hosts with the costs of purchasing
more, smaller ESXi hosts. Costs vary between vendors and models.

n Evaluate the operational costs for managing a few ESXi hosts with the costs of managing more
ESXi hosts.

n Consider the purpose of the cluster.

n Consider the total number of ESXi hosts and cluster limits.

Figure 2-8. vSphere Logical Cluster Layout with a Single Availability Zone

Region A Region A Region B Region B

Management Cluster Shared Edge and Management Cluster Shared Edge and
Workload Cluster Workload Cluster
Management Workload Management Workload
Domain Domain Domain Domain
vCenter vCenter vCenter vCenter
Server Server APP APP APP Server Server APP APP APP
OS OS OS OS OS OS

ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi

VMware, Inc. 42
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-9. vSphere Logical Cluster Layout with Two Availability Zones

Region A Region A Region B Region B

Management Cluster Shared Edge and Management Cluster Shared Edge and
Workload Cluster Workload Cluster
Management Workload Management Workload
Domain Domain Domain Domain
vCenter vCenter vCenter vCenter
Server Server APP APP APP Server Server APP APP APP
OS OS OS OS OS OS

ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi

Availability Zone 1 Availability Zone 1

Availability Zone 2 Availability Zone 2

ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi

Table 2-35. Number of Hosts in the Shared Edge and Workload Cluster of the Workload Domain

Attribute Specification

Number of ESXi hosts that is required to support management 3

virtual machines with no memory over-commitment

Number of ESXi hosts recommended to handle operational 4

constraint, that is, be able to take a host offline without losing
high availability capabilities.

Number of ESXi hosts recommended to operational constraints, Single availability zone 4

while using vSAN, that is, be able to take a host offline without
losing high availability capabilities. Two availability zones 8

Reserved capacity for handling ESXi host failures per cluster Single availability zone 25% CPU and RAM
Tolerates one host failure

Two availability zone 50% CPU and RAM

Tolerates one availability
zone failure

VMware, Inc. 43
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-36. Design Decisions on the Host Configuration for the Shared Edge and Workload
Cluster in the Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-014 For each region in the n Simplifies configuration Management of multiple

workload domain, create a by isolating clusters and vCenter
shared edge and workload tenant workloads Server instances increases
vSphere cluster. from management operational overhead.
workloads.
n Ensures that
management workloads
have no impact on
tenant workloads.
You can add ESXi hosts to
the cluster as needed.

SDDC-WLD-VI-VC-015 In Region A, create n Allocating 4 ESXi To support redundancy,

the shared edge and hosts provides full you must allocate additional
workload cluster with this redundancy for each ESXi host resources.
configuration. availability zone in the
n A minimum of 4 ESXi cluster.
hosts for a single n Having 4 ESXi hosts
availability zone in each availability
n A minimum of 8 ESXi zone guarantees vSAN
hosts for two availability and NSX redundancy
zones, that is, minimum during availability zone
of 4 ESXi hosts in each outages or maintenance
availability zone. operations.

SDDC-WLD-VI-VC-016 In Region B, create a n Allocating 4 ESXi To support redundancy,

shared edge and workload hosts provides full you must allocate additional
cluster with a minimum of 4 redundancy for the ESXi host resources .
ESXi hosts. cluster.
n Having four ESXi
hosts guarantees vSAN
and NSX redundancy
during maintenance
operations.

n vSphere HA Design for a Virtual Infrastructure Workload Domain

The vSphere HA configuration works to ensure uptime of virtual machines in the SDDC.
You consider the varying and sometimes significant CPU or memory reservations for virtual
machines and the requirements of vSAN.

n vSphere DRS Design for a Virtual Infrastructure Workload Domain

vSphere Distributed Resource Scheduling (vSphere DRS) provides automated load
distribution within a vSphere Cluster by migrating workloads from heavily loaded ESXi hosts
to ESXi hosts with more available resources in the vSphere Cluster, as well as by providing
intelligence around initial workload placement. vSphere DRS supports manual and automatic
modes

VMware, Inc. 44
Architecture and Design for a Virtual Infrastructure Workload Domain

n vSphere EVC Design for a Virtual Infrastructure Workload Domain

If you enable vSphere Enhanced vMotion Compatibility (EVC) in the domain, the virtual
machines of the SDDC can be migrated between ESXi hosts containing older CPUs. You
can use EVC for a rolling upgrade of all hardware with zero downtime.

vSphere HA Design for a Virtual Infrastructure Workload Domain

The vSphere HA configuration works to ensure uptime of virtual machines in the SDDC. You
consider the varying and sometimes significant CPU or memory reservations for virtual machines
and the requirements of vSAN.

If an ESXi host failure occurs, vSphere HA restarts virtual machines on remaining ESXi hosts in
the vSphere Cluster. A primary ESXi host communicates with the workload domain vCenter Server
and monitors the virtual machines and secondary ESXi hosts in the cluster. vSphere HA uses
admission control to ensure that sufficient resources are reserved for virtual machine recovery
when an ESXi host fails.

You configure several vSphere HA features to provide high availability for the VMs in the SDDC.

Table 2-37. vSphere HA Features Configured for the SDDC

vSphere HA Feature Description

Host failure response vSphere HA can respond to individual host failures by

restarting virtual machines on other hosts within the
cluster.

Response for host isolation If a host becomes isolated, vSphere HA can detect and
shut down or restart virtual machines on available hosts.

Datastore with PDL or APD When virtual machines are hosted on non-vSAN
datastores, vSphere HA can detect datastore outages
and restart virtual machines on hosts that have datastore
access.

Admission control policy Configure how the cluster determines available resources.
In a smaller vSphere HA cluster, a larger proportion of the
cluster resources are reserved to accommodate ESXi host
failures according to the selected admission control policy.

VM and Application Monitoring If a virtual machine failure occurs, the VM and Application
Monitoring service restarts that virtual machine. The
service uses VMware Tools to evaluate whether a virtual
machine in the cluster is running.

VMware, Inc. 45
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-38. Admission Control Policies in vSphere HA

Policy Name Description

Host failures the cluster tolerates vSphere HA ensures that a specified number of ESXi hosts
can fail and sufficient resources remain in the cluster to fail
over all the virtual machines from those ESXi hosts.

Percentage of cluster resources reserved vSphere HA reserves a specified percentage of aggregate

CPU and memory resources for failover.

Specify failover hosts When an ESXi host fails, vSphere HA attempts to restart
its virtual machines on any of the specified failover ESXi
hosts. If restart is not possible, for example, the failover
ESXi hosts have insufficient resources or have failed as
well, then vSphere HA attempts to restart the virtual
machines on other ESXi hosts in the vSphere Cluster.

Table 2-39. Design Decisions on the Admission Control Policy for the Shared Edge and Workload
Cluster in a Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-017 When using a single Using the percentage- In a shared edge and
availability zone, configure based reservation works workload cluster of 4 ESXi
admission control for 1 well in situations where hosts, only the resources of
ESXi host failure and virtual machines have 3 ESXi hosts are available
percentage-based failover varying and sometime for use.
capacity. significant CPU or memory
reservations.
vSphere automatically
calculates the reserved
percentage based on ESXi
host failures to tolerate and
the number of ESXi hosts
in the shared edge and
workload cluster.

SDDC-WLD-VI-VC-018 When using two availability Allocating only half of a In a shared edge and
zones, configure admission stretched cluster ensures workload cluster of 8 ESXi
control for percentage- that all VMs have enough hosts, only the resources of
based failover based on half resources if an availability 4 ESXi hosts are available
of the ESXi hosts in the zone outage occurs. for use.
cluster. If you add more ESXi
hosts to the cluster, add
them in pairs, one in each
availability zone.

SDDC-WLD-VI-VC-019 When using a single Allows vSphere HA to You must manually

availability zone, set the validate complete network configure the isolation
isolation address for the isolation if a connection address in case of using a
cluster to the gateway failure occurs on ESXi single availability zone.
IP address for the vSAN host or between availability
network. zones.

VMware, Inc. 46
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-39. Design Decisions on the Admission Control Policy for the Shared Edge and Workload
Cluster in a Workload Domain (continued)

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-020 When using two availability Allows vSphere HA to None.

zones, set two isolation validate complete network
addresses, one for each isolation if a connection
vSAN network gateway in failure occurs on an ESXi
each availability zone. host or between availability
zones.

SDDC-WLD-VI-VC-021 Set the advanced cluster Ensures that vSphere None.

setting HA uses the manual
das.usedefaultisolationa isolation addresses instead
ddress to false. of the default management
network gateway address.

SDDC-WLD-VI-VC-022 Use vSphere HA in vSphere HA supports a You must provide sufficient

the shared edge and robust level of protection resources on the remaining
workload cluster to increase for both ESXi host and hosts so that virtual
the availability of tenant virtual machine availability. machines can be migrated
workloads. to those hosts in the event
of a host outage.

SDDC-WLD-VI-VC-023 Set host isolation to Power vSAN requires that the If a false-positive event
Off and Restart VMs in host isolation response be occurs, VMs are powered
vSphere HA. set to Power Off and to off and an ESXi host
restart VMs on available is declared isolated
ESXi hosts. incorrectly.

SDDC-WLD-VI-VC-024 Set datastore with Availability of virtual Only virtual machines

Permanent Device Loss machines that run on that run on non-vSAN
(PDL) to Power Off and non-vSAN datastores can datastores can be protected
Restart VMs in vSphere HA. be increased by using by VM Component
VM Component Protection. Protection.
If a host experiences a
Permanent Device Loss
event, vSphere HA can
restart the affected virtual
machines on other hosts
that can still access that
datastore.

SDDC-WLD-VI-VC-025 Set datastore with All- Availability of virtual Only virtual machines
Paths-Down (APD) to machines that run on non- that run on non-vSAN
Power Off and Restart VMs vSAN datastores can be datastores can be protected
- Conservative restart policy increased by using VM by VM Component
in vSphere HA. Component Protection. If Protection.
a host experiences an All-
Paths-Down event, vSphere
HA can restart the affected
virtual machines on other
hosts that can still access
that datastore.

VMware, Inc. 47
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-40. Design Decisions on VM and Application Monitoring Service for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-026 Enable VM Monitoring for VM Monitoring provides None.

the shared edge and in-guest protection for
workload cluster. most VM workloads. The
application or service
running on the virtual
machine must be capable
of restarting successfully
after a reboot or the
virtual machine restart is
not sufficient.

vSphere DRS Design for a Virtual Infrastructure Workload Domain

vSphere Distributed Resource Scheduling (vSphere DRS) provides automated load distribution
within a vSphere Cluster by migrating workloads from heavily loaded ESXi hosts to ESXi hosts with
more available resources in the vSphere Cluster, as well as by providing intelligence around initial
workload placement. vSphere DRS supports manual and automatic modes

Table 2-41. vSphere DRS Automation Modes

Automation Mode Description

Manual vSphere DRS provides recommendations, but an

administrator must confirm the changes.

Partially Automated n vSphere DRS provides recommendations to migrate

virtual machines to fulfill certain criteria and an
administrator must confirm the changes.
n vSphere DRS automatically places virtual machines on
hosts with sufficient resources at power-on operations.

Fully Automated vSphere DRS automatically migrates VMs to fulfill

certain criteria and automatically places workloads onto
appropriate hosts at power-on operations.

When using two availability zones, enable vSphere DRS to create VM-Host group affinity rules for
initial placement of virtual machines, improving read locality by ensuring a local copy of virtual
machine data in each availability zone. In this way, you avoid unnecessary vSphere vMotion
migration of virtual machines between availability zones.

Because the vSAN stretched cluster is still one cluster, vSphere DRS is unaware that the cluster
stretches across different physical locations. As result, vSphere DRS might decide to move virtual
machines between these locations. By using VM-Host group affinity rules, you can constrain
virtual machines to an availability zone. Otherwise, if a virtual machine, VM1, that resides in
Availability Zone 1, moves across availability zones, VM1 might eventually be running in Availability
Zone 2. Because vSAN stretched clusters implement read locality, the cache for the virtual
machine in Availability Zone 1 is warm whereas the cache in Availability Zone 2 is cold. This
situation might impact the performance of VM1 until the cache for it in Availability Zone 2 is
warmed up.

VMware, Inc. 48
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-42. Design Decisions for vSphere DRS for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-027 Enable vSphere DRS Provides the best trade- If a vCenter Server outage
(Distributed Resource off between load balancing occurs, mapping from
Scheduling) on all clusters, and unnecessary migration virtual machines to ESXi
using the default fully with vSphere vMotion. hosts might be difficult to
automated mode (medium) determine.

SDDC-WLD-VI-VC-028 Create virtual machine By creating virtual machine Creating the groups is
groups for use in startup groups, you can use a manual task and adds
rules for the shared edge rules to configure the administrative overhead.
and workload cluster. startup order of NSX-T
Edge appliances and tenant
workloads.

SDDC-WLD-VI-VC-029 Create virtual machine rules The NSX-T Edge Creating the groups is
to ensure NSX-T Edge nodes in the workload a manual task and adds
nodes start before any domain provide north- administrative overhead.
tenant workloads. south network traffic
for tenant workloads.
Give precedence to the
functionality over tenant
workloads in the shared
edge and workload cluster.

SDDC-WLD-VI-VC-030 When using two availability Makes it easier to manage You must create and
zones, create a host group which virtual machines run maintain VM-Host DRS
and add the ESXi hosts in in which availability zone. group rules.
Availability Zone 1 in Region
A to it.

SDDC-WLD-VI-VC-031 When using two availability Makes it easier to manage You must create and
zones, create a host group which virtual machines run maintain VM-Host DRS
and add the ESXi hosts in which availability zone. group rules.
in Availability Zone 2 in
Region A to it.

SDDC-WLD-VI-VC-032 When using two availability Ensures that virtual You must add VMs to the
zones, create a virtual machines are located only allocated group manually.
machine group and add the in the assigned availability
virtual machines Availability zone.
Zone 1 in Region A to it.

SDDC-WLD-VI-VC-033 When using two availability Ensures that virtual You must add VMs to the
zones, create a virtual machines are located only allocated group manually.
machine group and add in the assigned availability
the virtual machines in zone.
Availability Zone 2 in
Region A to it.

VMware, Inc. 49
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-42. Design Decisions for vSphere DRS for a Workload Domain (continued)

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-034 When using two availability Ensures that virtual Creating the groups is
zones, create a should-run machines are located only a manual task and adds
VM-Host affinity rule to in the assigned availability administrative overhead.
run the group of VMs in zone.
Availability Zone 1 on the
group of hosts in the same
zone.

SDDC-WLD-VI-VC-035 When using two availability Ensures that virtual Creating the groups is
zones, create a should-run machines are located only a manual task and adds
VM-Host affinity rule to in the assigned availability administrative overhead.
run the group of VMs in zone.
Availability Zone 2 on the
group of hosts in the same
zone.

vSphere EVC Design for a Virtual Infrastructure Workload Domain

If you enable vSphere Enhanced vMotion Compatibility (EVC) in the domain, the virtual machines
of the SDDC can be migrated between ESXi hosts containing older CPUs. You can use EVC for a
rolling upgrade of all hardware with zero downtime.

vSphere Enhanced vMotion Compatibility (EVC) works by masking certain features of newer CPUs
to allow migration between ESXi hosts containing older CPUs. If you set EVC during cluster
creation, you can add ESXi hosts with newer CPUs later without disruption. You can use EVC for a
rolling upgrade of all hardware with zero downtime.

EVC works only with CPUs from the same manufacturer and there are limits to the version
difference gaps between the CPU families.

Table 2-43. Design Decisions on Enhanced vMotion Compatibility for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-036 Enable Enhanced vMotion Supports cluster upgrades You can enable EVC only
Compatibility (EVC) on all without virtual machine if clusters contain hosts
clusters. downtime. with CPUs from the same
vendor.

SDDC-WLD-VI-VC-037 Set the cluster EVC mode Supports cluster upgrades None.
to the highest available without virtual machine
baseline that is supported downtime.
for the lowest CPU
architecture on the hosts in
the cluster.

Information Security and Access Design for vCenter Server for a Virtual
Infrastructure Workload Domain
You design authentication access, controls, and certificate management for the workload domain
vCenter Server according to industry standards and the requirements of your organization.

VMware, Inc. 50
Architecture and Design for a Virtual Infrastructure Workload Domain

Identity Management
Users can log in to vCenter Server only if they are in a domain that was added as a vCenter Single
Sign-On identity source. vCenter Single Sign-On administrator users can add identity sources or
change the settings for identity sources that they added. An identity source can be a native Active
Directory (Integrated Windows Authentication) domain or an OpenLDAP directory service. For
backward compatibility, Active Directory as an LDAP server is also available.

Table 2-44. Design Decisions on Identity Management for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-038 Join the workload domain n Using Active Directory Joining vCenter Server to
vCenter Server to the membership provides the domain adds some
Active Directory domain for greater flexibility in administrative overhead.
the region that vCenter granting access to
Server resides in. vCenter Server.
n Ensuring that users
log in with a unique
user account provides
greater visibility for
auditing.

SDDC-WLD-VI-VC-039 Assign global permissions By assigning the The Active Directory group
to the vCenter Server Administrator role to an must be created in
inventory to an Active Active Directory group, advance to assigning it the
Directory group, such as you can easily create Administrator role.
ug-vc-admins, by using the user accounts that have
Administrator role. administrative rights on
vCenter Server.

Password Management and Account Lockout Behavior

vCenter Server enforces password requirements for access to the vCenter Server Management
Interface. By default, you must include at least six characters, which should not be any of
your previous five passwords. Account locking is supported for access to the vCenter Server
Management Interface. By default, passwords are set to expire after 365 days.

This design applies a password policy according to security best practices and standards.

Table 2-45. Example Password Policy Specification for vCenter Server

Password Setting Value

Minimum Length 15

Maximum lifetime 60 days

Remember old passwords 5 (remember previous 5 so they don't get reused)

Allow similar passwords Deny

Complexity At least 1 upper case, 1 lower case, 1 number, and 1 special

char

Max failed login attempts 3

VMware, Inc. 51
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-45. Example Password Policy Specification for vCenter Server (continued)

Password Setting Value

Time interval between failures 900 seconds

Unlock time 0

Table 2-46. Design Decisions on Password and Account Lockout for a Workload Domain vCenter
Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-040 Configure a password and Aligns with the industry None.

account lockout policy standard across your
for the appliance of the organization.
workload domain vCenter
Server according to the
industry standard for
security and compliance of
your organization.

Certificate Management
Access to all vCenter Server interfaces must use an SSL connection. By default, vCenter Server
uses a self-signed certificate for the appliance which is signed by the VMware Certificate Authority
(VMCA). To provide secure access to the vCenter Server appliance, replace the default VMCA-
signed certificate with a CA-signed certificate.

Table 2-47. Design Decisions on Certificate Management for a Workload Domain vCenter Server

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-VC-041 Replace the default VMCA- Ensures that the Replacing the default
signed certificate of the communication to the certificates with trusted CA-
appliance of the workload externally facing Web signed certificates from a
domain vCenter Server with user interface and API certificate authority might
a CA-signed certificate. to vCenter Server, and increase the deployment
between vCenter Server preparation time because
and other management you must generate and
components is encrypted. submit certificates requests.

SDDC-WLD-VI-VC-042 Use a SHA-2 algorithm The SHA-1 algorithm is Not all certificate authorities
or higher when signing considered less secure and support SHA-2.
certificates. has been deprecated.

SDDC-WLD-VI-VC-043 Perform SSL certificate SDDC Manager provides None.

life cycle management for automated SSL certificate
vCenter Server through lifecycle management
SDDC Manager. rather than requiring a
series of manual steps to be
performed.

VMware, Inc. 52
Architecture and Design for a Virtual Infrastructure Workload Domain

vSphere Networking Design for a Virtual Infrastructure Workload

Domain
The network design prevents unauthorized access and provides timely access to business
data. This design uses vSphere Distributed Switch and VMware NSX-T Data Center for virtual
networking.

Virtual Network Design Guidelines

The high-level design goals apply regardless of your environment.

Table 2-48. Goals of the vSphere Networking Design

Design Goal Description

Meet diverse needs The network must meet the diverse needs of many different entities in an organization.
These entities include applications, services, storage, administrators, and users.

Reduce costs Reducing costs is one of the simpler goals to achieve in the vSphere infrastructure. Server
consolidation alone reduces network costs by reducing the number of required network
ports and NICs, but a more efficient network design is desirable. For example, configuring
two 25-GbE NICs might be more cost effective than configuring four 10-GbE NICs.

Improve performance You can achieve performance improvement and decrease the time that is required to
perform maintenance by providing sufficient bandwidth, which reduces contention and
latency.

Improve availability A well-designed network improves availability, usually by providing network redundancy.

Support security A well-designed network supports an acceptable level of security through controlled access
and isolation, where required.

Enhance infrastructure You can configure the network to support vSphere features such as vSphere vMotion,
functionality vSphere High Availability, and vSphere Fault Tolerance.

Follow networking best practices throughout your environment.

n Separate network services from one another to achieve greater security and better
performance.

n Use Network I/O Control and traffic shaping to guarantee bandwidth to critical virtual
machines. During network contention, these critical virtual machines will receive a higher
percentage of the bandwidth.

n Separate network services on a single vSphere Distributed Switch by attaching them to port
groups with different VLAN IDs.

n Keep vSphere vMotion traffic on a separate network.

When a migration using vSphere vMotion occurs, the contents of the memory of the guest
operating system is transmitted over the network. You can place vSphere vMotion on a
separate network by using a dedicated vSphere vMotion VLAN.

n When using pass-through devices with Linux kernel version 2.6.20 or an earlier guest OS,
avoid MSI and MSI-X modes. These modes have significant performance impact.

n For best performance, use VMXNET3 virtual machine NICs.

VMware, Inc. 53
Architecture and Design for a Virtual Infrastructure Workload Domain

n Ensure that physical network adapters that are connected to the same vSphere Standard
Switch or vSphere Distributed Switch, are also connected to the same physical network.

Network Segmentation and VLANs

Separating different types of traffic is required to reduce contention and latency, and for access
security.

High latency on any network can negatively affect performance. Some components are more
sensitive to high latency than others. For example, reducing latency is important on the IP
storage and the vSphere Fault Tolerance logging network because latency on these networks
can negatively affect the performance of multiple virtual machines. According to the application or
service, high latency on specific virtual machine networks can also negatively affect performance.
Use information gathered from the current state analysis and from interviews with key stakeholder
and SMEs to determine which workloads and networks are especially sensitive to high latency.

Virtual Networks
Determine the number of networks or VLANs that are required depending on the type of traffic

Single Region Networks

n vSphere system traffic

n Management

n vSphere vMotion

n vSAN

n NFS

n TEP

n Traffic that supports the services and applications in the organization

n NSX-T Edge TEPs

n NSX-T Edge uplinks

Dual Regions Networks

n vSphere system traffic

n Management

n vSphere vMotion

n vSAN

n NFS

n TEP

n Traffic that supports the services and applications in the organization

n NSX-T Edge TEPs

VMware, Inc. 54
Architecture and Design for a Virtual Infrastructure Workload Domain

n NSX-T Edge RTEPs

n NSX-T Edge uplinks

n Virtual Switch Type Design for a Virtual Infrastructure Workload Domain

Virtual switches simplify the configuration process by providing a single pane of glass for
performing virtual network management tasks.

n vSphere Distributed Switch Design for a Virtual Infrastructure Workload Domain

The shared edge and workload cluster in the workload domain uses a single vSphere
Distributed Switch with two physical network cards whose design includes traffic types on
the switch, the number of required NICs, and MTU configuration.

n Distributed Port Group and VMkernel Adapter Design for a Virtual Infrastructure Workload
Domain
A distributed port group specifies port configuration options for each member port on a
vSphere distributed switch. Distributed port groups define how a connection is made to a
network.

n vMotion TCP/IP Stack Design for a Virtual Infrastructure Workload Domain

Use the vMotion TCP/IP stack to isolate traffic for vSphere vMotion and to assign a dedicated
default gateway for vSphere vMotion traffic.

n vSphere Network I/O Control Design for a Virtual Infrastructure Workload Domain
Use vSphere Network I/O Control to allocate network bandwidth to management applications
and to resolve situations where several types of traffic compete for common resources.

Virtual Switch Type Design for a Virtual Infrastructure Workload Domain

Virtual switches simplify the configuration process by providing a single pane of glass for
performing virtual network management tasks.

vSphere supports two types of virtual switches:

n vSphere Standard Switch

n vSphere Distributed Switch

A distributed switch offers several enhancements over a standard switch such as centralized
control plane and support for traffic monitoring features.

VMware, Inc. 55
Architecture and Design for a Virtual Infrastructure Workload Domain

Centralized management Because distributed switches are created and managed

centrally on a vCenter Server system, switch configuration
is more consistent across ESXi hosts. Centralized
management saves time, reduces mistakes, and reduces
operational costs.

Additional features Distributed switches have features that are not available on
standard virtual switches.
NetFlow and port mirroring provide monitoring and
troubleshooting capabilities to the virtual infrastructure.
To guarantee that traffic types with high priority have
enough network capacity, you can assign shares to these
traffic types by using Network I/O Control.
To ensure that every network adapter is used when the
network traffic is high, you can use the Route Based on
Physical NIC Load teaming policy. The distributed switch
directs the traffic from one physical network adapter to
another if the usage of an adapter remains at or above
75% for 30 seconds.

Disadvantages If vCenter Server is unavailable, distributed switches

are not manageable. Consider vCenter Server a Tier 1
application.

Table 2-49. Design Decisions on Virtual Switch Type

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-001 Use vSphere Distributed Simplifies management. Migration from a standard

Switches. switch to a distributed
switch requires a minimum
of two physical NICs to
maintain redundancy.

SDDC-WLD-VI-NET-002 Use a single vSphere n Reduces the complexity Increases the number
Distributed Switch per of the network design. of vSphere Distributed
vSphere Cluster. n Reduces the size of the Switches that must be
fault domain. managed.

vSphere Distributed Switch Design for a Virtual Infrastructure Workload Domain

The shared edge and workload cluster in the workload domain uses a single vSphere Distributed
Switch with two physical network cards whose design includes traffic types on the switch, the
number of required NICs, and MTU configuration.

VMware Validated Design also supports up to three distributed switches and up to six physical
network cards per host.

Table 2-50. sfo-w01-cl01-vds01 vSphere Distributed Switch Configuration

Number of Physical NIC Ports Network I/O Control MTU Size

2 Enabled 9,000 bytes

VMware, Inc. 56
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-51. Physical Uplinks on sfo-w01-cl01-vds01 vSphere Distributed Switch

Physical NIC Function

vmnic0 Uplink

vmnic1 Uplink

Table 2-52. Design Decisions for vSphere Distributed Switch

Design ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-003 Configure the MTU size n Supports the MTU size When adjusting the MTU
of the vSphere Distributed required by system packet size, you must
Switch to 9,000 for jumbo traffic types. also configure the entire
frames. n Improves traffic network path (VMkernel
throughput. ports, virtual switches,
physical switches, and
routers) to support the
same MTU packet size.

Distributed Port Group and VMkernel Adapter Design for a Virtual Infrastructure
Workload Domain
A distributed port group specifies port configuration options for each member port on a vSphere
distributed switch. Distributed port groups define how a connection is made to a network.

vSphere Distributed Switch introduces two abstractions that you use to create consistent
networking configuration for physical NICs, virtual machines, and VMkernel traffic.

Uplink port group

An uplink port group or dvuplink port group is defined during the creation of the distributed
switch and can have one or more uplinks. An uplink is a template that you use to configure
physical connections of hosts as well as failover and load balancing policies. You map physical
NICs of hosts to uplinks on the distributed switch. You set failover and load balancing policies
over uplinks and the policies are automatically propagated to the host proxy switches, or the
data plane.

Distributed port group

Distributed port groups provide network connectivity to virtual machines and accommodate
VMkernel traffic. You identify each distributed port group by using a network label, which
must be unique to the current data center. You configure NIC teaming, failover, load
balancing, VLAN, security, traffic shaping, and other policies on distributed port groups. As
with uplink port groups, the configuration that you set on distributed port groups on vCenter
Server (the management plane) is automatically propagated to all hosts on the distributed
switch through their host proxy switches (the data plane).

VMware, Inc. 57
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-53. Distributed Port Group Configuration

Parameter Setting

Failover detection Link status only

Notify switches Yes

Failback Yes

Failover order Active uplinks: Uplink1, Uplink2

Figure 2-10. vSphere Distributed Switch Design for a Workload Domain

Sample ESXi Host

for a Workload Domain

nic0 nic1

sfo01-w01-cl01-vds01

VLAN Management

VLAN vMotion

VLAN vSAN

VLAN NFS

VLAN Host Overlay

VLAN Trunk Edge Overlay

VLAN Trunk Uplink 01

VLAN Trunk Uplink 02

Table 2-54. Port Group Binding and Teaming

vSphere
Distributed Port Group Port Teaming Active Failover Notify
Switch Name Binding Policy Uplinks Detection Switches Failback

sfo01-w01- sfo-w01- Static Port Route based 1, 2 Link status Yes Yes
cl01-vds01 cl01-vds01- Binding on physical only
pg-mgmt NIC load

sfo01-w01- sfo-w01- Static Port Route based 1, 2 Link status Yes Yes
cl01-vds01 cl01-vds01- Binding on physical only
pg-vmotion NIC load

VMware, Inc. 58
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-54. Port Group Binding and Teaming (continued)

vSphere
Distributed Port Group Port Teaming Active Failover Notify
Switch Name Binding Policy Uplinks Detection Switches Failback

sfo01-w01- sfo-w01- Static Port Route based 1, 2 Link status Yes Yes
cl01-vds01 cl01-vds01- Binding on physical only
pg-vsan NIC load

sfo01-w01- Host See Overlay Design for NSX-T Data Center for a Virtual Infrastructure Workload Domain.
cl01-vds01 Overlay

sfo01-w01- sfo-w01-
cl01-vds01 cl01-vds01-
pg-uplink01

sfo01-w01- sfo-w01-
cl01-vds01 cl01-vds01-
pg-uplink02

NIC Teaming
For a predictable level of performance, use multiple network adapters in one of the following
configurations.

n An active-passive configuration that uses explicit failover when connected to two separate
switches.

n An active-active configuration in which two or more physical NICs in the server are assigned
the active role.

This design uses an active-active configuration.

VMware, Inc. 59
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-55. Design Decisions on Distributed Port Groups

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-004 Use static port binding Static binding ensures a None

for all port groups in the virtual machine connects
shared edge and workload to the same port on
cluster. the vSphere Distributed
Switch. This allows for
historical data and port
level monitoring.
Since the vCenter Server
managing the workload
domain resides in the
management domain, there
is no need for an ephemeral
port group for vCenter
Server recoverability.

SDDC-WLD-VI-NET-005 Use the Route based Reduces the complexity of None

on physical NIC load the network design and
teaming algorithm for the increases resiliency and
Management Port Group. performance.

SDDC-WLD-VI-NET-006 Use the Route based on Reduces the complexity of None

physical NIC load teaming the network design and
algorithm for the vMotion increases resiliency and
Port Group. performance.

VMkernel Network Adapter Configuration

The VMkernel networking layer provides connectivity to hosts and handles the system traffic for
vSphere vMotion, IP storage, vSphere HA, vSAN, and others.

You can also create VMkernel network adapters on the source and target vSphere Replication
hosts to isolate the replication data traffic.

Table 2-56. VMkernel Adapters for the Workload Domain

vSphere
Distributed Availability Connected Port
Switch Zones Network Label Group Enabled Services MTU

sfo-w01-cl01- Availability Zone 1 Management sfo01-w01-cl01- Management 1500 (Default)

vds01 vds01-pg-mgmt Traffic

sfo-w01-cl01- vMotion sfo01-w01- vMotion Traffic 9000

vds01 cl01-vds01-pg-
vmotion

sfo-w01-cl01- vSAN sfo01-w01-cl01- vSAN 9000

vds01 vds01-pg-vsan

sfo-w01-cl01- Availability Zone Management az2_sfo01-w01- Management 1500 (Default)

vds01 2 cl01-vds01-pg- Traffic
mgmt

VMware, Inc. 60
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-56. VMkernel Adapters for the Workload Domain (continued)

vSphere
Distributed Availability Connected Port
Switch Zones Network Label Group Enabled Services MTU

sfo-w01-cl01- vMotion az2_sfo01-w01- vMotion Traffic 9000

vds01 cl01-vds01-pg-
vmotion

sfo-w01-cl01- vSAN az2_sfo01-w01- vSAN 9000

vds01 cl01-vds01-pg-
vsan

vMotion TCP/IP Stack Design for a Virtual Infrastructure Workload Domain

Use the vMotion TCP/IP stack to isolate traffic for vSphere vMotion and to assign a dedicated
default gateway for vSphere vMotion traffic.

By using a separate TCP/IP stack, you can manage vSphere vMotion and cold migration traffic
according to the topology of the network, and as required by your organization.

n Route the traffic for the migration of virtual machines that are powered on or powered off by
using a default gateway that is different from the gateway assigned to the default stack on the
ESXi host.

n Assign a separate set of buffers and sockets.

n Avoid routing table conflicts that might otherwise appear when many features are using a
common TCP/IP stack.

n Isolate traffic to improve security.

Table 2-57. Design Decisions on the vMotion TCP/IP Stack

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-007 Use the vMotion TCP/IP By using the vMotion In the vSphere Client, the
stack for vSphere vMotion TCP/IP stack, vSphere vMotion TCP/IP stack is
traffic. vMotion traffic can be not available in the wizard
assigned a default gateway for creating a VMkernel
on its own subnet and can network adapter wizard at
go over Layer 3 networks. the distributed port group
level. You must create the
VMkernel adapter directly
on the ESXi host.

vSphere Network I/O Control Design for a Virtual Infrastructure Workload

Domain
Use vSphere Network I/O Control to allocate network bandwidth to management applications and
to resolve situations where several types of traffic compete for common resources.

VMware, Inc. 61
Architecture and Design for a Virtual Infrastructure Workload Domain

When Network I/O Control is enabled, the distributed switch allocates bandwidth for the traffic
that is related to the main vSphere features.

n Fault tolerance traffic

n iSCSI traffic

n vSphere vMotion traffic

n Management traffic

n VMware vSphere Replication traffic

n NFS traffic

n vSAN traffic

n Backup traffic

n Virtual machine traffic

Network I/O Control Heuristics

The following heuristics can help with design decisions for Network I/O Control.

Shares and Limits

When you use bandwidth allocation, consider using shares instead of limits. Limits impose
hard limits on the amount of bandwidth used by a traffic flow even when network bandwidth is
available.

Limits on Network Resource Pools

Consider imposing limits on a given network resource pool. For example, if you put a limit
on vSphere vMotion traffic, you can benefit in situations where multiple vSphere vMotion data
transfers, initiated on different ESXi hosts at the same time, result in oversubscription at the
physical network level. By limiting the available bandwidth for vSphere vMotion at the ESXi
host level, you can prevent performance degradation for other traffic.

Teaming Policy

When you use Network I/O Control, use Route based on physical NIC load teaming as a
distributed switch teaming policy to maximize the networking capacity utilization. With load-
based teaming, traffic might move among uplinks, and reordering of packets at the receiver
can result occasionally.

Traffic Shaping

Use distributed port groups to apply configuration policies to different traffic types. Traffic
shaping can help in situations where multiple vSphere vMotion migrations initiated on different
ESXi hosts converge on the same destination ESXi host. The actual limit and reservation also
depend on the traffic shaping policy for the distributed port group where the adapter is
connected to.

VMware, Inc. 62
Architecture and Design for a Virtual Infrastructure Workload Domain

How Network I/O Control Works

Network I/O Control enforces the share value specified for the different traffic types when a
network contention occurs. Network I/O Control applies the share values set to each traffic type.
As a result, less important traffic, as defined by the share percentage, is throttled, granting access
to more network resources to more important traffic types.

Network I/O Control also supports reservation of bandwidth for system traffic based on the
capacity of the physical adapters on an ESXi host and enables fine-grained resource control at
the virtual machine network adapter level. Resource control is similar to the model for CPU and
memory reservations in vSphere DRS.

Table 2-58. Design Decisions on vSphere Network I/O Control

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-008 Enable Network I/O Control Increases resiliency and If configured incorrectly,
on the vSphere Distributed performance of the Network I/O Control
Switch for the workload network. might impact network
domain. performance for critical
traffic types.

SDDC-WLD-VI-NET-009 Set the share value for By keeping the default None.
management traffic to setting of Normal,
Normal. management traffic is
prioritized higher than
vSphere vMotion and
vSphere Replication but
lower than vSAN traffic.
Management traffic is
important because it
ensures that the hosts
can still be managed
during times of network
contention.

SDDC-WLD-VI-NET-010 Set the share value for During times of network During times of network
vSphere vMotion traffic to contention, vSphere contention, vMotion takes
Low. vMotion traffic is not longer than usual to
as important as virtual complete.
machine or storage traffic.

SDDC-WLD-VI-NET-011 Set the share value for Virtual machines are the None
virtual machines to High. most important asset in the
SDDC. Leaving the default
setting of High ensures that
they always have access to
the network resources they
need.

SDDC-WLD-VI-NET-012 Set the share value for This design does not use None
vSphere Fault Tolerance to vSphere Fault Tolerance.
Low. Fault tolerance traffic can
be set the lowest priority.

VMware, Inc. 63
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-58. Design Decisions on vSphere Network I/O Control (continued)

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-NET-013 Set the share value for During times of None

vSAN to High. network contention,
vSAN traffic needs a
guaranteed bandwidth to
support virtual machine
performance.

SDDC-WLD-VI-NET-014 Set the share value for NFS Because NFS is used for During times of network
traffic to Low (25). secondary storage, such as contention, backups are
backups and vRealize Log slower than usual.
Insight archives, its priority
is lower than the priority of
the vSAN traffic.

SDDC-WLD-VI-NET-015 Set the share value for During times of network During times of network
backup traffic to Low. contention, the primary contention, backups are
functions of the SDDC must slower than usual.
continue to have access
to network resources with
priority over backup traffic.

SDDC-WLD-VI-NET-016 Set the share value for iSCSI This design does not use None
traffic to Low iSCSI. iSCSI traffic can be
set the lowest priority.

SDDC-WLD-VI-NET-017 Set the share value for During times of network During times of network
vSphere Replication traffic contention, vSphere contention, vSphere
to Low (25). Replication traffic is not Replication takes longer
as important as virtual and might violate the
machine or storage traffic. defined SLA.

Software-Defined Networking Design for a Virtual Infrastructure

Workload Domain
In this design, you use NSX-T Data Center to provide network connectivity for tenant workloads
by using virtual network segments and routing. You also create constructs for region-specific
and cross-region solutions. These constructs isolate the solutions from the rest of the network,
providing routing to the data center and load balancing.

NSX-T Data Center

NSX-T Data Center provides network virtualization capabilities in workload domains. With network
virtualization, networking components that are usually part of the physical infrastructure, can be
programmatically created and managed by using this software-defined network (SDN) platform.
NSX-T Data Center provides both a declarative intent-based policy model, and an imperative
based model to define and manage the SDN.

The deployment of NSX-T Data Center includes management, control plane, and services
components.

VMware, Inc. 64
Architecture and Design for a Virtual Infrastructure Workload Domain

NSX-T Federation
You use NSX-T Federation to propagate configurations that span multiple NSX-T instances. You
can stretch overlay segments in NSX-T between regions, enable failover of segment ingress and
egress traffic between regions, and implement a unified firewall configuration.

In the virtual infrastructure domain in a multi-region SDDC, you can use NSX-T to provide
cross-region services to customer workloads which do not have native support for multi-region
availability.

Note If you do not plan to use any workloads which require NSX-T Federation for multi-region
availability, consider this design extension optional.

NSX-T Manager
NSX-T Manager provides the user interface and the RESTful API for creating, configuring, and
monitoring NSX-T components, such as virtual network segments, and Tier-0 and Tier-1 gateways.

NSX-T Manager implements the management and control plane for the NSX-T infrastructure.
NSX-T Manager is the centralized network management component of NSX-T, providing an
aggregated view on all components in the NSX-T Data Center system.

In a deployment using NSX-T Federation, such as a multi-region SDDC, NSX-T Manager is called
NSX-T Local Manager.

Table 2-59. Components of NSX-T Manager

Component Description

Services n Logical switching and routing

n Networking and edge services
n Security services and distributed firewall

RESTful API You can automate all configuration and monitoring operations by using any cloud automation
platform, security vendor platform, or automation framework.

Management Plane Available on each ESXi host. The MPA is in charge of persisting the desired state of the system
Agent (MPA) and for communicating non-flow-controlling (NFC) messages such as configuration, statistics,
status, and real-time data between transport nodes and the management plane.

NSX-T Controller NSX-T Controllers implement the central control plane (CCP). They control the virtual
networks and overlay transport tunnels. The controllers are responsible for the programmatic
deployment of virtual networks across the entire NSX-T architecture.
The CCP is logically separated from all data plane traffic, that is, a failure in the control plane
does not affect existing data plane operations. The controller provides configuration to other
NSX-T Data Center components, such as segment, gateway, and edge node configuration.

Integration with NSX-T Data Center components are not assigned to a specific vCenter Server or vSphere
vCenter Server construct. You can share them across different vSphere environments.

VMware, Inc. 65
Architecture and Design for a Virtual Infrastructure Workload Domain

NSX-T Global Manager

NSX-T Global Manager is part of multi-region deployments where NSX-T Federation is required.
NSX-T Global Manager can connect multiple NSX-T Manager instances under a single global
management plane. NSX-T Global Manager provides the user interface and the RESTful API
for creating, configuring, and monitoring NSX-T global objects, such as global virtual network
segments, and global Tier-0 and Tier-1 gateways.

Connected NSX-T Local Manager instances create the global objects on the underlying software-
defined network that you define from NSX-T Global Manager. An NSX-T Local Manager instance
in an individual region directly communicates with NSX-T Local Manager instances in other regions
to synchronize configuration and state needed to implement a global policy.

NSX-T Global Manager is a deployment-time role that is assigned to an NSX-T Manager appliance.

NSX-T Edge Nodes

An NSX-T Edge node is a special type of transport node which contains service router
components.

NSX-T Edge nodes provide north-south traffic connectivity between the physical data center
networks and the NSX-T SDN networks. Each NSX-T Edge node has multiple interfaces where
traffic flows.

You also use the NSX-T Edge nodes in east-west traffic flow between virtualized workloads.
They provide stateful services such as load balancers and DHCP. In a multi-region deployment,
east-west traffic between the regions flows through the NSX-T Edge nodes, as well.

n Logical Design for NSX-T Data Center for a Virtual Infrastructure Workload Domain
NSX-T Data Center provides networking services to tenant workloads such as load balancing,
routing, and virtual networking. NSX-T Data Center is connected to the region-specific
Workspace ONE Access for centralized user management.

n Physical Network Infrastructure Design for a Virtual Infrastructure Workload Domain

Design of the physical data center network includes defining the network topology for
connecting the physical switches and the ESXi hosts, determining switch port settings for
VLANs and link aggregation, and designing routing.

n NSX-T Manager Deployment Specification and Network Design for a Virtual Infrastructure
Workload Domain
You determine the size of the compute resources, high availability implementation, and
patching and upgrade support for the NSX-T Manager instance for the workload domain
according to the design objectives and aggregated requirements of the tenant workloads of
the SDDC.

VMware, Inc. 66
Architecture and Design for a Virtual Infrastructure Workload Domain

n NSX-T Global Manager Deployment Specification and Network Design for a Virtual
Infrastructure Workload Domain
To implement a multi-region SDDC, you must use NSX-T Federation that requires the
deployment of NSX-T Global Manager nodes in the first two regions of the SDDC. You
determine the size of the compute resources, high availability implementation, and patching
and upgrade support for the NSX-T Global Manager instance for the management domain
according to the design objectives and aggregated requirements of the management
components of the SDDC.

n NSX-T Edge Deployment Specification and Network Design for a Virtual Infrastructure
Workload Domain
For traffic segmentation, you place NSX-T Manager for the workload domain on the
management VLAN in the management domain and decide on the IP addressing scheme
and name resolution for optimal support for the NSX-T Edge nodes.

n Life Cycle Management Design of NSX-T Data Center for a Virtual Infrastructure Workload
Domain
You decide on the life cycle management of the NSX-T Data Center components according
to the amount of time and effort to perform a deployment, upgrade, or patch operation. You
also consider the impact such an operation has on the solutions that are connected to NSX-T
Data Center for the workload domain.

n NSX-T Services Design for a Virtual Infrastructure Workload Domain

NSX-T Edge clusters are pools of capacity for NSX-T service router and load balancing
functions.

n Overlay Design for NSX-T Data Center for a Virtual Infrastructure Workload Domain
As part of the overlay design, you determine the NSX-T Data Center configuration for
handling traffic between tenant workloads. You determine the configuration of vSphere
Distributed Switch and virtual segments on it, and of the transport zones.

n Virtual Network Segment Design for NSX-T for a Virtual Infrastructure Workload Domain
Applications that are deployed on top of the workload domain can use a pre-defined
configuration of NSX-T virtual network segments.

n Information Security and Access Design for NSX-T for a Virtual Infrastructure Workload
Domain
You design authentication access, controls, and certificate management for the NSX-T
Data Center instance in the workload domain according to industry standards and the
requirements of your organization.

Logical Design for NSX-T Data Center for a Virtual Infrastructure Workload
Domain
NSX-T Data Center provides networking services to tenant workloads such as load balancing,
routing, and virtual networking. NSX-T Data Center is connected to the region-specific Workspace
ONE Access for centralized user management.

VMware, Inc. 67
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-11. NSX-T Logical Design for the Workload Domain

Access Identity Management Supporting

NSX-T Edge Mnagement Domain
Cluster Infrastructure
vCenter Server
User Interface
Region-Specific
DNS NTP
Workspace ONE Access
API NSX-T Edge NSX-T Edge
Node 1 Node 2

NSX-T
Manager Cluster

Virtual Infrastructure Management

Internal VIP Load Balancer

Workload Domain
vCenter Server
NSX-T NSX-T NSX-T
Manager Manager Manager
1 2 3

NSX-T
Transport Nodes

Shared Edge and Workload Cluster

ESXi ESXi ESXi ESXi

VMware, Inc. 68
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-12. NSX-T Logical Design for Virtual Infrastructure Workload Domains for a Multi-
Region SDDC
Region A Region B

NSX-T Global NSX-T Global

Manager Cluster (Active) Manager Cluster (Standby)
Identity Management Identity Management

Internal VIP Load Balancer Internal VIP Load Balancer

Region-Specific Region-Specific
Workspace ONE Access Workspace ONE Access
Global Global Global Global Global Global
Manager Manager Manager Manager Manager Manager
1 2 3 1 2 3

NSX-T Local NSX-T Local

Manager Cluster Manager Cluster
Virtual Infrastructure Management Virtual Infrastructure Management

Internal VIP Load Balancer Internal VIP Load Balancer

Workload Domain Workload Domain
vCenter Server vCenter Server
Local Local Local Local Local Local
Manager Manager Manager Manager Manager Manager
1 2 3 1 2 3

NSX-T NSX-T
NSX-T Edge Transport Nodes Transport Nodes NSX-T Edge
Node Cluster Node Cluster

Shared Edge and Workload Cluster Shared Edge and Workload Cluster
Edge Edge Edge Edge
Node 1 Node 2 Node 1 Node 2

ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi

An NSX-T Data Center deployment consists of these components:

n Unified appliances that have both the NSX-T Local Manager and NSX-T Controller roles. They
provide management and control plane capabilities.

n NSX-T Edge nodes that provide advanced services such as load balancing and north-south
connectivity.

n The ESXi hosts in the workload domain are registered as NSX-T transport nodes to provide
distribute routing and firewall services to tenant workloads.

To support a multi-region SDDC architecture, you add the following components:

n NSX-T Global Manager cluster in each of the first two regions.

You deploy the NSX-T Global Manager cluster in each region so that you can use NSX-T
Federation for global management of networking and security services.

n An additional infrastructure VLAN in each region to carry region-to-region traffic.

VMware, Inc. 69
Architecture and Design for a Virtual Infrastructure Workload Domain

Component Single Region Multiple Availability Zones Multiple Regions

NSX-T n Three large-size NSX- n Three large-size NSX-T Local In Region A:

Manager T Local Manager Manager nodes in Availability n Three medium-size NSX-T
Cluster nodes with an internal Zone 1 with an internal virtual Global Manager nodes with an
virtual IP (VIP) IP (VIP) address for high internal VIP address for high
address for high availability availability.
availability n vSphere should-run DRS rule
The NSX-T Global Manager
n vSphere HA protects keeps the NSX-T Manager
cluster is
the NSX-T Manager nodes running in Availability
cluster nodes Zone 1. Failover to Availability set as Active
applying high restart Zone 2 occurs only if a failure In Region B:
priority in Availability Zone 1 occurs. n Three medium-size NSX-T
n vSphere DRS rule n In the availability zone, Global Manager nodes with an
keeps the NSX- vSphere HA protects the internal VIP address for high
T Manager nodes cluster nodes applying high availability.
running on different restart priority
The NSX-T Global Manager
hosts n In the availability zone,
cluster is
vSphere DRS rule keeps the
nodes running on different set as Standby
hosts In each region in an SDDC with two
or more regions:
n Three large-size NSX-T Local
Manager nodes with an internal
virtual IP (VIP) address for high
availability
n vSphere HA protects the nodes
of both the NSX-T Global
Manager and NSX-T Manager
clusters applying high restart
priority
n vSphere DRS rule keeps the
nodes of both the NSX-T Global
Manager and NSX-T Manager
clusters running on different
hosts

NSX-T Edge n Two large-size NSX-T n Two large-size NSX-T Edge In each regionin an SDDC with two
Cluster Edge nodes nodes in Availability Zone 1 or more regions:
n vSphere HA protects n vSphere should-run DRS rule n Two large-size NSX-T Edge
the NSX-T Edge keeps the NSX-T Edge nodes nodes
nodes applying high running in Availability Zone 1. n vSphere HA protects the NSX-
restart priority Failover to Availability Zone T Edge nodes applying high
n vSphere DRS rule 2 occurs only if a failure in restart priority
keeps the NSX-T Availability Zone 1 occurs. n vSphere DRS rule keeps the
Edge nodes running n In the availability zone, NSX-T Edge nodes running on
on different hosts vSphere HA protects the NSX- different hosts
T Edge nodes applying high
restart priority
n In the availability zone,
vSphere DRS rule keeps the
NSX-T Edge nodes running on
different hosts

VMware, Inc. 70
Architecture and Design for a Virtual Infrastructure Workload Domain

Component Single Region Multiple Availability Zones Multiple Regions

Transport n Four ESXi host n In each availability zone, four In each region in an SDDC with two
Nodes transport nodes ESXi host transport nodes or more regions:
n Two Edge transport n Two Edge transport nodes in n Four ESXi host transport nodes
nodes Availability Zone 1 n Two Edge transport nodes

Transport n One VLAN transport n One VLAN transport zone for In each region in an SDDC with two
Zones zone for NSX-T Edge NSX-T Edge uplink traffic or more regions:
uplink traffic n One overlay transport zone n One VLAN transport zone for
n One overlay for SDDC management NSX-T Edge uplink traffic
transport zone for components and NSX-T Edge n One overlay transport zone
SDDC management nodes for SDDC management
components and components and NSX-T Edge
NSX-T Edge nodes nodes

VLANs and IP n Management Networks for Availability Zone 1: In each region in an SDDC with two
Subnets n vSphere vMotion n Management (stretched) or more regions:

n vSAN n vSAN n Management

n Host Overlay n vSphere vMotion n vSphere vMotion

n NFS n Host Overlay n vSAN

n Uplink01 n NFS n Host Overlay

n Uplink02 n Uplink01 (stretched) n NFS

n Edge Overlay n Uplink02 (stretched) n Uplink01

n Edge Overlay (stretched) n Uplink02

Networks for Availability Zone 2: n Edge Overlay

n Management (stretched) n For a configuration with a single

availability zone in each region:
n vSAN
n vSphere vMotion Edge RTEP
n Host Overlay n For a configuration with multiple
availability zones in Region
A: Edge RTEP (stretched) in
Region A and Edge RTEP
(regular) in the other regions

Routing BGP BGP with Tier-0 gateway BGP

Configuration configuration for directing ingress
and egress traffic to Availability
Zone 1 unless a failure in the zone
occurs.

VMware, Inc. 71
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-13. NSX-T Data Center Components

Management
Workload Domain

Management
vCenter Server

Management
NSX-T Managers Workload Domain
Shared Edge and
Workload Workload Cluster
vCenter Server

Workload NSX-T Edge 1

NSX-T Managers

NSX-T Edge 2
NSX-T Edge 1
Active/Active HA Pair
NSX-T Edge 2

Active/Active HA Pair

Physical Network Infrastructure Design for a Virtual Infrastructure Workload

Domain
Design of the physical data center network includes defining the network topology for connecting
the physical switches and the ESXi hosts, determining switch port settings for VLANs and link
aggregation, and designing routing.

A software-defined network (SDN) both integrates with and uses components of the physical data
center. SDN integrates with your physical network to support east-west transit in the data center
and north-south transit to and from the SDDC networks.

VMware Validated Design uses the leaf-spine networking topology, because in a single data
center deployment, it provides predictable performance, scalable nature, and applicability across
multiple vendors.

In an environment with multiple availability zones, Layer 2 networks must be stretched between
the availability zones by the physical infrastructure. You also must provide a Layer 3 gateway
that is highly available between regions. The method for stretching these Layer 2 networks and
providing a highly available Layer 3 gateway is vendor-specific.

In an environment with multiple availability zones or regions, dynamic routing is needed to

provide networks with the ability to fail ingress and egress of traffic over from availability zone
to availability zone, or region to region. This design uses BGP as the dynamic routing protocol. As
such, BGP must be present in the customer environment to facilitate the failover of networks from
site to site. Because of the complexity of local-ingress, local-egress is not generally in use. In this
design, network traffic flows in and out of a primary site or region.

VMware, Inc. 72
Architecture and Design for a Virtual Infrastructure Workload Domain

Switch Types and Network Connectivity

Follow the best practices for physical switches, switch connectivity, VLANs and subnets, and
access port settings.

Figure 2-14. Host-to-ToR Connectivity

ToR ToR

25 GbE 25 GbE

ESXi
Host

Table 2-60. Design Components for Physical Switches in the SDDC

Design Component Configuration Best Practices

Top of rack (ToR) n Configure redundant physical switches to enhance availability.

physical switches n Configure switch ports that connect to ESXi hosts manually as trunk ports.
n Modify the Spanning Tree Protocol (STP) on any port that is connected to an ESXi NIC
to reduce the time to transition ports over to the forwarding state, for example using the
Trunk PortFast feature found in a Cisco physical switch.
n Provide DHCP or DHCP Helper capabilities on all VLANs used by TEP VMkernel ports.
This setup simplifies the configuration by using DHCP to assign IP address based on the IP
subnet in use.
n Configure jumbo frames on all switch ports, inter-switch link (ISL), and switched virtual
interfaces (SVIs).

Top of rack Each ESXi host is connected redundantly to the top of rack switches SDDC network fabric by
connectivity and two 25 GbE ports. Configure the top of rack switches to provide all necessary VLANs using
network settings an 802.1Q trunk. These redundant connections use features in vSphere Distributed Switch and
NSX-T to guarantee that no physical interface is overrun, and available redundant paths are
used.

VLANs and Subnets for a Single Region and Single Availability Zone
Each ESXi host uses VLANs and corresponding subnets.

Follow these guidelines:

n Use only /24 subnets to reduce confusion and mistakes when handling IPv4 subnetting.

n Use the IP address of the floating interface for Virtual Router Redundancy Protocol (VRPP) or
Hot Standby Routing Protocol (HSRP) as the gateway.

VMware, Inc. 73
Architecture and Design for a Virtual Infrastructure Workload Domain

n Use the RFC1918 IPv4 address space for these subnets and allocate one octet by region and
another octet by function.

Note Implement VLAN and IP subnet configuration according to the requirements of your
organization.

Table 2-61. VLANs and IP Ranges in This Design

Function VLAN ID IP Range

Management 1631 172.16.31.0/24

vSphere vMotion 1632 172.16.32.0/24

vSAN 1633 172.16.33.0/24

Host Overlay 1634 172.16.34.0/24

NFS 1635 172.16.35.0/24

Uplink01 2731 172.27.31.0/24

Uplink02 2732 172.27.32.0/24

Edge Overlay 2733 172.27.33.0/24

VLANs and Subnets for Multiple Available Zones

In an environment with multiple availability zones, you can apply this configuration. The
management, Uplink 01, Uplink 02, and Edge Overlay networks in each availability zone must
be stretched to facilitate failover of the NSX-T Edge appliances between availability zones. The
Layer 3 gateway for the management and Edge Overlay networks must be highly available across
the availability zones.

Availability Zone Availability Zone HA Layer 3

Function 1 2 VLAN ID IP Range Gateway

Management ✓ ✓ 1631 (Stretched) 172.16.31.0/24 ✓

vSphere vMotion ✓ X 1632 172.16.32.0/24 ✓

vSAN ✓ X 1633 172.16.33.0/24 ✓

Host Overlay ✓ X 1634 172.16.34.0/24 ✓

Uplink01 ✓ ✓ 2731 (Stretched) 172.27.31.0/24 X

Uplink02 ✓ ✓ 2732 172.27.32.0/24 X

(Stretched)

Edge Overlay ✓ ✓ 2733 172.27.33.0/24 ✓

(Stretched)

Management X ✓ 1641 172.16.41.0/24 ✓

vSphere vMotion X ✓ 1642 172.16.42.0/24 ✓

VMware, Inc. 74
Architecture and Design for a Virtual Infrastructure Workload Domain

Availability Zone Availability Zone HA Layer 3

Function 1 2 VLAN ID IP Range Gateway

vSAN X ✓ 1643 172.16.43.0/24 ✓

Host Overlay X ✓ 1644 172.16.44.0/24 ✓

VLANs and Subnets for Multiple Regions

In a deployment with multiple regions, add VLANs for remote tunnel end point (RTEP) traffic for
the NSX-T Edge nodes in each region. Edge RTEP VLANs carry region-to-region dataplane traffic.
An Edge RTEP VLAN must be routed to the Edge RTEP VLANs in all other regions.

Availability Zone Availability Zone HA Layer 3

Function 1 2 VLAN ID IP Range Gateway

Edge RTEP - ✓ ✓ 2734 172.27.34.0/24 ✓

Region A

Edge RTEP - ✓ X 2834 172.28.34.0/24 ✓

Region B

Physical Network Requirements

Physical requirements determine the MTU size for networks that carry overlay traffic, dynamic
routing support, time synchronization through an NTP server, and forward and reverse DNS
resolution.

Requirement Comment

Use 25 GbE (10 GbE minimum) port on each ToR switch for 25 GbE provides appropriate bandwidth for
ESXi host uplinks. Connect each host to two ToR switches. hyperconverged networking traffic. Connection to two ToR
switches provides redundant physical network paths to
each host.

Provide an MTU size of 1700 bytes or greater for host Geneve packets cannot be fragmented. The MTU size must
overlay traffic. be large enough to support extra encapsulation overhead.
Geneve is an extensible protocol, therefore the MTU might
increase with future capabilities. While 1600 is sufficient,
an MTU size of 1700 bytes provides more room for
increasing the Geneve MTU without the need to change
physical infrastructure MTU.
This design uses an MTU size of 9000 bytes for Geneve
traffic.

VMware, Inc. 75
Architecture and Design for a Virtual Infrastructure Workload Domain

Requirement Comment

Enable BGP dynamic routing support on the upstream You use BGP on the upstream Layer 3 devices to establish
Layer 3 devices. routing adjacency with the Tier-0 service routers (SRs).
NSX-T supports only the BGP routing protocol.
Dynamic routing enables ECMP failover for upstream
connectivity.
Consider the following requirements for the BGP
configuration:
n Configure each ToR switch with only one uplink VLAN.
The first ToR switch to which vmnic0 of each ESXi
host is connected must be configured only with the
Uplink VLAN 1 gateway. The second ToR switch to
which vmnic1 of each ESXi host is connected must be
configured only with the Uplink VLAN 2 gateway.
n Make sure that BGP default-originate or a similar
feature is enabled on the ToR switches to inject default
route into BGP routes exchange with the NSX-T Tier-0
gateway.

BGP Autonomous System Number (ASN) allocation A BGP ASN must be allocated for the NSX-T SDN. Use a
private ASN according to RFC1930.

Physical Network Design Decisions

The physical network design decisions determine the physical layout and use of VLANs. They also
include decisions on jumbo frames and on other network-related requirements such as DNS and
NTP.

Table 2-62. Design Decisions on the Physical Network Infrastructure for NSX-T Data Center in a
Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-001 Use two ToR switches for Supports the use of Requires two ToR switches
each rack. two 10 GbE (25 GbE per rack which can increase
or greater recommended) costs.
links to each server and
provides redundancy and
reduces the overall design
complexity.

VCF-WLD-VI-SDN-002 Implement the following n Guarantees availability n Might limit the

physical network during a switch failure. hardware choices.
architecture: n Provides support n Requires dynamic
n One 25 GbE (10 GbE for BGP as the routing protocol
minimum) port on each only dynamic routing configuration in the
ToR switch for ESXi host protocol that is physical network
uplinks. supported by NSX-T
n Layer 3 device that Data Center.
supports BGP.

VMware, Inc. 76
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-62. Design Decisions on the Physical Network Infrastructure for NSX-T Data Center in a
Workload Domain (continued)

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-003 Do not use EtherChannel n Simplifies configuration None.

(LAG, LACP, or vPC) of top of rack switches.
configuration for ESXi host n Teaming options
uplinks available with vSphere
Distributed Switch and
N-VDS provide load
balancing and failover.
n EtherChannel
implementations might
have vendor-specific
limitations.

VCF-WLD-VI-SDN-004 Use a physical network n Supports flexibility Requires BGP configuration

that is configured for BGP in network design in the physical network.
routing adjacency for routing multi-site
and multi-tenancy
workloads.
n Uses BGP as the
only dynamic routing
protocol that is
supported by NSX-T.
n Supports failover
between ECMP Edge
uplinks.

Access Port Network Settings

Configure additional network settings on the access ports that connect the ToR switches to the
corresponding servers.

Table 2-63. Access Port Network Configuration

Setting Value

Spanning Tree Protocol (STP) Although this design does not use the Spanning Tree
Protocol, switches usually include STP configured by
default. Designate the access ports as trunk PortFast.

Trunking Configure the VLANs as members of an 802.1Q trunk.

Optionally, the management VLAN can act as the native
VLAN.

MTU n Set MTU for management VLANs and SVIs to 1500

bytes.
n Set MTU for vMotion, vSAN, NFS, Uplinks, Host
Overlay, and Edge Overlay VLANs and SVIs to 9000
bytes.

DHCP Helper Configure a DHCP helper (sometimes called a DHCP relay)

on all Overlay VLANs. Set the DHCP helper (relay) to point
to a DHCP server by IPv4 address.

VMware, Inc. 77
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-64. Design Decisions on Access Ports for NSX-T Data Center in a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-005 Assign persistent IP Ensures that endpoints Requires precise IP address

configurations to each have a persistent management.
management component management IP address. In
in the SDDC with the VMware Cloud Foundation,
exception for NSX-T tunnel you assign storage (vSAN
endpoints (TEPs) that use and NFS) and vSphere
dynamic IP allocation. vMotion IP configurations
by using user-defined
network pools.

VCF-WLD-VI-SDN-006 Set the lease duration for NSX-T Host Overlay Requires configuration and
the NSX-T Host Overlay VMkernel port IP addresses management of a DHCP
network DHCP scope to at are assigned by using a server.
least 7 days. DHCP server.
n NSX-T Host Overlay
VMkernel ports do not
have an administrative
endpoint. As a result,
they can use DHCP for
automatic IP address
assignment. IP pools
are an option, but
the NSX-T administrator
must create them. If you
must change or expand
the subnet, changing
the DHCP scope is
simpler than creating an
IP pool and assigning it
to the ESXi hosts.
n DHCP simplifies the
configuration of default
gateway for Host
Overlay VMkernel ports
if hosts within same
cluster are on separate
Layer 2 domains.

VCF-WLD-VI-SDN-007 Use VLANs to separate n Supports physical Requires uniform

physical network functions. network connectivity configuration and
without requiring many presentation on all the
NICs. trunks that are made
n Isolates the different available to the ESXi hosts.
network functions of
the SDDC so that you
can have differentiated
services and prioritized
traffic as needed.

VMware, Inc. 78
Architecture and Design for a Virtual Infrastructure Workload Domain

Jumbo Frames
IP storage throughput can benefit from the configuration of jumbo frames. Increasing the per-
frame payload from 1500 bytes to the jumbo frame setting improves the efficiency of data transfer.
You must configure jumbo frames end-to-end. Select an MTU that matches the MTU of the
physical switch ports.

n According to the purpose of the workload, determine whether to configure jumbo frames
on a virtual machine. If the workload consistently transfers large amounts of network data,
configure jumbo frames, if possible. In that case, confirm that both the virtual machine
operating system and the virtual machine NICs support jumbo frames.

n Using jumbo frames also improves the performance of vSphere vMotion.

n The Geneve overlay requires an MTU value of 1700 bytes or greater.

Table 2-65. Design Decisions on the Jumbo Frames for NSX-T Data Center for the Workload
Domain

Decision ID Design Decision Decision Justification Decision Implication

VCF-WLD-VI-SDN-008 Set the MTU size to at least n Improves traffic When adjusting the MTU
1700 bytes (recommended throughput. size, you must also
9000 bytes for jumbo n Supports Geneve by configure the entire
frames) on the physical increasing the MTU size network path (VMkernel
switch ports, vSphere to a minimum of 1600 ports, virtual switches,
Distributed Switches, bytes. physical switches, and
vSphere Distributed Switch routers) to support the
n Geneve is an extensible
port groups, and N-VDS same MTU size.
protocol. The MTU
switches that support the size might increase
following traffic types: with future capabilities.
n Geneve (overlay) While 1600 is sufficient,
n vSAN an MTU size of 1700
n vSphere vMotion bytes provides more
room for increasing
n NFS
the Geneve MTU size
without the need
to change the MTU
size of the physical
infrastructure.

Networking for Multiple Availability Zones

Specific requirements for the physical data center network exist for a topology with multiple
availability zones. These requirements extend those for an environment with a single availability
zone.

VMware, Inc. 79
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-66. Physical Network Requirements for Multiple Availability Zones

Component Requirement

MTU n VLANs which are stretched between availability zones

must meet the same requirements as the VLANs for
intra-zone connection including MTU.
n MTU value must be consistent end-to-end including
components on the inter zone networking path.
n Set MTU for management VLANs and SVIs to 1500
bytes.
n Set MTU for vMotion, vSAN, NFS, Uplinks, Host
Overlay, and Edge Overlay VLANs and SVIs to 9000
bytes.

Layer 3 gateway availability For VLANs that are stretched between available zones,
configure data center provided method, for example,
VRRP or HSRP, to failover the Layer 3 gateway between
availability zones.

DHCP availability For VLANs that are stretched between availability zones,
provide high availability for the DHCP server so that a
failover operation of a single availability zone will not
impact DHCP availability.

BGP routing Each availability zone data center can have its own
Autonomous System Number (ASN) or both availability
zones can be sharing same ASN.

Ingress and egress traffic n For VLANs that are stretched between availability
zones, traffic flows in and out of a single zone. Local
egress is not supported.
n For VLANs that are not stretched between availability
zones, traffic flows in and out of the zone where the
VLAN is located.
n For NSX-T virtual network segments that are stretched
between regions, traffic flows in and out of a single
availability zone. Local egress is not supported.

Latency n Maximum network latency between NSX-T Managers is

10 ms.
n Maximum network latency between the NSX-T
Manager cluster and transport nodes is 150 ms.

VMware, Inc. 80
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-67. Design Decisions on the Physical Network for Multiple Available Zones for NSX-T
Data Center in a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-009 Set the MTU size to at least n Improves traffic When adjusting the MTU
1700 bytes (recommended throughput. size, you must also
9000 bytes for jumbo n Geneve packets are configure the entire
frames) on physical inter- tagged as do not network path (VMkernel
availability zone networking fragment. ports, virtual switches,
components which are part physical switches, and
n For optional
of the networking path routers) to support the
performance, provides
between availability zones same MTU size.
a consistent MTU size
for the following traffic In multi-AZ deployments,
across the environment.
types. the MTU must be
n Geneve is an extensible
n Geneve (overlay) configured on the entire
protocol. The MTU
n vSAN network path between AZs.
size might increase
n vSphere vMotion with future capabilities.
n NFS While 1600 is sufficient,
an MTU size of 1700
bytes provides more
room for increasing
the Geneve MTU size
without the need
to change the MTU
size of the physical
infrastructure.

VCF-WLD-VI-SDN-010 Configure VRRP, HSRP, or Ensures that the VLANs Requires configuration of a
another Layer 3 gateway that are stretched between high availability technology
availability method. availability zones are for the Layer 3 gateways in
connected to a highly- the data center.
available gateway if a failure
of an availability zone
occurs. Otherwise, a failure
in the Layer 3 gateway will
cause disruption in traffic in
the SDN setup.

Networking for Multiple Regions

For a topology with multiple regions, specific requirements for the networking in a data center
and between data centers exist. These requirements extend those for an environment with a single
availability zone and those for multiple availability zones.

VMware, Inc. 81
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-68. Additional Requirements for the Physical Network for a Multiple-Region SDDC

Component Requirement

MTU n The Edge RTEP VLAN must have standard MTU or

greater over the entire end-to-end data path to the
RTEP VLAN in other regions.
n Set the MTU for the RTEP VLAN to 1,700 bytes or
greater for best performance.

BGP Routing n Each region must have its own ASN.

n Provide connectivity for BGP between the all data
centers.
n Deployments without BGP are not supported.

Ingress and egress traffic n For NSX-T virtual network segments that are not
stretched between regions, traffic flows in and out of
the zone where the segment is located.
n For NSX-T virtual network segments that are stretched
between regions, traffic flows in and out of a
single region or availability zone. Local-egress is not
supported. Failover to other regions occurs over BGP
route withdrawal or advertisement.

Latency Intra-Region
n Maximum network latency between NSX-T Managers
nodes within a NSX-T Manager cluster is 10 ms.
n Maximum network latency between the NSX-T
Manager cluster and transport nodes is 150 ms.
Inter-Region
n Maximum network latency between the primary and
standby NSX-T Global Manager clusters is 150 ms.
n Maximum network latency between NSX-T Local
Manager clusters is 150 ms.

Required connectivity between data centers n Edge Node RTEP interfaces: Region A and Region B
n NSX-T Local Manager clusters: Region A and Region B
n NSX-T Global Manager clusters: Region A and Region
B
n NSX-T Global Manager to NSX-T Local Manager
clusters: Region A to Region A and Region B and
Region B to Region A and Region B

VMware, Inc. 82
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-69. Design Decisions on the Physical Network Infrastructure between Data Centers for
NSX-T Data Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-011 For a dual-region SDDC, n Jumbo frames When adjusting the MTU
set the MTU size to are not required packet size, you must
at least 1,500 bytes between regions. also configure the entire
(1,700 bytes preferred, However, increased network path, that is,
9,000 bytes recommended MTU improves traffic virtual interfaces, virtual
for jumbo frames) on throughput. switches, physical switches,
the physical inter-region n Increasing the RTEP and routers to support the
network components which MTU to 1,700 same MTU packet size.
are part of the network path bytes minimizes
between availability zones fragmentation for
for the following traffic standard size workload
types. packets between
n Edge RTEP regions.

VCF-WLD-VI-SDN-012 For a dual-region SDDC, Configuring NSX-T Requires unique routable IP

provide a connection Federation requires addresses for each region.
between regions that is connectivity between NSX-
capable of routing between T Global Managers, NSX-T
each NSX-T Manager Local Managers, and NSX-T
cluster. Edge clusters.

VCF-WLD-VI-SDN-013 For a dual-region SDDC, A latency below 150 ms is None.

ensure that latency required for the following
between regions is less features.
than 150 ms n Cross vCenter Server
vMotion
n NSX-T SDDC design

VCF-WLD-VI-SDN-014 For a dual-region SDDC, Automated failover of None.

provide BGP routing is networks requires a
available between all dynamic routing protocol,
regions such as BGP.

NSX-T Manager Deployment Specification and Network Design for a Virtual

Infrastructure Workload Domain
You determine the size of the compute resources, high availability implementation, and patching
and upgrade support for the NSX-T Manager instance for the workload domain according to the
design objectives and aggregated requirements of the tenant workloads of the SDDC.

n Deployment Model for NSX-T Manager

As a best practice, you must deploy a highly available NSX-T Manager instance so that the
NSX-T central control place can continue propagating configuration to the transport nodes.
You also select an NSX-T Manager appliance size according to the number of ESXi hosts
required to run the SDDC tenant workloads.

VMware, Inc. 83
Architecture and Design for a Virtual Infrastructure Workload Domain

n Network Design for NSX-T Manager for a Virtual Infrastructure Workload Domain
For traffic segmentation, you place NSX-T Manager for the workload domain on the
management VLAN and decide on the IP addressing scheme and name resolution for optimal
support for the SDDC tenant workloads.

Deployment Model for NSX-T Manager

As a best practice, you must deploy a highly available NSX-T Manager instance so that the NSX-T
central control place can continue propagating configuration to the transport nodes. You also
select an NSX-T Manager appliance size according to the number of ESXi hosts required to run the
SDDC tenant workloads.
Deployment Type
Table 2-70. Design Decisions on NSX-T Manager Deployment Type for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-015 Deploy three NSX-T SDDC tenant workloads None.

Manager nodes for the can be placed on isolated
workload domain in the first virtual networks, using load
cluster in the management balancing, logical switching,
domain for configuring dynamic routing, and
and managing the network logical firewalls services.
services for SDDC tenant
workloads.

Sizing Compute and Storage Resources for NSX-T Manager

When you size the resources for NSX-T management components, consider the compute and
storage requirements for each component, and the number of nodes per component type.

Table 2-71. NSX-T Manager Resource Specification

Appliance Size vCPU Memory (GB) Storage (GB) Scale

Extra-Small 2 8 200 Cloud Service

Manager only

Small 4 16 200 Proof of concept

Medium 6 24 200 Up to 64 ESXi hosts

Large 12 48 200 More than 64 ESXi

hosts

Table 2-72. Design Decisions on Sizing Resources for NSX-T Manager for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- Deploy each node in the A large-size appliance is You must provide enough
SDN-016 NSX-T Manager cluster for sufficient for providing compute and storage resources
the workload domain as a network services to the SDDC in the management domain to
large-size appliance. tenant workloads. support this NSX-T Manager
cluster.

VMware, Inc. 84
Architecture and Design for a Virtual Infrastructure Workload Domain

High Availability of NSX-T Manager in a Single Region

The NSX-T Manager cluster for the workload domain runs on the first cluster in the management
domain. vSphere HA protects the NSX-T Manager appliances by restarting an NSX-T Manager
appliance on a different ESXi host if a primary ESXi host failure occurs.

Table 2-73. Design Decisions on the High Availability Configuration for NSX-T Manager for a
Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-017 Create a virtual IP (VIP) Provides high availability of n The VIP address
address for the NSX-T the user interface and API feature provides high
Manager cluster for the of NSX-T Manager. availability only. It
workload domain. does not load balance
requests across the
cluster.
n When using the VIP
address feature, all
NSX-T Manager nodes
must be deployed on
the same Layer 2
network.

VCF-WLD-VI-SDN-018 Apply VM-VM anti-affinity Keeps the NSX-T Manager n You must allocate at
rules in vSphere Distributed appliances running on least four physical hosts
Resource Scheduler different ESXi hosts for high so that the three NSX-
(vSphere DRS) to the NSX-T availability. T Manager appliances
Manager appliances. continue running if an
ESXi host failure occurs.
n You must perform
additional configuration
for the anti-affinity
rules.

VCF-WLD-VI-SDN-019 In vSphere HA, set the n NSX-T Manager If the restart priority
restart priority policy for implements the control for another management
each NSX-T Manager plane for virtual appliance is set to highest,
appliance to high. network segments. If the connectivity delays for
the NSX-T Manager services will be longer.
cluster is restarted,
applications that are
connected to the NSX-
T VLAN or virtual
network segments lose
connectivity only for a
short time until the
control plane quorum is
re-established.
n Setting the restart
priority to high reserves
highest for future
needs.

VMware, Inc. 85
Architecture and Design for a Virtual Infrastructure Workload Domain

High Availability of NSX-T Manager in Multiple Availability Zones

In an environment with multiple availability zones, the NSX-T Manager cluster runs in Availability
Zone 1. If a failure in Availability Zone 1 occurs, the NSX-T Manager cluster is failed over to
Availability Zone 1. All three appliances are in the same availability zone because separating the
manager appliances across availability zone does not improve high availability of the cluster.

Table 2-74. Design Decisions on the High Availability Configuration for NSX-T Manager for
Multiple Availability Zones for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-020 When using two availability Ensures that, by default, the None.
zones, add NSX-T Manager NSX-T Manager appliances
appliances to the VM group are powered on within the
of Availability Zone 1. primary availability zone
hosts group.

Network Design for NSX-T Manager for a Virtual Infrastructure Workload Domain
For traffic segmentation, you place NSX-T Manager for the workload domain on the management
VLAN and decide on the IP addressing scheme and name resolution for optimal support for the
SDDC tenant workloads.

Network Segment
For secure access to the ESXi hosts and vCenter Server, in each region, NSX-T Manager for the
workload domain is connected to the management VLAN segment.

Table 2-75. Design Decisions on the Network Segment for NSX-T Manager for a Workload
Domain
Decision
Decision ID Design Decision Design Justification Implication

VCF-WLD-VI- Place the appliances of the n Provides direct secure connection None.
SDN-021 NSX-T Manager cluster on the to the ESXi hosts and vCenter
management VLAN segment in Server for edge node management
each region, that is, sfo-m01-cl01- and distributed network services.
vds01-pg-mgmt for Region A and n Reduces the number of required
lax-m01-cl01-vds01-pg-mgmt for VLANs because a single VLAN
Region B. can be allocated to both, vCenter
Server and NSX-T.

IP Addressing Scheme
You can assign the IP addresses of the NSX-T Manager appliances by using DHCP or statically
according to the network configuration in your environment.

VMware, Inc. 86
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-76. Design Decisions on IP Addressing for NSX-T Manager for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- Allocate a statically assigned Ensures stability across the SDDC, Requires precise IP
SDN-022 IP address and host name makes it simpler to maintain and address management.
to the nodes of the NSX-T track, and to implement a DNS
Manager cluster. configuration.

Name Resolution
NSX-T Manager node name resolution, including the internal virtual IP addresses (VIPs), uses a
region-specific suffix, that is, sfo.rainpole.io for Region A, andlax.rainpole.io for Region B.

Table 2-77. Design Decisions on Name Resolution for NSX-T Manager for a Workload Domain

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- Configure forward and reverse The NSX-T Manager nodes and You must provide DNS
SDN-023 DNS records for the nodes of VIP address are accessible by records for the NSX-T
the NSX-T Manager cluster for using fully qualified domain Manager nodes for the
the workload domain, assigning names instead of by using IP workload domain in each
the record to the child domain addresses only. region.
in each region.

Time Synchronization
Time synchronization provided by the Network Time Protocol (NTP) is important to ensure that all
components within the SDDC are synchronized to the same time source.

Table 2-78. Design Decisions on Time Synchronization for NSX-T Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-024 Configure NTP on each NSX-T NSX-T Manager depends on time None.
Manager appliance. synchronization.

NSX-T Global Manager Deployment Specification and Network Design for a

Virtual Infrastructure Workload Domain
To implement a multi-region SDDC, you must use NSX-T Federation that requires the deployment
of NSX-T Global Manager nodes in the first two regions of the SDDC. You determine the size
of the compute resources, high availability implementation, and patching and upgrade support
for the NSX-T Global Manager instance for the management domain according to the design
objectives and aggregated requirements of the management components of the SDDC.

Deployment Model for NSX-T Global Manager for a Virtual Infrastructure Workload Domain
As a best practice, you must deploy a highly available NSX-T Global Manager instance so that the
NSX-T management plane can continue propagating configuration to the NSX-T Local Manager
nodes in each region. You also select an NSX-T Global Manager appliance size according to the
number of anticipated objects required to run the SDDC management components.

VMware, Inc. 87
Architecture and Design for a Virtual Infrastructure Workload Domain

Deployment Type
You can deploy NSX-T Global Manager in a one-node configuration or as a cluster for high
availability.

Table 2-79. Design Decisions on the Deployment Type of NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-025 For a dual-region SDDC, in Mutli-region deployment n You must turn on

Region A and Region B, requires that some SDDC vSphere HA in the
deploy three NSX-T Global management components first cluster in the
Manager nodes for the are placed on isolated management domain.
workload domain in the first virtual networks, using load n The first cluster in
cluster of the management balancing, logical switching, the workload domain
domain. dynamic routing, and requires four physical
logical firewalls services. ESXi hosts for vSphere
HA and for high
availability of the NSX-T
Manager cluster.

Sizing Compute and Storage Resources for NSX-T Global Manager

When you size the resources for NSX-T management components, consider the compute and
storage requirements for each component, and the number of nodes per component type.

Table 2-80. NSX-T Global Manager Resource Specification

Appliance Size vCPU Memory (GB) Storage (GB) Scale

Small 4 16 300 Proof of concept

Medium 6 24 300 Up to 64 ESXi hosts

Large 12 48 300 More than 64 ESXi

hosts

Table 2-81. Design Decisions on Sizing Resources for NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- For a dual-region SDDC, A medium-size appliance You must provide enough compute
SDN-026 deploy each node in the is sufficient for providing and storage resources in the
NSX-T Global Manager network services to the management domain to support this
cluster for the workload SDDC tenant workloads. NSX-T Global Manager cluster.
domain as a medium-size If you extend the workload domain,
appliance or larger. increasing the size of the NSX-T
Global Manager appliances might be
required.

High Availability of NSX-T Global Manager in a Single Region

The NSX-T Global Manager cluster runs on the first cluster in the workload domain. vSphere HA
protects the NSX-T Manager appliances by restarting an NSX-T Manager appliance on a different
ESXi host if a primary ESXi host failure occurs.

VMware, Inc. 88
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-82. Design Decisions on the High Availability Configuration for NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-027 For a dual-region SDDC, Provides high availability of n The VIP address
create a virtual IP (VIP) the user interface and API feature provides high
address for the NSX-T of NSX-T Global Manager. availability only. It
Global Manager cluster for does not load-balance
the workload domain. requests across the
cluster.
n When using the
VIP address feature,
all NSX-T Global
Manager nodes must be
deployed on the same
Layer 2 network.

VCF-WLD-VI-SDN-028 For a dual-region Keeps the NSX-T n You must allocate at

SDDC, apply VM-VM anti- Global Manager appliances least four physical hosts
affinity rules in vSphere running on different ESXi so that the three NSX-
Distributed Resource hosts for high availability. T Manager appliances
Scheduler (vSphere DRS) to continue running if an
the NSX-T Global Manager ESXi host failure occurs.
appliances. n You must perform
additional configuration
for the anti-affinity
rules.

VCF-WLD-VI-SDN-029 For a dual-region SDDC, in n NSX-T Global Manager n Management of NSX-

vSphere HA, set the restart implements the T global components
priority policy for each management plane for will be unavailable until
NSX-T Global Manager global segments and at least one NSX-T
appliance to medium. firewalls. Global Manager virtual
machine restarts.
NSX-T Global Manager
n The NSX-T Global
is not required for
Manager cluster is
control plane and data
deployed in the
plane connectivity.
management domain,
n Setting the restart
where the total number
priority to medium
of virtual machines
reserves the high
is limited and where
priority for services
it competes with
that impact the NSX-T
other management
control or data planes.
components for restart
priority.

High Availability of NSX-T Global Manager in Multiple Availability Zones

In an environment with multiple availability zones, the NSX-T Global Manager cluster runs in
Availability Zone 1. If a failure in Availability Zone 1 occurs, the NSX-T Global Manager cluster is
failed over to Availability Zone 2.

VMware, Inc. 89
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-83. Design Decisions on the High Availability Configuration for NSX-T Global Manager
for Multiple Availability Zones

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-030 For a dual-region SDDC, Ensures that, by default, None.

when using two availability the NSX-T Global Manager
zones in Region A, appliances are powered on
add the NSX-T Global on a host in the primary
Manager appliances to the availability zone.
virtual machine group for
Availability Zone 1.

High Availability of NSX-T Global Manager in Multiple Regions

In an environment with multiple regions, you can deploy multiple NSX-T Global Manager clusters
in an Active/Standby model. In this scenario, the NSX-T Global Manager cluster in one region is
active, and a second NSX-T Global Manager cluster in the second region is in a standby mode.

Table 2-84. Design Decisions on the High Availability Configuration for NSX-T Global Manager
for Multiple Regions

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-031 For a dual-region SDDC, Enables recoverablity of Requires additional NSX-T

deploy an additional NSX-T NSX-T Global Manager in a Global Manager nodes in
Global Manager Cluster in second region if a failure in Region B.
Region B. Region A occurs.

VCF-WLD-VI-SDN-032 For a dual-region SDDC, Enables recoverablity of the None.

set the NSX-T Global NSX-T Global Manager in a
Manager cluster in Region second region if a failure in
B as standby for the Region A occurs.
management domain.

Network Design for NSX-T Global Manager for a Virtual Infrastructure Workload Domain
For traffic segmentation, you place NSX-T Global Manager for the workload domain on the
management VLAN, and decide on the IP addressing scheme and name resolution for optimal
support for the SDDC management components and host management.
Network Segment
For secure access to the ESXi hosts and vCenter Server, in each region, NSX-T Global Manager for
the management domain is connected to the management VLAN segment.

Table 2-85. Design Decisions on the Network Segment for NSX-T Global Manager
Decision
Decision ID Design Decision Design Justification Implication

VCF-WLD-VI- For a dual-region SDDC, place the Reduces the number of None.
SDN-033 appliances of the NSX-T Global Manager required VLANs because a
cluster on the management VLAN network single VLAN can be allocated
in each region, that is, sfo-w01-cl01- to both vCenter Server and
vds01-pg-mgmt for Region A and lax-w01- NSX-T.
cl01-vds01-pg-mgmt for Region B.

VMware, Inc. 90
Architecture and Design for a Virtual Infrastructure Workload Domain

IP Addressing Scheme
You can assign the IP addresses of the NSX-T Global Manager appliances by using DHCP or
statically according to the network configuration in your environment.

Table 2-86. Design Decisions on IP Addressing for NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- For a dual-region SDDC, Ensures stability across the Requires precise IP
SDN-034 allocate a statically assigned SDDC, makes it simpler to address management.
IP address and host name to maintain and track, and to
the nodes of the NSX-T Global implement a DNS configuration.
Manager cluster.

Name Resolution
NSX-T Global Manager node name resolution, including the internal virtual IP addresses (VIPs),
uses a region-specific suffix, that is, sfo.rainpole.io for Region A, andlax.rainpole.io for
Region B.

Table 2-87. Design Decisions on Name Resolution for NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI- For a dual-region SDDC, The NSX-T Global Manager You must provide DNS
SDN-035 configure forward and reverse nodes and VIP address are records for the NSX-T
DNS records for the nodes of the accessible by using fully Global Manager nodes for
NSX-T Global Manager cluster for qualified domain names instead the workload domain in
the workload domain, assigning of by using IP addresses only. each region.
the record to the child domain in
the region.

Time Synchronization
Time synchronization provided by the Network Time Protocol (NTP) is important to ensure that all
components within the SDDC are synchronized to the same time source

Table 2-88. Design Decisions on Time Synchronization for NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-036 For a dual-region SDDC, NSX-T Global Manager depends None.

configure NTP on each NSX-T on time synchronization across all
Global Manager appliance. SDDC components.

NSX-T Edge Deployment Specification and Network Design for a Virtual

Infrastructure Workload Domain
For traffic segmentation, you place NSX-T Manager for the workload domain on the management
VLAN in the management domain and decide on the IP addressing scheme and name resolution
for optimal support for the NSX-T Edge nodes.

VMware, Inc. 91
Architecture and Design for a Virtual Infrastructure Workload Domain

n Deployment Specification of the NSX-T Edge Nodes for a Virtual Infrastructure Workload
Domain
You determine the size of the compute resources, high availability implementation, and
patching and upgrade support for the NSX-T Edge nodes for the workload domain according
to the design objectives and aggregated requirements of the tenant workloads of the SDDC.

n Network Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
You implement an NSX-T Edge configuration with a single N-VDS. You connect the uplink
network interfaces of the edge appliance to VLAN trunk port groups that are connected to
particular physical NICs on the host.

n Uplink Policy Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
By using uplink profiles, you can apply consistent policy on the uplinks of the N-VDS instance
on each NSX-T Edge appliance. The uplink profile for the NSX-T Edge appliances supports
the VLANs for connection to physical Layer 3 devices.

Deployment Specification of the NSX-T Edge Nodes for a Virtual Infrastructure Workload
Domain
You determine the size of the compute resources, high availability implementation, and patching
and upgrade support for the NSX-T Edge nodes for the workload domain according to the design
objectives and aggregated requirements of the tenant workloads of the SDDC.

n Deployment Model for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
For NSX-T Edge nodes, you determine the form factor and the appliance number and place
according to the requirements for network services in the workload domain.

n High Availability Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload
Domain
The NSX-T Edge cluster for a workload domain runs on the shared edge and workload
cluster. vSphere HA and vSphere DRS protect the NSX-T Edge appliances. In an environment
with multiple availability zones, to configure the first availability zone as the main location for
the NSX-T Edge nodes, you use vSphere DRS.

Deployment Model for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
For NSX-T Edge nodes, you determine the form factor and the appliance number and place
according to the requirements for network services in the workload domain.

An NSX-T Edge node is an appliance that provides centralized networking services which can not
be distributed to hypervisors, such as load balancing, NAT, VPN, and physical network uplinks.
Some services, such as T-0 gateways, are limited to a single instance per NSX-T Edge node.
However, most services can coexist in these nodes.

NSX-T Edge nodes are grouped in one or more Edge Clusters, representing a pools of capacity for
NSX-T Data Center services.

Form Factors of NSX-T Edge Nodes

VMware, Inc. 92
Architecture and Design for a Virtual Infrastructure Workload Domain

An NSX-T Edge node can be deployed as a virtual appliance or installed on bare metal hardware.
The edge node on bare-metal hardware can have better performance capabilities at the expense
of more difficult deployment and limited deployment topology use cases.

Design Component Edge Virtual Appliance Edge Bare Metal Appliance Considerations

Ease of deployment and ↑↑ ↓ n You can deploy NSX-T

ease of expansion Edge virtual appliances
by using the vSphere
Client user interface
or via SDDC Manager
automated deployment.
n Deployment of bare
metal appliances
have certain
hardware compatibility
requirements and must
be manually deployed
and connected to the
environment.

Ease of upgrade and life o ↓ n NSX-T Manager

cycle management provides life cycle
management for
associated NSX-T Edge
nodes.
n NSX-T Edge nodes
on bare metal
hardware require
individual hardware life
cycle management of
firmware, drivers, and
so on.

Manageability ↑↑ o n NSX-T Edge nodes

on bare metal
hardware require
individual monitoring
and management of
the hardware, such as
failures, firmware, and
so on.
n SDDC Manager can
rotate the passwords
only of the NSX-
T Edge virtual
appliances. Because
SDDC Manager is not
aware of the bare-metal
NSX-T Edge appliances,
it is unable to rotate
their passwords.

VMware, Inc. 93
Architecture and Design for a Virtual Infrastructure Workload Domain

Design Component Edge Virtual Appliance Edge Bare Metal Appliance Considerations

Availability and ↑↑ ↓ n If an ESXi host

recoverability running an NSX-T
Edge virtual appliance
node fails, vSphere HA
automatically restarts
the appliance on a
remaining ESXi host in
the vSphere Cluster.
n If a failure of an NSX-
T Edge node on bare
metal hardware occurs,
it must be redeployed.

Capability o o NSX-T Edge virtual

appliances and NSX-T Edge
nodes on bare metal
hardware provide the same
network services.

Design agility ↑ ↓ You can use bare-metal

NSX-T Edge nodes only
in an SDDC with a single
availability zone in the
management domain.

Performance ↑ ↑↑ n In certain use cases,

an NSX-T Edge node
on bare metal hardware
can support more raw
performance and lower
latency.
n The performance of
an NSX-T Edge virtual
appliance depends on
the underlying ESXi
host. To improve
performance, migrate
the appliance to a
host that has higher
performance.

Capacity o ↑ n In rare use cases, an

NSX-T Edge node on
bare metal hardware
can support more raw
capacity.
n NSX-T Edge virtual
appliance can easily be
scaled up and down as
needed.

Sizing Compute and Storage Resources for NSX-T Edge Nodes

VMware, Inc. 94
Architecture and Design for a Virtual Infrastructure Workload Domain

When you size the resources for NSX-T components, consider the compute and storage
requirements for each component, and the number of nodes per component type.

Table 2-89. Resource Specifications of NSX-T Edge Nodes per Form Factor

Form Factor Appliance Size CPU or vCPU Memory (GB) Storage (GB)

NSX-T Edge Minimum - 8 CPU 8 200

on bare metal requirements
hardware
Recommended - 24 CPU 256 500
requirements

NSX-T Edge virtual appliance Small 2 vCPU 4 200

Use for proof of
concept only

Medium 4 vCPU 8 200

Large 8 vCPU 32 200

Use in large
environments
that require load
balancers

Extra Large 16 64 200

Table 2-90. Design Decisions on the Form Factor and Sizing for the NSX-T Edge Nodes

Design ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-037 Use large-sized NSX-T n The large-sized None.

Edge virtual appliances. appliance provides the
required performance
characteristics for most
tenant workloads.
n The deployment and life
cycle management of
virtual edges node is
simpler.
n You can migrate
virtual machine form
factor between physical
locations to support
advanced deployment
topologies such as
multiple availability
zones.

High Availability Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
The NSX-T Edge cluster for a workload domain runs on the shared edge and workload cluster.
vSphere HA and vSphere DRS protect the NSX-T Edge appliances. In an environment with
multiple availability zones, to configure the first availability zone as the main location for the NSX-T
Edge nodes, you use vSphere DRS.

NSX-T Edge Cluster Design

VMware, Inc. 95
Architecture and Design for a Virtual Infrastructure Workload Domain

The NSX-T Edge cluster is a logical grouping of NSX-T Edge transport nodes. These NSX-T Edge
appliances run on a vSphere Cluster, and provide north-south routing and network services for
tenant workloads. You can dedicate this vSphere Cluster only to NSX-T Edge appliances or it can
be shared with tenant workloads.

Shared Edge and Workload Cluster in the workload domain

The first cluster in the workload domain contains NSX-T Edge components and tenant
workloads. See the vSphere Cluster Design for a Virtual Infrastructure Workload Domain.

Dedicated Edge vSphere Cluster

A dedicated vSphere Cluster contains only NSX-T Edge appliances.

Table 2-91. Design Decisions on the NSX-T Edge Cluster Configuration

Decision ID Design Decision Design Justification Design Implications

VCF-WLD-VI-SDN-038 n Deploy the NSX-T Edge n Keeps tenant network NSX-T Edge appliances
virtual appliances to traffic local to the are co-located with tenant
the shared edge and workload domain. workloads. Ensure that
workload cluster in the n Simplifies configuration tenant workloads do not
workload domain. and minimizes the prevent NSX-T Edge nodes
n Do not configure number of ESXi hosts from moving network
a dedicated vSphere required for initial traffic.
cluster for edge nodes. deployment.

VCF-WLD-VI-SDN-039 Deploy two NSX-T Edge Creates the NSX-T Edge None.
appliances in an edge Cluster for satisfying the
cluster in the shared edge requirements for availability
and workload cluster. and scale.

VCF-WLD-VI-SDN-040 Create a Resource Pool in n Tenant workloads must Deploying a Resource Pool
the root of the shared edge be deployed to a for NSX-T Edge appliances
and workload cluster object separate Resource Pool allows you to ensure
for NSX-T Edge appliances. at the root of the the NSX-T Edge Cluster
Create a separate Resource cluster. receives adequate compute
Pool in the root of the n To ensure adequate resources during times of
shared edge and workload resources for tenant contention.
vSphere Cluster object for and control plane Tenant workloads might
tenant workloads. workloads, there must not be able to use their
be no virtual machines allocated memory during
in the root of the times of contention.
cluster.

VMware, Inc. 96
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-91. Design Decisions on the NSX-T Edge Cluster Configuration (continued)

Decision ID Design Decision Design Justification Design Implications

VCF-WLD-VI-SDN-041 Configure the NSX-T Edge n Configuring the Tenant workloads might
Resource Pool with a 64GB Resource Pool for NSX- not be able to use their
memory reservation and T Edge appliances allocated memory or CPU
high CPU share value. with a full memory during times of contention.
reservation allows you
to ensure the NSX-T
Edge Cluster receives
adequate memory
resources during times
of contention.
n Setting the CPU share
value to high gives
priority to NSX-T Edge
appliances in times of
CPU contention.

VCF-WLD-VI-SDN-042 Apply VM-VM anti-affinity Keeps the NSX-T Edge None.

rules for vSphere DRS to nodes running on different
the virtual machines of the ESXi hosts for high
NSX-T Edge cluster. availability.

VCF-WLD-VI-SDN-043 In vSphere HA, set the n NSX-T Edge VMs are If the restart priority for
restart priority policy for part of the north/south another VM is set to
each NSX-T Edge appliance data path for overlay highest, the connectivity
to high. segments. High priority delays for other appliances
order ensure lower will be longer.
priority VMs will have
connectivity.
n Setting the restart
priority to high reserves
highest for future
needs.

VCF-WLD-VI-SDN-044 Configure all NSX-T Edge Enables the participation of None.

nodes as transport nodes. NSX-T Edge nodes in the
overlay network for delivery
of services to the SDDC
workloads such as routing
and load balancing.

VCF-WLD-VI-SDN-045 Create an NSX-T n Satisfies the availability None.

Edge Cluster with requirements by
the default Bidirectional default.
Forwarding Detection n Edge nodes must
(BFD) configuration remain available to
between the NSX-T Edge create services such
nodes in the cluster. as NAT, routing to
physical networks, and
load balancing.

High Availability for Multiple Availability Zones

VMware, Inc. 97
Architecture and Design for a Virtual Infrastructure Workload Domain

NSX-T Edge nodes connect to top of rack switches in each data center to support northbound
uplinks and route peering for SDN network advertisement. This connection is specific to the top of
rack switch that you are connected to.

If an outage of an availability zone occurs, vSphere HA fails over the edge appliances to the other
availability zone by using vSphere HA. Availability Zone 2 must provide an analog of the network
infrastructure which the edge node is connected to in Availability Zone 1.

To support failover of the NSX-T Edge appliances, the following networks are stretched across
Availability Zone 1 to Availability Zone 2 to support failover. For information about the networks in
a workload domain with multiple availability zones, see Network Design for NSX-T Manager for a
Virtual Infrastructure Workload Domain.

Table 2-92. Networks That Are Stretched Across Availability Zones

Function Sample VLAN ID Sample Subnet

AZ1 Management 1631 172.16.31.0/24

Uplink01 2731 172.27.31.0/24

Uplink02 2732 172.27.32.0/24

NSX-T Edge Overlay 2733 172.27.33.0/24

Table 2-93. Design Decisions on the High Availability Configuration of the NSX-T Edge Nodes for
Multiple Availability Zones

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-046 When using two availability Ensures that, by default, None.

zones, add NSX-T Edge the NSX-T Edge appliances
appliances to the VM group are powered on within the
of Availability Zone 1. primary availability zone
hosts group.

High Availability for a Multiple-Region SDDC

In a multi-region deployment, each region has its own NSX-T Edge cluster. In each region, the
edge nodes and clusters are deployed with the same design but with region-specific settings
such as IP addressing, VLAN IDs, and names. Each edge cluster is managed by the NSX-T Local
Manager instance for that region and workload domain.

Region-to-region workload traffic traverses the inter-region overlay tunnels which terminate on
the RTEPs on the NSX-T Edge nodes. This tunnel is the data plane for inter-region traffic.

To support inter-region communication, you must allocate additional VLANs on the edge nodes.
If the region also contains multiple availability zones, this network must be stretched across all
availability zones.

VMware, Inc. 98
Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-94. Edge RTEP VLAN Configuration for a Multi-Region SDDC

Availability Zone Availability Zone HA Layer 3
Function 1 2 VLAN ID IP Range Gateway

Edge RTEP - ✓ ✓ 2734 172.27.34.0/24 ✓

Region A

Edge RTEP - ✓ X 2834 172.28.34.0/24 ✓

Region B

Note If the workload domain spans additional regions, each region requires an Edge RTEP VLAN
configured with a VLAN ID and IP range that are appropriate for the data center in that region.

Network Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
You implement an NSX-T Edge configuration with a single N-VDS. You connect the uplink
network interfaces of the edge appliance to VLAN trunk port groups that are connected to
particular physical NICs on the host.

The NSX-T Edge node contains an NSX-T managed virtual switch called an N-VDS. This internal
N-VDS is used to define traffic flow through the interfaces of the edge node. An N-VDS can be
connected to one or more interfaces. Interfaces can not be shared between N-VDS instances.

If you plan to deploy a multi-region SDDC, apply the same network design to the NSX-T Edge
cluster in the recovery and other additional regions.

VMware, Inc. 99
Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-15. NSX-T Edge Networking

ToR
Switches

vmnic0 vmnic1

sfo-w01-cl01-vds01

sfo01-w01-cl01-vds01-pg sfo01-w01-cl01-vds01-pg sfo01-w01-cl01-vds01-pg

-mgmt -uplink01 -uplink02

Eth0 fp-eth0 fp-eth1 fp-eth2

(mgmt) (uplink 1) (uplink 2) (unused)

sfo-w01-cl01-ndvs01

TEP1 RTEP TEP2

Uplink 1 Uplink 2
(VLAN) (VLAN)
Overlay Segments

NSX-T Edge Node

ESXi Host

VMware, Inc. 100

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-95. Design Decisions on the Network Configuration of the NSX-T Edge Appliances

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-047 Connect the management Provides connection to the None.

interface eth0 of each NSX-T Manager cluster.
NSX-T Edge node to the
management VLAN.

VCF-WLD-VI-SDN-048 n Connect the fp-eth0 Because VLAN trunk port None.

interface of each NSX- groups pass traffic for all
T Edge appliance to a VLANs, VLAN tagging can
VLAN trunk port group occur in the NSX-T Edge
pinned to physical NIC node itself for easy post-
0 of the host. deployment configuration.
n Connect the fp-eth1 n By using two separate
interface of each NSX- VLAN trunk port
T Edge appliance to a groups, you can direct
VLAN trunk port group traffic from the NSX-
pinned to physical NIC 1 T Edge node to a
of the host. particular host network
n Leave the fp-eth2 interface and top of
interface of each NSX-T rack switch as needed.
Edge appliance unused.

VCF-WLD-VI-SDN-049 Use a single N-VDS in the n Simplifies deployment None.

NSX-T Edge nodes. of the edge nodes.
n The same N-VDS switch
design can be used
regardless of edge form
factor.
n Supports multiple TEP
interfaces in the edge
node.
n vSphere Distributed
Switch is not supported
in the edge node.

Uplink Policy Design for the NSX-T Edge Nodes for a Virtual Infrastructure Workload Domain
By using uplink profiles, you can apply consistent policy on the uplinks of the N-VDS instance
on each NSX-T Edge appliance. The uplink profile for the NSX-T Edge appliances supports the
VLANs for connection to physical Layer 3 devices.

Uplink profiles define policies for the links from the NSX-T Edge transport nodes to top of rack
switches. Uplink profiles are containers for the properties or capabilities for the network adapters.
Uplink profiles are applied to the N-VDS of the NSX-T Edge transport node.

Uplink profiles can use either load balance source or failover order teaming. If using load balance
source, multiple uplinks can be active. If using failover order, only a single uplink can be active.

Teaming can be configured by using the default teaming policy or a user-defined named teaming
policy. You can use named teaming policies to pin traffic segments to designated edge uplinks.

VMware, Inc. 101

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-96. Design Decisions on the NSX-T Edge Uplink Policy

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-050 Create one uplink profile for n An NSX-T Edge node You can use this policy
the edge node with three that uses a single N- only with ESXi hosts. Edge
teaming policies. VDS can have only one virtual machines must use
n Default teaming policy uplink profile. the failover order teaming
of load balance source n For increased resiliency policy.
both active uplinks and performance,
uplink-1 and uplink-2. supports the concurrent
n Named teaming policy use of both edge
of failover order uplinks through both
with a single active physical NICs on the
uplink uplink-1 without ESXi hosts.
standby uplinks. n The default teaming
n Named teaming policy policy increases overlay
of failover order performance and
with a single active availability by using
uplink uplink-2 without multiple Host Overlay
standby uplinks. VMkernel ports and
appropriate balancing
of overlay traffic.
n By using named
teaming policies, you
can connect an edge
uplink to a specific host
uplink and from there
to a specific top of
rack switch in the data
center.
n Enables ECMP in
each availability zone
because the NSX-T
Edge nodes can uplink
to the physical network
over two different
VLANs.
VLAN ID is required in
the uplink profile. Hence,
you must create an uplink
profile for each VLAN used
by the NSX-T Edge virtual
machines.

VCF-WLD-VI-SDN-051 Use a dedicated VLAN for The NSX-T Edge Overlay n You must have routing
the NSX-T Edge Overlay network must be isolated between the VLANs for
network that is segmented from the Host Overlay NSX-T Edge Overlay
from the Host Overlay network to protect the Host and Host Overlay.
VLAN. Overlay traffic from NSX- n You must allocate
T Edge-generated overlay another VLAN in
traffic. the data center
infrastructure for NSX-T
Edge Overlay traffic.

VMware, Inc. 102

Architecture and Design for a Virtual Infrastructure Workload Domain

For a multi-region SDDC, an RTEP VLAN is needed for overlay traffic between regions.

Table 2-97. Design Decisions on the Network Configuration of the NSX-T Edge Appliances for a
Multi-Region SDDC

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-052 For a dual-region SDDC, n The RTEP network must You must allocate another
allocate a separate VLAN on a VLAN that is VLAN in the data center
for edge RTEP overlay that different from the edge infrastructure for edge
is different from the edge overlay VLAN. RTEP overlay.
overlay VLAN. n Dedicated VLAN
for inter-site
communication.

Life Cycle Management Design of NSX-T Data Center for a Virtual Infrastructure
Workload Domain
You decide on the life cycle management of the NSX-T Data Center components according to
the amount of time and effort to perform a deployment, upgrade, or patch operation. You also
consider the impact such an operation has on the solutions that are connected to NSX-T Data
Center for the workload domain.

Life cycle management of NSX-T involves the process of applying patches, updates, or upgrades
to the NSX-T Data Center appliances and hypervisor components. In a typical environment, you
perform life cycle management by using the Upgrade Coordinator which is a service in NSX-T
Manager. When you implement a solution by using VMware Cloud Foundation, you use SDDC
Manager for life cycle management of the entire SDDC.

Table 2-98. Design Decisions on Life Cycle Management of NSX-T Data Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-053 Use SDDC Manager Because the deployment The operations team must
to perform the life scope of SDDC Manager understand and be aware
cycle management of covers the full SDDC of the impact of a
NSX-T Manager and stack, SDDC Manager patch, update, or upgrade
related components in the performs patching, update, operation by using SDDC
workload domain. or upgrade of the workload Manager.
domain as a single process.

Life Cycle Management in a Multi-Region Deployment

In a multi-region deployment, the NSX-T Global Manager and NSX-T instances in the individual
workload domains must be compatible and supported version with each other both during the
upgrade and post-upgrade. To manage the life cycle of NSX-T Global Manager, use the Upgrade
Coordinator on the the NSX-T Global Manager appliances. The version of SDDC Manager in this
design does not handle the life cycle of NSX-T Global Manager nodes. For the NSX-T Local
Manager appliances, you continue using SDDC Manager.

VMware, Inc. 103

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-99. Design Decisions on Life Cycle Management of NSX-T Data Center for a Multi-Region
SDDC

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-054 For a dual-region SDDC, The version of SDDC You must always align the
use NSX-T Upgrade Manager in this design is version of the NSX-T Global
Coordinator to perform not currently capable of life Manager nodes with the
Perform life cycle cycle operations (patching, rest of the SDDC stack in
management on the NSX-T update, or upgrade) for VMware Cloud Foundation.
Global Manager appliances. NSX-T Global Manager. You must explicitly plan
upgrades of the NSX-T
Global Manager nodes.
An upgrade of the NSX-
T Global Manager nodes
might require a cascading
upgrade of the NSX-T
Local Manager nodes and
underlying SDDC Manager
infrastructure prior to the
upgrade of the NSX-T
Global Manager nodes.
An upgrade of the
workload domain from
SDDC Manager might
include an upgrade of
the NSX-T Local Manager
cluster. which might require
an upgrade of the NSX-
T Global Manager cluster.
An upgrade of NSX-T
Global Manager might
then require that you
upgrade all other workload
domains connected to it
before you can proceed
with upgrading the NSX-T
Global Manager instance.

VCF-WLD-VI-SDN-055 For a dual-region SDDC, The versions of NSX-T The administrator must
establish an operations Global Manager and NSX-T establish and follow an
practice to ensure that Local Manager nodes must operational practice by
prior to the upgrade of be compatible with each using a runbook or
any workload domain, the other. automated process to
impact of any version Because the version of ensure a fully supported
upgrades is evaluated SDDC Manager in this and compliant bill of
regarding the need to design does not provide materials prior to any
upgrade NSX-T Global of life cycle operations upgrade operation.
Manager. (patching, update, or
upgrade) for the NSX-T
Global Manager nodes,
upgrade to an unsupported
version cannot be
prevented.

VCF-WLD-VI-SDN-056 For a dual-region SDDC, The versions of NSX-T The administrator must
establish an operations Global Manager and NSX-T establish and follow an

VMware, Inc. 104

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-99. Design Decisions on Life Cycle Management of NSX-T Data Center for a Multi-Region
SDDC (continued)

Decision ID Design Decision Design Justification Design Implication

practice to ensure that Local Manager nodes must operational practice by

prior to the upgrade be compatible with each using a runbook or
of the NSX-T Global other. automated process to
Manager, the impact of any Because the version of ensure a fully supported
version change is evaluated SDDC Manager in this and compliant bill of
against the existing NSX-T design does not provide materials prior to any
Local Manager nodes and of life cycle operations upgrade operation.
workload domains. (patching, update, or
upgrade) for the NSX-T
Global Manager nodes,
upgrade to an unsupported
version cannot be
prevented.

NSX-T Services Design for a Virtual Infrastructure Workload Domain

NSX-T Edge clusters are pools of capacity for NSX-T service router and load balancing functions.

NSX-T Services Design for a Virtual Infrastructure Workload Domain

NSX-T Edge clusters are pools of capacity for NSX-T service router and load balancing functions.
North - South Routing
The routing design considers different levels of routing in the environment, such as number and
type of NSX-T gateways, dynamic routing protocol, and so on. At each level, you apply a set of
principles for designing a scalable routing solution.

Routing can be defined in the following directions:

n North-south traffic is traffic leaving or entering the NSX-T domain, for example, a virtual
machine on an overlay network communicating with an end-user device on the corporate
network.

n East-west traffic is traffic that remains in the NSX-T domain, for example, two virtual machines
on the same or different segments communicating with each other.

As traffic flows north-south, edge nodes can be configured to pass traffic in an active-standby or
an active-active model, where active-active can scale up to 8 active nodes. NSX-T service routers
(SRs) for north-south routing are configured an active-active equal-cost multi-path (ECMP) mode
that supports route failover of Tier-0 gateways in seconds.

VMware, Inc. 105

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-100. Features of Active-Active and Active-Standby SRs

Design Component Active-Active Active-Standby Comment

Bandwidth per node 0 0 Bandwidth per node is

the same because it is
independent of the Tier- 0
gateway failover model.

Total aggregate bandwidth ↑↑↑↑ 0 n The active-active mode

can support up to 8
NSX-T Edge nodes per
northbound SR.
n The active-standby
mode is limited to
a single active NSX-T
node.

Availability ↑ 0 With up to 8 active-

active NSX-T Edge nodes,
availability can be as high
as N+7, while for the active-
standby mode it is N+1.

Failover Time ↓ ↓ Both are capable of sub-

second failover with use of
BFD, only when using Bare
Metal Edge form factor.

Routing Protocol Support ↓ 0 The active-active mode

requires BGP for ECMP
failover.

VMware, Inc. 106

Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-16. Dynamic Routing in a Single Availability Zone

Region A
ToR
Switches

Region A
BGP ASN

Uplink VLAN 1 BGP

ECMP
Uplink VLAN 2 BFD (Optional) Default Route

Tier-0
Gateway
SR SR
SDDC
BGP ASN DR DR

ESXi Transport Nodes

SR SR

Tier-1 DR DR DR DR DR DR
Gateway
NSX-T NSX-T ESXi ESXi ESXi ESXi
Edge Edge Host 1 Host 2 Host 3 Host 4
Node 1 Node 2

VM VM VM

Table 2-101. Design Decisions on the High Availability Mode of Tier-0 Gateways

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-057 Deploy an active-active Supports ECMP north- Active-active Tier-0

Tier-0 gateway. south routing on all Edge gateways cannot provide
nodes in the NSX-T Edge stateful services such as
cluster. NAT.

VMware, Inc. 107

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-102. Design Decisions on Edge Uplink Configuration for North-South Routing

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-058 To enable ECMP between Supports multiple equal- Additional VLANs are
the Tier-0 gateway and cost routes on the Tier-0 required.
the Layer 3 devices gateway and provides
(ToR switches or upstream more resiliency and better
devices), create two bandwidth use in the
VLANs. network.
The ToR switches or
upstream Layer 3 devices
have an SVI on one of the
two VLANs and each NSX-
T Edge node in the cluster
has an interface on each
VLAN.

VCF-WLD-VI-SDN-059 Assign a named teaming Pins the VLAN traffic on None.

policy to the VLAN each segment to its target
segments to the Layer 3 Edge node interface. From
device pair. there the traffic is directed
to the host physical NIC that
is connected to the target
top of rack switch.

VCF-WLD-VI-SDN-060 Create a VLAN transport Enabled the configuration Additional VLAN transport
zone for NSX-T Edge uplink of VLAN segments on the zones are required if
traffic. N-VDS in the Edge nodes. the edge nodes are not
connected to the same top
of rack switch pair.

Table 2-103. Design Decisions on Dynamic Routing

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-061 Use BGP as the dynamic n Enables the dynamic In environments where
routing protocol. routing by using NSX- BGP cannot be used, you
T. NSX-T supports must configure and manage
only BGP for dynamic static routes.
routing.
n SDDC architectures
with multiple availability
zones or multiple
regions architectures
require BGP.

VCF-WLD-VI-SDN-062 Configure the BGP Keep Provides a balance By using longer timers to
Alive Timer to 4 and Hold between failure detection detect if a router is not
Down Timer to 12 between between the top of responding, the data about
the top of tack switches and rack switches and the such a router remains in
the Tier-0 gateway. Tier-0 gateway and the routing table longer. As
overburdening the top of a result, the active router
rack switches with keep- continues to send traffic to
alive traffic. a router that is down.

VMware, Inc. 108

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-103. Design Decisions on Dynamic Routing (continued)

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-063 Do not enable Graceful Avoids loss of traffic. None.

Restart between BGP On the Tier-0 gateway,
neighbors. BGP peers from all the
gateways are always active.
On a failover, the Graceful
Restart capability increases
the time a remote neighbor
takes to select an alternate
Tier-0 gateway. As a result,
BFD-based convergence is
delayed.

VCF-WLD-VI-SDN-064 Enable helper mode for Avoids loss of traffic. None.

Graceful Restart mode During a router restart,
between BGP neighbors. helper mode works with the
graceful restart capability
of upstream routers to
maintain the forwarding
table which in turn will
forward packets to a down
neighbor even after the
BGP timers have expired
causing loss of traffic.

VCF-WLD-VI-SDN-065 Enable Inter-SR iBGP In the event that an None.

routing. NSX-T Edge node as all
of its northbound eBGP
sessions are down, north-
south traffic will continue to
flow by routing traffic to a
different NSX-T Edge node.

This design assumes that physical network does not support Bidirectional Forwarding Detection
(BFD). Enable BFD if the network supports and is configured for BFD.
Intra-SDN Routing
Gateways are needed to provide routing between logical segments created in the NSX-T based
SDN. Logical segments can be connected directly to a Tier-0 or Tier-1 gateway.

VMware, Inc. 109

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-104. Design Decisions on Tier-1 Gateway Configuration

Decision ID Design Decision Design Implication Design Justification

VCF-WLD-VI-SDN-066 Deploy a Tier-1 gateway Creates a two-tier routing A Tier-1 gateway can only
and connect it to the Tier-0 architecture. be connected to a single
gateway. Tier-0 gateway.
In cases where multiple
Tier-0 gateways are
required, you must create
multiple Tier-1 gateways.

VCF-WLD-VI-SDN-067 Deploy a Tier-1 gateway to Enables stateful services, None.

the NSX-T Edge cluster. such as load balancers
and NAT, for SDDC
management components.
Because a Tier-1 gateway
always works in active-
standby mode, the gateway
supports stateful services.

VCF-WLD-VI-SDN-068 Deploy a Tier-1 gateway Ensures that after a failed None.

in non-preemptive failover NSX-T Edge transport node
mode. is back online, it does
not take over the gateway
services thus causing a
short service outage.

VCF-WLD-VI-SDN-069 Enable standby relocation Ensures that if an edge None.

of the Tier-1 gateway. failure occurs, a standby
Tier-1 gateway will be
created on another edge
node.

Dynamic Routing in Multiple Availability Zones

In an environment with multiple availability zones, plan for failover of the NSX-T Edge nodes and
configuring BGP so that traffic from the top of rack switches is directed to Availability Zone 1 unless
a failure in Availability Zone 1 occurs.

VMware, Inc. 110

Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-17. Dynamic Routing in Multiple Availability Zones

Region A

Availability Zone1 eBGP (loc-pref Availability Zone 2

ToR & ASPath_prepend)
eBGP ToR
Switches ECMP
ECMP Switches
BFD (Optional
BDF (OPtional)
Region A - AZ 1
BGP ASN Region A - AZ 2
Default Route Default Route BGP ASN
Uplink VLAN 1 (low loc-pref)

Uplink VLAN 2

Tier-0 SR SR
Gateway
DR DR
SDDC BGP ASN

ESXi Transport Nodes ESXi Transport Nodes

SR SR

DR DR DR DR DR DR Tier-1 DR DR DR DR
Gateway
ESXi ESXi ESXi ESXi NSX-T NSX-T ESXi ESXi ESXi ESXi
Host 1 Host 2 Host 3 Host 4 Edge Edge Host 1 Host 2 Host 3 Host 4
Node 1 Node 2

VM VM VM

VMware, Inc. 111

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-105. Design Decisions on North-South Routing for Multiple Availability Zones

Decision ID Design Decision Design Implication Design Justification

VCF-WLD-VI-SDN-070 When you have two Because the NSX-T Edge You must configure a
availability zones, extend nodes will fail over between stretched Layer 2 network
the uplink VLANs to the the availability zones, this between the availability
top of rack switches so that ensures uplink connectivity zones by using physical
the VLANs are stretched to the top of rack switches network infrastructure.
between both availability in both availability zones
zones. regardless of the zone the
NSX-T Edge nodes are
presently in.

VCF-WLD-VI-SDN-071 When you have two Enables the communication You must configure a
availability zones, provide of the NSX-T Edge nodes stretched Layer 2 network
this SVI configuration on to both the top of rack between the availability
the top of the rack switches in both availability zones by using the physical
switches or upstream Layer zones over the same uplink network infrastructure.
3 devices. VLANs.
n In Availability Zone
2, configure the top
of rack switches or
upstream Layer 3
devices with an SVI on
each of the two uplink
VLANs.
n Make the top of
rack switch SVI in
both availability zones
part of a common
stretched Layer 2
network between the
availability zones.

VCF-WLD-VI-SDN-072 When you have two Supports multiple equal- Extra VLANs are required.
availability zones, provide cost routes on the Tier-0 Requires stretching uplink
this VLAN configuration. gateway and provides VLANs between Availability
n Use two VLANs to more resiliency and better zones
enable ECMP between bandwidth use in the
the Tier-0 gateway and network.
the Layer 3 devices
(top of rack switches or
upstream devices).
n The ToR switches or
upstream Layer 3
devices have an SVI to
one of the two VLANS
and each NSX-T Edge
node has an interface to
each VLAN.

VMware, Inc. 112

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-105. Design Decisions on North-South Routing for Multiple Availability Zones (continued)

Decision ID Design Decision Design Implication Design Justification

VCF-WLD-VI-SDN-073 Create an IP prefix list Used in a route map to You must manually create
that permits access to prepend a path to one or an IP prefix list that is
route advertisement by any more autonomous system identical to the default one.
network instead of using (AS-path prepend) for BGP
the default IP prefix list. neighbors in Availability
Zone 2.

VCF-WLD-VI-SDN-074 Create a route map-out n Used for configuring You must manually create
that contains the custom neighbor relationships the route map.
IP prefix list and an AS- with the Layer 3 devices The two NSX-T Edge nodes
path prepend value set to in Availability Zone 2. will route north-south traffic
the Tier-0 local AS added n Ensures that all ingress through Availability Zone
twice. traffic passes through 2 only if the connection
Availability Zone 1. to their BGP neighbors in
Availability Zone 1 is lost, for
example, if a failure of the
top of the rack switch pair
or in the availability zone
occurs.

VCF-WLD-VI-SDN-075 Create an IP prefix list that Used in a route map to You must manually create
permits access to route configure local-reference an IP prefix list that is
advertisement by network on learned default-route identical to the default one.
0.0.0.0/0 instead of using for BGP neighbors in
the default IP prefix list. Availability Zone 2.

VCF-WLD-VI-SDN-076 Apply a route map-in that n Used for configuring You must manually create
contains the IP prefix list for neighbor relationships the route map.
the default route 0.0.0.0/0 with the Layer 3 devices The two NSX-T Edge nodes
and assign a lower local- in Availability Zone 2. will route north-south traffic
preference , for example, n Ensures that all egress through Availability Zone
80 to the learned default traffic passes through 2 only if the connection
route and a lower local- Availability Zone 1. to their BGP neighbors in
preference, for example, 90 Availability Zone 1 is lost, for
any routes learned. example, if a failure of the
top of the rack switch pair
or in the availability zone
occurs.

VCF-WLD-VI-SDN-077 Configure Availability Zone Makes the path in and out The two NSX-T Edge nodes
2 neighbors to use the of Availability Zone 2 less will route north-south traffic
route maps as In and Out preferred because the AS through Availability Zone
filters respectively. path is longer. As a result, 2 only if the connection
all traffic passes through to their BGP neighbors in
Availability Zone 1. Availability Zone 1 is lost, for
example, if a failure of the
top of the rack switch pair
or in the availability zone
occurs.

VMware, Inc. 113

Architecture and Design for a Virtual Infrastructure Workload Domain

NSX-T Routing for a Multi-Region SDDC for a Virtual Infrastructure Workload Domain
Design the routing configuration in NSX-T Data Center for multiple regions to support network
span between regions for management applications that require resilient connectivity at multiple
locations and to enable granular control of traffic from and to each region.
North-South Routing in Multiple Regions
In a routing design for a multi-region deployment, you identify which regions an SDN network
must span and which regions must let ingress and egress traffic.

Network traffic that is entering or leaving the SDN networks with region preference and failover
is a key design choice for a multi-site deployment. This design does not use local-egress, that
is, traffic leaving and entering any location which the network spans. Instead,this design uses a
preferred and failover region for all networks. The complexities of local-egress that is, controlling
local-ingress to prevent asymmetrical routing, is not necessary for this design.

In this design, an NSX-T component can be primary at one or more regions. During a region
failure, setting a network component as primary at another region is a manual operation.

VMware, Inc. 114

Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-18. Example North-South Routing in a Multi-Region SDDC

Tier-0 Gateways
In NSX-T Federation, a Tier-0 gateway can span multiple regions. Each region contains a logical
unit of the Tier-0 gateway which is assigned to the edge cluster in the region and is configured to
interface with the data center top of rack switches in the region.

Each region that is in the scope of a Tier-0 gateway can be configured as primary or secondary.
Primary regions pass traffic for any other SDN service such as Tier-0 logical segments or Tier-1
gateways. Secondary regions route traffic locally but do not egress traffic outside the SDN or
advertise networks in the data center.

VMware, Inc. 115

Architecture and Design for a Virtual Infrastructure Workload Domain

When deploying an additional region, the Tier-0 gateway in the first region is extended to the new
region.

In this design, the Tier-0 gateway in each region is configured as primary. Although the Tier-1
gateway technically supports local-egress, the design does not recommend the use of local-
egress. Control ingress and egress traffic at the Tier-1 gateway level.

Table 2-106. Design Decisions on the Tier-0 Gateway Configuration for a Multi-Region SDDC

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-078 For a dual-region SDDC, n Supports ECMP north- Active-active Tier-0

extend the workload south routing on all gateways cannot provide
domain active-active Tier-0 nodes in the NSX-T stateful services such as
gateway to the second Edge cluster. NAT.
region. n Enables support for
cross-region Tier-1
gateways and cross-
region network
segments.

VCF-WLD-VI-SDN-079 For a dual-region SDDC, n In NSX-T Federation, None.

set the Tier-0 gateway as a Tier-0 gateway
primary in all regions. lets egress traffic
from connected Tier-1
gateways only in its
primary region.
n Local ingress
and egress traffic
is controlled
independently at
the Tier-1 level.
No segments are
provisioned directly to
the Tier-0 gateway.
n In a workload domain,
this architecture
improves flexibility for
unique use cases.
n A mixture of network
spans (isolated to a
region or spanning
multiple regions)
is enabled without
requiring additional
Tier-0 gateways and
hence edge nodes.
n If a region failure
occurs, the region-
specific networking in
the other regions will
remain available without
manual intervention.

VMware, Inc. 116

Architecture and Design for a Virtual Infrastructure Workload Domain

Each region has its own NSX-T Edge cluster with associated uplink VLANs for north-south traffic
flow for that data center. Similarly to the single-region design, each Tier-0 gateway unit peers with
the top of rack switches over eBGP.

The NSX-T Tier-0 gateway behaves like a standard eBGP router. By default, any routes that the
Tier-0 gateway learns from one eBGP neighbor are advertised to the other eBGP neighbours.
Because the underlying network connectivity between the regions is not an independent path,
but rather relies on the data center networks for connectivity, avoid advertising any learned
networks from one data center to another. To prevent route advertising, apply the no-export BGP
community to any routes learned from the top of rack switches in each data center.

Figure 2-19. BGP Peering to Top of Rack Switches

Region A Data Center Region B Data Center

Region A BGP ASN RTEP Region B BGP ASN
Tunnel

Data Center
Network
ToR ToR
Management Management
VLAN VLAN

eBGP Routes eBGP Routes

Inbound: Inbound:
Default Gateway Default Gateway
Region A local networks Region B local networks
Tag NO_EXPORT Tag NO_EXPORT

Outbound: Outbound:
NSX-T Edge NSX-T Edge
Tier-0 -No segments attached Tier-0 -No segments attached
Cluster Cluster
Tier-1- Global and local segments Tier-1- Global and local segments
Primary at this region Primary at this region

Edge Edge
Node iBGP - Inter-SR Routing Node
Tier-0 SR Tier-0 SR
Active/Active Active/Active

SDN G Tier-0
Active/Active
Primary/Primary

G Tier-1 G Tier-1 G Tier-1

Active/Standby NSX-T SDN
Active/Standby Active/Standby
Primary SDDC BGP ASN
Primary/Secondary Primary

Region A Only Multi-Region Segments Region B Only

Segments Egress through single site. Segments

Region A Region B

VMware, Inc. 117

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-107. Design Decisions on Routing Configuration for a Multi-Region SDDC

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-080 For a dual-region SDDC, n Enables the learning None.

from the global Tier-0 and advertising of
gateway, establish BGP routes between in the
neighbor peering to the second region.
top of rack switches in the n Facilitates the
second region. automated failover of
networks from the first
to the second region.

VCF-WLD-VI-SDN-081 For a dual-region SDCC, You disable re-advertising None.

on the global Tier-0 data center routes that
gateway, apply the no- are learned from the
export BGP community to first-region data center
all routes learned from networks to the second-
external neighbors. region data center or the
opposite.
By default, routes learned
from one autonomous
system over eBGP will
be advertised to another
autonomous system as
a valid path connected
over the NSX-T SDN.
Because the NSX-T SDN
in the first and second
regions does not have an
independent path between
those autonomous systems,
re-advertising data center
networks would give a
false indication of a valid,
independent path.

Tier-1 Gateways
A Tier-1 gateway can span one or more regions. Similarly to a Tier-0 gateway, you can configure a
region as primary or secondary for a Tier-1 gateway. The gateway passes ingress and egress traffic
for the logical segments connected to it.

Any logical segments connected to the Tier-1 gateway follow the span of the Tier-1 gateway. If the
Tier-1 gateway spans Region A and Region B, any segments connected to that gateway become
available in both regions. To define which regions a Tier-1 gateway spans, you associate the Tier-1
gateway with the edge cluster at each region.

Using a Tier-1 gateway enables more granular control on logical segments in the primary
and secondary regions. Design the region-specific and cross-region configuration of the Tier-1
gateways in the workload domain according to the requirements of the tenant workloads.

VMware, Inc. 118

Architecture and Design for a Virtual Infrastructure Workload Domain

A Tier-1 gateway advertises its networks to the connected region-specific unit of the Tier-0
gateway. In the case of primary-secondary location configuration, the Tier-1 gateway advertises
its networks only to the Tier-0 gateway unit in the region where the Tier-1 gateway is primary.
The Tier-0 gateway unit then re-advertises those networks to the data center in the regions where
that Tier-1 gateway is primary. During a region failover, the IT administrator must manually set the
Tier-1 gateway in Region B as primary. Then, networks become advertised through Region B. The
Tier-1 gateway does not advertise its attached networks through the secondary region.

Overlay Design for NSX-T Data Center for a Virtual Infrastructure Workload
Domain
As part of the overlay design, you determine the NSX-T Data Center configuration for handling
traffic between tenant workloads. You determine the configuration of vSphere Distributed Switch
and virtual segments on it, and of the transport zones.

This conceptual design for NSX-T provides the network virtualization design of the logical
components that handle the data to and from tenant workloads in the environment.

For a multi-region SDDC, in the recovery and additional regions, you replicate the design for the
first region of the SDDC.

ESXi Host Transport Nodes

An NSX-T transport node is a node that is capable of participating in an NSX-T overlay network.
The workload domain contains multiple ESXi hosts in a vSphere cluster to support workloads. You
register these ESXi hosts as NSX-T transport nodes so that networks and workloads on that host
can use the capabilities of NSX-T Data Center. During the preparation process, the native vSphere
Distributed Switch for the workload domain is extended with NSX-T capabilities.

VMware, Inc. 119

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-108. Design Decisions on ESXi Host Transport Nodes

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-082 Enable all ESXi hosts in the Enables distributed routing, None.
workload domain as NSX-T logical segments, and
transport nodes. distributed firewall.

VCF-WLD-VI-SDN-083 Configure each ESXi host n Enables the You must configure each
as a transport node participation of ESXi transport node with an
without using transport hosts and the virtual uplink profile individually.
node profiles. machines on them in
NSX-T overlay and
VLAN networks.
n Transport node profiles
can only be applied
at the cluster
level. Because in
an environment with
multiple availability
zones each availability
zone is connected to a
different set of VLANs,
you cannot use a
transport node profile.

Virtual Switches
NSX-T segments are logically abstracted segments to which you can connect tenant workloads.
A single segment is mapped to a unique Geneve segment that is distributed across the ESXi
hosts in a transport zone. The segment supports line-rate switching in the ESXi host without the
constraints of VLAN sprawl or spanning tree issues.

Consider the following limitations of distributed switches:

n Distributed switches are manageable only when the vCenter Server instance is available. You
can consider vCenter Server a Tier-1 application.

n Distributed switches with NSX-T capabilities are manageable only when the vCenter Server
instance and NSX-T Manager cluster is available. You can consider vCenter Server and NSX-T
Manager as Tier-1 applications.

n N-VDS instances are manageable only when the NSX-T Manager cluster is available. You can
consider the NSX-T Manager cluster as a Tier-1 application.

VMware, Inc. 120

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-109. Design Decision on Virtual Switches for NSX-T Data Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-084 Use a vSphere Distributed n Use the existing Management occurs jointly
Switch for the shared edge vSphere Distributed from the vSphere Client to
and workload cluster that Switch. NSX-T Manager. However,
is enabled for NSX-T Data n Provides NSX-T logical you must perform all
Center. segment capabilities to network monitoring in
support advanced use the NSX-T Manager user
cases. interface or another
solution.
To use features such
as distributed routing,
tenant workloads must
be connected to NSX-T
segments.

Configuration of the vSphere Distributed Switch with NSX-T

The shared edge and workload cluster in the workload domain uses a single vSphere Distributed
Switch with a configuration for system traffic types, NIC teaming, and MTU size. See vSphere
Networking Design for a Virtual Infrastructure Workload Domain.

To support traffic uplink and overlay traffic for the NSX-T Edge nodes for the workload domain,
you must create several port groups on the vSphere Distributed Switch for the workload domain.
The VMkernel adapter for the Host TEP is connected to the host overlay VLAN but does not
require a dedicated port group on the distributed switch. The VMkernel network adapter for Host
TEP is automatically created when you configure the ESXi host as a transport node.

NSX-T Edge appliances and the VMkernel adapter for Host Overlay be connected to different
VLANs and subnets. The VLAN IDs for the NSX-T Edge nodes are mapped to the VLAN trunk port
groups sfo-w01-cl01-vds01-pg-uplink01 and sfo-w01-cl01-vds01-pg-uplink02 on the host.

Table 2-110. vSphere Distributed Switch Configuration for the Workload Domain
Number of
Physical NIC
Switch Name Type Function Ports Teaming Policy MTU

sfo-w01-cl01- vSphere n Management 2 n Load balance 9000

vds01 Distributed n vSphere source for the
Switch 7.0 vMotion ESXi traffic

n vSAN n Failover
order for the
n NFS
NSX-T Edge
n Host Overlay
uplinks
n Edge Uplinks
and Overlay -
VLAN
Trunking

VMware, Inc. 121

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-111. sfo-w01-cl01-vds01 Switch Configuration Per Physical NIC

vmnic Function Connected to

0 Uplink Top of rack switch 1

1 Uplink Top of rack switch 2

Table 2-112. Segments on sfo-w01-cl01-vds01 in a Single Availability Zone

Segment Name Type Purpose

sfo01-w01-cl01-vds01-pg-mgmt VLAN Management traffic

sfo01-w01-cl01-vds01-pg-vmotion VLAN vSphere vMotion traffic

sfo01-w01-cl01-vds01-pg-vsan VLAN vSAN traffic

sfo-w01-cl01-vds01-pg-uplink01 VLAN Trunk Edge node overlay and uplink traffic

to the first top of rack switch

sfo-w01-cl01-vds01-pg-uplink02 VLAN Trunk Edge node overlay and uplink traffic

to the second top of rack switch

sfo-w01-cl01-vds01-pg-nfs VLAN NFS traffic

auto created (Host TEP) - Host overlay

auto created (Host TEP) - Host overlay

auto created (Hyperbus) - -

In a deployment that has multiple availability zones, you must provide new or extend existing
networks.

Table 2-113. Additional Segments on sfo-w01-cl01-vds01 in Multiple Availability Zones

Segment Name Type Availability Zone Purpose

az2_sfo01-w01-cl01-vds01- VLAN Availability Zone 2 Management traffic in

pg-mgmt Availability Zone 2

az2_sfo01-w01-cl01-vds01- VLAN Availability Zone 2 vSphere vMotion traffic in

pg-vmotion Availability Zone 2

az2_sfo01-w01-cl01-vds01- VLAN Availability Zone 2 vSAN traffic in Availability

pg-vsan Zone 2

az2_sfo01-w01-cl01-vds01- VLAN Availability Zone 2 -

pg-nfs

auto created (Host TEP) - - Host overlay

auto created (Host TEP) - - Host overlay

auto created (Hyperbus) - - -

VMware, Inc. 122

Architecture and Design for a Virtual Infrastructure Workload Domain

Geneve Overlay
Geneve provides the overlay capability in NSX-T to create isolated, multi-tenant broadcast
domains across data center fabrics, and enables customers to create elastic, logical networks that
span physical network boundaries.

The first step in creating these logical networks is to isolate and pool the networking resources.
By using the Geneve overlay, NSX-T isolates the network into a pool of capacity and separates the
consumption of these services from the underlying physical infrastructure. This model is similar to
the model vSphere uses to abstract compute capacity from the server hardware to create virtual
pools of resources that can be consumed as a service. You can then organize the pool of network
capacity in logical networks that are directly attached to specific applications.

Geneve is a tunneling mechanism which provides extensibility while still using the offload
capabilities of NICs for performance improvement.

Geneve works by creating Layer 2 logical networks that are encapsulated in UDP packets. A
Segment ID in every frame identifies the Geneve logical networks without the need for VLAN tags.
As a result, many isolated Layer 2 networks can coexist on a common Layer 3 infrastructure using
the same VLAN ID.

In the vSphere architecture, the encapsulation is performed at ESXi host TEP VMkernel adapter
and before being sent on physical network, making the Geneve overlay transparent to both the
guest virtual machines and the underlying Layer 3 network. The Tier-0 Gateway performs gateway
services between overlay and non-overlay hosts, for example, a physical server or the Internet
router. The NSX-T Edge node translates overlay segment IDs to VLAN IDs, so that non-overlay
hosts can communicate with virtual machines on an overlay network.

The NSX-T Edge cluster hosts all NSX-T Edge node instances that connect to the corporate
network for secure and centralized network administration.

Table 2-114. Design Decisions on Geneve Overlay

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-085 To provide virtualized n Creates isolated, multi- Requires configuring

network capabilities to tenant broadcast transport networks with an
tenant workloads, use domains across data MTU size of at least 1700
overlay networks with center fabrics to deploy bytes.
NSX-T Edge nodes and elastic, logical networks
distributed routing. that span physical
network boundaries.
n Enables advanced
deployment topologies
by introducing Layer
2 abstraction from the
data center networks.

VMware, Inc. 123

Architecture and Design for a Virtual Infrastructure Workload Domain

Transport Zones
Transport zones determine which hosts can participate in the use of a particular network. A
transport zone identifies the type of traffic, VLAN or overlay, and the vSphere Distributed Switch
name. You can configure one or more VLAN transport zones and a single overlay transport zone
per virtual switch. A transport zone does not represent a security boundary.

Figure 2-20. Transport Zone Design

NSX-T ESXi Hosts

Edge Nodes

Edge Edge
Node 1 Node 2 ESXi

vSphere Distributed
N-VDS
Switch with NSX-T

Overlay Transport Zone

sfo-w01-tz-overlay01

VLAN Transport Zone

VLAN Transport Zone
sfo-w01-tz-vlan01
sfo-w01-tz-vlane01
(Optional - Workload
(Edge Uplinks to ToR)
VLANs)

Table 2-115. Design Decision on the Transport Zone Configuration for NSX-T Data Center

Decision ID Design Decision Design Implication Design Justification

VCF-WLD-VI-SDN-086 Create a single overlay n Ensures that overlay None.

transport zone for all segments are
overlay traffic across the connected to an NSX-
workload domain and NSX- T Edge node for
T Edge nodes. services and north-
south routing.
n Ensures that all
segments are available
to all ESXi hosts and
NSX-T Edge nodes
configured as transport
nodes.

VCF-WLD-VI-SDN-087 Create a single VLAN Ensures that uplink VLAN If VLAN segments are
transport zone for uplink segments are configured on needed on hosts, you
VLAN traffic that is applied the NSX-T Edge transport must create another VLAN
only to NSX-T Edge nodes. nodes. transport zone for the host
transport nodes only.

VMware, Inc. 124

Architecture and Design for a Virtual Infrastructure Workload Domain

Uplink Policy for ESXi Host Transport Nodes

Uplink profiles define policies for the links from ESXi hosts to NSX-T segments or from NSX-T
Edge appliances to top of rack switches. By using uplink profiles, you can apply consistent
configuration of capabilities for network adapters across multiple ESXi hosts or NSX-T Edge
nodes.

Uplink profiles can use either load balance source or failover order teaming. If using load balance
source, multiple uplinks can be active. If using failover order, only a single uplink can be active.

Table 2-116. Design Decisions on the Uplink Profile for ESXi Transport Nodes

Decision ID Design Decision Decision Justification Decision Implication

VCF-WLD-VI-SDN-088 Create an uplink profile with For increased resiliency and None.
the load balance source performance, supports the
teaming policy with two concurrent use of both
active uplinks for ESXi physical NICs on the ESXi
hosts. hosts that are configured as
transport nodes.

Replication Mode of Segments

The control plane decouples NSX-T Data Center from the physical network. The control plane
handles the broadcast, unknown unicast, and multicast (BUM) traffic in the virtual segments.

The following options are available for BUM replication on segments.

Table 2-117. BUM Replication Modes of NSX-T Segments

BUM Replication Mode Description

Hierarchical Two-Tier The ESXi host transport nodes are grouped according
to their TEP IP subnet. One ESXi host in each subnet
is responsible for replication to an ESXi host in another
subnet. The receiving ESXi host replicates the traffic to the
ESXi hosts in its local subnet.
The source ESXi host transport node knows about the
groups based on information it has received from the NSX-
T Controller. The system can select an arbitrary ESXi host
transport node as the mediator for the source subnet if the
remote mediator ESXi host node is available.

Head-End In this mode, the ESXi host transport node at the origin
of the frame to be flooded on a segment sends a copy to
every other ESXi host transport node that is connected to
this segment.

VMware, Inc. 125

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-118. Design Decisions on Segment Replication Mode

Decision ID Design Decision Design Justification Design Implications

VCF-WLD-VI-SDN-089 Use hierarchical two-tier Hierarchical two-tier None.

replication on all NSX-T replication is more efficient
overlay segments. by reducing the number of
ESXi hosts the source ESXi
host must replicate traffic
to.

Virtual Network Segment Design for NSX-T for a Virtual Infrastructure Workload
Domain
Applications that are deployed on top of the workload domain can use a pre-defined configuration
of NSX-T virtual network segments.

NSX-T segments provide flexibility for workload placement by removing the dependence on
traditional physical data center networks. This approach also improves security and mobility of
the applications and reduces the integration effort with existing customer networks.

Table 2-119. Design Decisions on Virtual Network Segments in NSX-T Data Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-090 Create one or more region- Enables workload mobility Each NSX-T virtual network
specific NSX-T virtual within the data center segment requires a unique
network segments for without complex physical IP address space.
workloads that are assigned network configuration.
to a specific region.

VCF-WLD-VI-SDN-091 Create one or more Enables workload Each NSX-T virtual network
cross-region NSX-T virtual mobility without complex segment requires a unique
network segments for physical network IP address space.
workloads which require configuration.Workloads
mobility between regions. must be easily portable
between regions without
requiring reconfiguration

With NSX-T Federation, an NSX-T virtual segment can span multiple NSX-T instances and regions.
A single network segment can be available in different physical regions over the NSX-T SDN for IP
mobility of workloads.

VMware, Inc. 126

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-120. Design Decisions on Virtual Network Segments for a Multi-Region SDDC

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-092 For a dual-region SDDC, Enables management The NSX-T virtual network
extend the cross-region workload mobility without segment requires a unique
NSX-T virtual network complex physical network IP address space.
segments to Region B configuration.
for workload components Workloads are easily
which require mobility portable between
between regions. regions without requiring
reconfiguration.

VCF-WLD-VI-SDN-093 For a dual-region SDDC, Enables workload mobility Each NSX-T virtual network
in each additional region, within the data center segment requires a unique
create additional region- without complex physical IP address space.
specific NSX-T virtual network configuration.
network segments for Each region should have
workload components that network segments to
are allocated in a specific support workloads which
region. are isolated to that region.

VCF-WLD-VI-SDN-094 In each additional region, Configures region-specific Might require an individual

connect or migrate region- network segments at Tier-1 gateway for region-
specific NSX-T virtual required sites only. specific segments.
network segments to the
Tier-1 gateways that route
east-west traffic in the
region.

Information Security and Access Design for NSX-T for a Virtual Infrastructure
Workload Domain
You design authentication access, controls, and certificate management for the NSX-T Data
Center instance in the workload domain according to industry standards and the requirements
of your organization.

Identity Management
Users can authenticate to NSX-T Manager from several sources. Role-based access control is not
available with local user accounts.

n Local user accounts

n Active Directory by using LDAP

n Active Directory by using Workspace ONE Access

n Principal identity

VMware, Inc. 127

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-121. Design Decisions on Identity Management in NSX-T Data Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-095 Limit the use of local n Local accounts are not You must use Active
accounts. user specific and do not Directory for user accounts.
offer complete auditing
from solutions back to
users.
n Local accounts do
not provide full role-
based access control
capabilities.

VCF-WLD-VI-SDN-096 Enable NSX-T Manager n Provides integration You must have the
integration with your with Active Directory region-specific Workspace
corporate identity source for role-based access ONE Access deployed
by using the region-specific control. You can before configuring role-
Workspace ONE Access introduce authorization based access in NSX-T
instance. policies by assignment Manager.
of organization and
cloud services roles
to enterprise users
and groups defined in
your corporate identity
source.
n Simplifies deployment
by consolidating
the Active Directory
integration for the
SDDC in single
component, that is,
Workspace ONE
Access.

VCF-WLD-VI-SDN-097 Use Active Directory groups n Centralizes role-based n You must create
to grant privileges to roles access control by the role configuration
in NSX-T Data Center. mapping roles in NSX- outside of the SDDC
T Data Center to Active stack.
Directory groups. n You must set the
n Simplifies user appropriate directory
management. synchronization interval
in Workspace ONE
Access to ensure that
changes meet your
recoverability SLAs.

VCF-WLD-VI-SDN-098 Create an NSX-T Provides administrator You must maintain the life
Enterprise Admin access to the NSX-T cycle and availability of
group rainpole.io\ug-nsx- Manager user interface. the Active Directory group
enterprise-admins in Active outside of the SDDC stack.
Directory and map it to the
Enterprise Administrator
role in NSX-T Data Center.

VMware, Inc. 128

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-121. Design Decisions on Identity Management in NSX-T Data Center (continued)

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-099 Create an NSX-T Auditor Provides read-only access You must maintain the life
group rainpole.io\ug-nsx- account to NSX-T Data cycle and availability of
auditors in Active Directory Center. the Active Directory group
and map it to the Auditor outside of the SDDC stack.
role in NSX-T Data Center.

VCF-WLD-VI-SDN-100 Create more Active Each organization has You must maintain the life
Directory groups and map its own internal business cycle and availability of
them to roles in NSX- processes. You evaluate the Active Directory group
T Data Center according the role separation needs outside of the SDDC stack.
to the business and in your business and
security requirements of implement mapping from
your organization. individual user accounts to
Active Directory groups and
roles in NSX-T Data Center.

VCF-WLD-VI-SDN-101 Grant administrators access Administrators interact with None.

to both the NSX-T Manager NSX-T Data Center by using
user interface and its its user interface and API.
RESTful API endpoint.

VMware, Inc. 129

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-122. Design Decisions on Password Management and Account Lockout for NSX-T Data
Center

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-102 Configure the passwords Aligns with the industry You must run console
for CLI access to NSX- standard across your commands on the NSX-T
T Manager for the organization. Manager appliances.
root, admin, and audit
users, and account lockout
behavior for CLI according
to the industry standard for
security and compliance of
your organization.

VCF-WLD-VI-SDN-103 Configure the passwords Aligns with the industry You must run console
for access to the NSX- standard across your commands on the NSX-T
T Edge nodes for the organization. Edge appliances.
root, admin, and audit
users, and account lockout
behavior for CLI according
to the industry standard for
security and compliance of
your organization.

VCF-WLD-VI-SDN-104 Configure the passwords Aligns with the industry You must run console
for access to the NSX- standard across your commands on the NSX-T
T Manager user interface organization. Manager appliances.
and RESTful API or the
root, admin, and audit
users, and account lockout
behavior for CLI according
to the industry standard for
security and compliance of
your organization.

Password Management and Account Lockout Behavior for NSX-T Global Manager
The version of SDDC Manager in this design does not support password rotation for the NSX-T
Global Manager appliances. All password change operations must be done manually.

VMware, Inc. 130

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-123. Design Decisions on Password Management and Account Lockout for NSX-T Global
Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-105 For a dual-region If repeated authentication You must call the NSX-T
SDDC, add the NSX-T failures on the NSX-T Local Manager API to add the
Global Manager to the Manager occur, such as IP address of the NSX-T
lockout_immune_addresses password mismatch after a Global Manager on the
list on all NSX-T Local password change, the NSX- NSX-T Local Manager.
Manager it is connected to. T Local Manager denies
requests from that IP
address for a period of time
even after the administrator
provides the correct
password. Adding the NSX-
T Global Manager to the
lockout_immune_addresses
list on an NSX-
T Local Manager
minimizes disruption in
communication caused by
a password mismatch or
update.

VCF-WLD-VI-SDN-106 For dual-region SDDC, Because the NSX-T Global The administrator must
establish an operations Manager communicates establish and follow an
practice to capture and with the NSX-T Local operational practice by
update the admin password Manager by using the using a runbook or
on the NSX-T Global admin account, ensures automated process to
Manager appliance every connectivity between the ensure that the admin
time you perform rotation NSX-T Global Manager and password is updated.
of the admin password on the connected NSX-T Local
the NSX-T Local Manager Manager instances b
appliance. If an authentication failure
between the NSX-T Global
Manager and NSX-T Local
Manager occurs, objects
that are created from the
NSX-T Global Manager will
not be propagated on to
the SDN.

Certificate Management
Access to all NSX-T Manager interfaces must use a Secure Sockets Layer (SSL) connection. By
default, NSX-T Manager uses a self-signed SSL certificate. This certificate is not trusted by end-
user devices or Web browsers.

As a best practice, replace self-signed certificates with certificates that are signed by a third-party
or enterprise Certificate Authority (CA).

VMware, Inc. 131

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-124. Design Decisions on Certificate Management in NSX-T Manager

Decision ID Design Decision Design Implication Design Justification

VCF-WLD-VI-SDN-107 Replace the default self- Ensures that the Replacing the default
signed certificate of the communication between certificates with trusted CA-
NSX-T Manager instance NSX-T administrators and signed certificates from a
for the workload domain the NSX-T Manager certificate authority might
with a certificate that is instance is encrypted by increase the deployment
signed by a third-party using a trusted certificate. preparation time because
certificate authority. you must generate and
submit certificates requests.

VCF-WLD-VI-SDN-108 Use a SHA-2 algorithm The SHA-1 algorithm is Not all certificate authorities
or stronger when signing considered less secure and support SHA-2.
certificates. has been deprecated.

VCF-WLD-VI-SDN-109 Use SDDC Manager for Ensures consistent life None

NSX-T Manager certificate cycle management across
life cycle management. management components
in the SDDC.

Certificate Management in a Multi-Region SDDC

The version of SDDC Manager in this design does not support certificate replacement for NSX-T
Global Manager appliances. When the certificate of the NSX-T Local Manager cluster is replaced,
you must update the thumbprint of the new certificate on the connected NSX-T Global Manager.

VMware, Inc. 132

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-125. Design Decisions on Certificate Management in NSX-T Global Manager

Decision ID Design Decision Design Justification Design Implication

VCF-WLD-VI-SDN-110 For a dual-region SDDC, Ensures that the Replacing the default
replace the default self- communication between certificates with trusted CA-
signed certificate of the NSX-T administrators and signed certificates from a
NSX-T Global Manager the NSX-T Global Manager certificate authority might
instance for the workload instance is encrypted by increase the deployment
domain with a certificate using a trusted certificate. preparation time because
that is signed by a third- you must generate and
party certificate authority. submit certificates requests.

VCF-WLD-VI-SDN-111 For a dual-region SDDC, Ensures secured The administrator must

establish an operations connectivity between the establish and follow an
practice to capture and NSX-T Manager instances. operational practice by
update on the NSX-T Global Each certificate has its using a runbook or
Manager the thumbprint of own unique thumbprint. automated process to
the NSX-T Local Manager The NSX-T Global ensure that the thumbprint
certificate every time the Manager stores the unique up-to-date.
certificate is updated by thumbprint of the NSX-T
using SDDC Manager. Local Manager instances for
enhanced security.
If an authentication failure
between the NSX-T Global
Manager and NSX-T Local
Manager occurs, objects
that are created from the
NSX-T Global Manager will
not be propagated on to
the SDN.

Shared Storage Design for a Virtual Infrastructure Workload Domain

The shared storage design includes the design for VMware vSAN storage.

Follow these guidelines when designing shared storage for your environment.

n Optimize the storage design to meet the diverse needs of applications, services,
administrators, and users.

n Strategically align business applications and the storage infrastructure to reduce costs, boost
performance, improve availability, provide security, and enhance functionality.

n Provide multiple tiers of storage to match application data access to application requirements.

VMware, Inc. 133

Architecture and Design for a Virtual Infrastructure Workload Domain

n Design each tier of storage with different performance, capacity, and availability
characteristics. Because not every application requires expensive, high-performance, highly
available storage, designing different storage tiers reduces cost.

n Logical Design for Shared Storage for a Virtual Infrastructure Workload Domain
Either dedicated edge clusters or shared edge and workload clusters can use vSAN, NFS,
®
VMware vSphere Virtual Volumes™, or Fibre Channel storage as principle storage. No
specific guidance is given as user workloads and other factors determine storage type and
SLA for user workloads.

n Deployment Specification for Shared Storage for a Virtual Infrastructure Workload Domain
The shared storage design includes determining the physical storage infrastructure required
for using VMware vSAN and the policy configuration for delivering reliable storage service to
the SDDC tenant workloads.

n Network Design for Shared Storage for a Virtual Infrastructure Workload Domain
In the network design for shared storage in the workload domain, you determine the network
configuration for vSAN traffic.

Logical Design for Shared Storage for a Virtual Infrastructure Workload Domain
Either dedicated edge clusters or shared edge and workload clusters can use vSAN, NFS, VMware
®
vSphere Virtual Volumes™, or Fibre Channel storage as principle storage. No specific guidance is
given as user workloads and other factors determine storage type and SLA for user workloads.

Figure 2-21. Logical Storage Design

Management Cluster Shared Edge and Workload Cluster

Virtual Virtual Virtual Tenant n

Appliance Appliance Appliance

Tenant 1

Virtual Virtual Virtual

Appliance Appliance Appliance
APP APP APP
OS OS OS

ESXi Host ESXi Host

Principal Datastore Supplemental Datastores Principal Datastore Supplemental Datastores

Mgmt Backups Templates Workloads Workloads Workloads

VMs and Logs SLA 1 SLA 2 SLA N

Software-Defined Storage vSAN, vVols, NFS, or VMFS on FC vVols, NFS, VMFS on iSCSI, and VMFS on FC

Policy-Based Storage Management

Virtualized Data Services
Hypervisor Storage Abstraction

vSAN NAS

Physical Disks Physical Disks

FLASH FLASH FLASH FLASH FLASH FC15K FC10K SATA

VMware, Inc. 134

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-126. vSAN Logical Design

Single Region Multiple Availability Zones Multiple Regions

n vSAN clusters in the workload n Shared edge and workload cluster In each region:
domain is a vSAN stretched cluster n vSAN clusters in the shared edge
n Four ESXi hosts per vSAN cluster n Eight ESXi hosts in the stretched and workload cluster
n All-flash vSAN configuration cluster n Four ESXi hosts per vSAN cluster

Four ESXi hosts in each availability n All-flash vSAN configuration

zone
n vSAN witness appliance in a third
location to the SDDC
n All-flash configuration

Deployment Specification for Shared Storage for a Virtual Infrastructure

Workload Domain
The shared storage design includes determining the physical storage infrastructure required for
using VMware vSAN and the policy configuration for delivering reliable storage service to the
SDDC tenant workloads.

n Shared Storage Platform for a Virtual Infrastructure Workload Domain

For principal storage, you can select between traditional storage, VMware vSphere Virtual
Volumes, and VMware vSAN.

n vSAN Physical Design for a Virtual Infrastructure Workload Domain

This design uses VMware vSAN to implement software-defined storage as the primary
storage type for the shared edge and workload cluster. By using vSAN, you have a high
level of control over the storage subsystem.

n vSAN Deployment Specification for a Virtual Infrastructure Workload Domain

When determining the vSAN deployment specification, you decide on the datastore size,
number of ESXi hosts per vSphere Cluster, number of disk groups per ESXi host, and the
vSAN policy.

Shared Storage Platform for a Virtual Infrastructure Workload Domain

For principal storage, you can select between traditional storage, VMware vSphere Virtual
Volumes, and VMware vSAN.

Traditional Storage

Fibre Channel and NFS are applicable options for virtual machines, however they are not used
in this design. iSCSI storage is currently not supported as an option for principal storage for
workload domains.

VMware vSAN Storage

VMware, Inc. 135

Architecture and Design for a Virtual Infrastructure Workload Domain

vSAN is a software-based distributed storage platform that combines the compute and
storage resources of VMware ESXi hosts. When you design and size a vSAN cluster, host
hardware choices can be more limited compared to traditional storage.

vSphere Virtual Volumes

vSphere Virtual Volumes is an applicable option when using storage arrays which support the
vSphere Virtual Volume feature.
Traditional Storage and vSAN Storage
Fibre Channel and NFS are mature and applicable options to support workload needs.

Your decision to implement one technology or another can be based on performance and
functionality, and on other considerations, for example:

n The current in-house expertise and installation base in your organization.

n The cost, including both capital and long-term operational expenses.

n The current relationship of your organization with a storage vendor.

vSAN is a software-based distributed storage platform that combines the compute and storage
resources of ESXi hosts. It provides a simple storage management experience for the user.
However, you must carefully consider supported hardware options when sizing and designing
a vSAN cluster.
Storage Type Comparison
ESXi hosts support different storage types. Each storage type supports different vSphere features.

Table 2-127. Shared Storage Supported by ESXi Hosts

Technology Protocols Transfers Interface

Fibre Channel Fibre Channel/SCSI Block access of data/LUN Fibre Channel HBA

NAS IP/NFS File (no direct LUN access) Network adapter

vSAN IP Block access of data Network adapter

vSphere Virtual Volumes Fibre Channel/iSCSI/NFS Block access of data/File Fibre Channel HB/ Network
Adapter

Table 2-128. vSphere Features Supported by Storage Type

Raw Device Application or vSphere HA Storage APIs
vSphere Mapping Block-Level and vSphere Data
Type vMotion Datastore (RDM) Clustering DRS Protection

Fibre Channel Yes VMFS Yes Yes Yes Yes

NFS Yes NFS No No Yes Yes

VMware, Inc. 136

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-128. vSphere Features Supported by Storage Type (continued)

Raw Device Application or vSphere HA Storage APIs
vSphere Mapping Block-Level and vSphere Data
Type vMotion Datastore (RDM) Clustering DRS Protection

vSAN Yes vSAN No Yes Yes Yes

vSphere Yes vVOL No Yes Yes Yes

Virtual
Volumes

Logical Design for Shared Storage

This vSAN design is limited to the shared edge and workload cluster in the workload domain. The
design uses the default storage policy to achieve redundancy and performance within the vSAN
Cluster.

Table 2-129. Design Decisions on Storage Type

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-STO-001 When using a single By using vSAN as the Minimizes storage platform
availability zone for the primary shared storage complexity by standardizing
shared edge and workload solution, you can take on a single type.
cluster, use vSAN for advantage of more cost-
shared storage. effective local storage.

SDDC-WLD-VI-STO-002 In all clusters, ensure that at If a datastore runs out Monitoring and capacity
least 20% of free space is of free space, applications management must be
always available on all non- and services in the SDDC, proactive operations.
vSAN datastores. including but not limited to
the NSX Edge core network
services, the provisioning
portal, and backup, fail.

vSAN Physical Design for a Virtual Infrastructure Workload Domain

This design uses VMware vSAN to implement software-defined storage as the primary storage
type for the shared edge and workload cluster. By using vSAN, you have a high level of control
over the storage subsystem.

All functional testing and validation of the design is on vSAN. Although VMware Validated Design
uses vSAN, in particular for the clusters running tenant workloads, you can use any supported
storage solution. If you select a storage solution other than vSAN, consider that all the design,
deployment, and Day-2 guidance in VMware Validated Design applies under the context of
vSAN and adjust appropriately. Your storage design must match or exceed the capacity and
performance capabilities of the vSAN configuration in the design. For multiple availability zones,
the vSAN configuration includes vSAN stretched clusters.

vSAN is a hyper-converged storage software that is fully integrated with the hypervisor. vSAN
creates a cluster of local ESXi host hard disk drives and solid-state drives, and presents a
flash-optimized, highly resilient, shared storage datastore to ESXi hosts. By using vSAN storage
policies, you can control capacity, performance, and availability on a per virtual machine basis.

VMware, Inc. 137

Architecture and Design for a Virtual Infrastructure Workload Domain

vSAN Physical Requirements and Dependencies

The software-defined storage module has the following requirements and options.

Requirement Category Requirements

Number of hosts Minimum of three ESXi hosts providing storage resources

to the vSAN cluster.

vSAN configuration vSAN is configured as hybrid storage or all-flash storage.

n A vSAN hybrid storage configuration requires both
magnetic devices and flash caching devices.
n An all-flash vSAN configuration requires flash devices
for both the caching and capacity tiers.

Requirements for individual hosts that provide storage n Minimum of one flash device. The flash-based
resources cache tier must be sized to handle the anticipated
performance requirements. For sizing the vSAN
caching tier, see the Design Considerations for Flash
Caching Devices in vSAN in the VMware vSAN product
documentation.
n Minimum of two additional devices for capacity tier.
n RAID controller that is compatible with vSAN.
n Minimum 10 Gbps network for vSAN traffic.
n vSphere High Availability host isolation response set
to power off virtual machines. With this setting, you
prevent split-brain conditions if isolation or network
partition occurs. In a split-brain condition, the virtual
machine might be powered on by two ESXi hosts by
accident.

vSAN Hardware Considerations

While VMware supports building your own vSAN cluster from compatible components, vSAN
ReadyNodes are selected for this VMware Validated Design.

Build Your Own Use hardware from the VMware Compatibility Guide for
the following vSAN components:
n Flash-based drives
n Magnetic hard drives
n I/O controllers, including vSAN certified driver and
firmware combinations

Use VMware vSAN ReadyNodes A vSAN ReadyNode is a server configuration that is

validated in a tested, certified hardware form factor
for vSAN deployment, jointly recommended by the
server OEM and VMware. See the vSAN ReadyNode
documentation. The vSAN Compatibility Guide for
vSAN ReadyNodes documentation provides examples of
standardized configurations, including supported numbers
of VMs and estimated number of 4K IOPS delivered.

VMware, Inc. 138

Architecture and Design for a Virtual Infrastructure Workload Domain

I/O Controllers for vSAN

The I/O controllers are as important to a vSAN configuration as the selection of disk drives. vSAN
supports SAS, SATA, and SCSI adapters in either pass-through or RAID 0 mode. vSAN supports
multiple controllers per ESXi host.

n You select between single- and multi-controller configuration in the following way: Multiple
controllers can improve performance and mitigate a controller or SSD failure to a smaller
number of drives or vSAN disk groups.

n With a single controller, all disks are controlled by one device. A controller failure impacts all
storage, including the boot media (if configured).

Controller queue depth is possibly the most important aspect for performance. All I/O
controllers in the VMware vSAN Hardware Compatibility Guide have a minimum queue depth
of 256. Consider regular day-to-day operations and increase of I/O because of virtual machine
deployment operations, or re-sync I/O activity as a result of automatic or manual fault
remediation.

Table 2-130. Design Decisions on the vSAN I/O Controller Configuration

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-001 Ensure that the storage I/O Storage controllers with Limits the number of
controller that is running lower queue depths can compatible I/O controllers
the vSAN disk groups is cause performance and that can be used for
capable and has a minimum stability problems when storage.
queue depth of 256 set. running vSAN.
vSAN ReadyNode servers
are configured with the
right queue depths for
vSAN.

vSAN Flash Options

vSAN has two configuration options: all-flash and hybrid.

Hybrid Mode

In a hybrid storage architecture, vSAN pools server-attached capacity devices (in this case
magnetic devices) and flash-based caching devices, typically SSDs, or PCI-e devices, to create
a distributed shared datastore.

All-Flash Mode

All-flash storage uses flash-based devices (SSD or PCI-e) as a write cache while other flash-
based devices provide high endurance for capacity and data persistence.

VMware, Inc. 139

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-131. Design Decisions on the vSAN Mode

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-002 Configure vSAN in All-Flash n Provides support for All vSAN disks must be
mode in the shared edge vSAN deduplication and flash disks, which might
and workload cluster. compression. cost more than magnetic
n Meets the performance disks.
needs of the shared
edge and workload
cluster.
Using high speed magnetic
disks in a hybrid vSAN
configuration can provide
satisfactory performance
and is supported.

Sizing Storage
You usually base sizing on the requirements of the IT organization. However, this design provides
calculations that are based on a single-region implementation, and is then implemented on a
per-region basis. In this way, you can handle storage in a dual-region deployment that has failover
capabilities enabled.

This sizing is calculated according to a certain node configuration per region. Although VMware
Validated Design has enough memory capacity to handle N-1 host failures and uses thin-
provisioned swap for the vSAN configuration, the potential thin-provisioned swap capacity is
factored in the calculation.

Category Quantity Resource Type Consumption

Physical Infrastructure 4 Memory 1024 GB

(ESXi)

NSX-T Edge Appliances 2 Disk 400 GB

Swap 64 GB

Tenant Workloads 60 Disk 7200 GB

(example, 8 GB memory &
120 GB disk per VM)

Swap 480 GB

Total n 60 tenant workload Disk 7600 GB

VMs
Swap 544 GB
n 2 NSX-T Edge VMs
n 4 ESXi hosts Memory 1024 GB

The storage space that is required for the vSAN capacity tier according is worked out using
the following calculations. For sizing the vSAN caching tier, see Design Considerations for Flash
Caching Devices in vSAN in the VMware vSAN product documentation. For vSAN memory
consumption by ESXi hosts in the workload domain, see VMware Knowledge Base article 2113954.

VMware, Inc. 140

Architecture and Design for a Virtual Infrastructure Workload Domain

Derive the consumption of storage space by the management virtual machines according to the
following calculations. See vSAN Design and Sizing Guide.

7,600 GB Disk + 544 GB Swap = 8,144 GB Virtual Machine Raw Capacity Requirements

8,144 GB Virtual Machine Raw Capacity Requirements * 2 (FTT=1, RAID1) = 16,288 GB Final
Virtual Machine Raw Capacity Requirements

34 GB vSAN Memory Consumption + 16,288 GB VM Raw Capacity = 16,322 GB Total Raw Storage
Capacity

16,322 GB Total Raw Storage Capacity * 30% Slack Overhead * 20% Estimated Growth ≈ 24,485 GB
≈ 24 TB Raw Unformatted Storage Capacity (with 20% Growth Capacity)

24 TB Raw Unformatted Storage Capacity / 3 ESXi hosts ≈ 8 TB Final Raw Storage Capacity per
host (considering worst-case scenario with one host failure)

8 TB / 2 disk groups = 4 TB raw capacity per disk group

Table 2-132. Design Decisions on the vSAN Disk Configuration

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-003 Use a 600 GB or greater Provides enough cache for Larger flash disks can
flash-based drive for the both hybrid or all-flash increase initial host cost
cache tier in each disk vSAN configurations to
group. buffer I/O and ensure disk
group performance.

SDDC-WLD-VI-SDS-004 Have at least 4TB of Provides enough capacity None.

flash-based drives for the for the NSX-T Edge nodes
capacity tier in each disk and tenant workloads with
group. a minimum of 10% caching,
30% of overhead, and 20%
growth when the number of
primary failures to tolerate
is 1.

vSAN Hardware Considerations

While VMware supports building your own vSAN cluster from compatible components, vSAN
ReadyNodes are selected for this VMware Validated Design. See Table 1.

vSAN Deployment Specification for a Virtual Infrastructure Workload Domain

When determining the vSAN deployment specification, you decide on the datastore size, number
of ESXi hosts per vSphere Cluster, number of disk groups per ESXi host, and the vSAN policy.
vSAN Datastore Size
The size of the vSAN datastore depends on the requirements for the datastore. Consider cost
against availability to provide the appropriate sizing.

VMware, Inc. 141

Architecture and Design for a Virtual Infrastructure Workload Domain

As per the calculations in Sizing Storage, a minimum size is required to run the workloads and
infrastructure. If you plan to add more solutions or additions to this environment, you must
increase this size.

Table 2-133. Design Decisions on the vSAN Datastore

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-005 Provide the shared edge NSX-T Edge nodes and If you scale the
and workload cluster with a sample tenant workloads environment out with
minimum of 24 TB of raw require at least 8 TB of more workloads, additional
capacity for vSAN. raw storage (prior to FTT=1) storage is required in the
and 16 TB when using workload domain.
the default vSAN storage
policy.
By allocating at least 24
TB, initially there is 20%
free space that you can
use for additional tenant
workloads.

SDDC-WLD-VI-SDS-006 On all vSAN datastores, When vSAN reaches 80% Increases the amount of
ensure that at least 30% usage, a rebalance task available storage needed.
of free space is always is started which can be
available. resource-intensive.

Number of vSAN-enabled ESXi Hosts Per Cluster

The number of ESXi hosts in the cluster depends on these factors:

n Amount of available space on the vSAN datastore

n Number of failures you can tolerate in the cluster

For example, if the vSAN cluster has only 3 ESXi hosts, only a single failure is supported. If a
higher level of availability is required, you must add more hosts.

VMware, Inc. 142

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-134. Design Decision on the Cluster Size for vSAN

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-007 When using a single n Having 4 ESXi The availability

availability zone, the shared hosts addresses the requirements for the shared
edge and workload cluster availability and sizing edge and workload cluster
requires a minimum of requirements. might cause underutilization
4 ESXi hosts to support n You can take an of the cluster's ESXi hosts.
vSAN. ESXi host offline
for maintenance or
upgrades without
impacting the overall
vSAN cluster health.

SDDC-WLD-VI-SDS-008 When using two availability n Having 8 ESXi The capacity of the
zones, the shared edge and hosts addresses the additional 4 hosts is not
workload cluster requires a availability and sizing added to capacity of the
minimum of 8 ESXi hosts (4 requirements. cluster. They are only
in each availability zone) to n You can take an used to provide additional
support a stretched vSAN availability zone offline availability.
configuration. for maintenance or
upgrades without
impacting the overall
vSAN cluster health.

Number of Disk Groups Per ESXi Host

Disk group sizing is an important factor during volume design. The number of disk groups can
affect availability and performance. If more ESXi hosts are available in the cluster, more failures
are tolerated in the cluster. This capability adds cost because additional hardware for the disk
groups is required. More available disk groups can increase the recoverability of vSAN during a
failure. Consider these data points when deciding on the number of disk groups per ESXi host:

n Amount of available space on the vSAN datastore.

n Number of failures you can tolerate in the cluster.

n Performance required when recovering vSAN objects.

The optimal number of disk groups is a balance between hardware and space requirements for
the vSAN datastore. More disk groups increase space and provide higher availability. However,
adding disk groups can be cost-prohibitive.

Table 2-135. Design Decision on the Disk Groups per ESXi Host

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-009 Configure vSAN with a Reduces the size of the fault Multiple disks groups
minimum of two disk domain and spreads the I/O require more disks in each
groups per ESXi host. load over more disks for ESXi host.
better performance.

VMware, Inc. 143

Architecture and Design for a Virtual Infrastructure Workload Domain

vSAN Policy Design

After you enable and configure VMware vSAN, you can create storage policies that define the
virtual machine storage characteristics. Storage characteristics specify different levels of service
for different virtual machines.

The default storage policy tolerates a single failure and has a single disk stripe. Use the default
policy. If you configure a custom policy, vSAN should guarantee its application. However, if vSAN
cannot guarantee a policy, you cannot provision a virtual machine that uses the policy unless you
enable force provisioning.

Policy design starts with assessment of business needs and application requirements. Use cases
for VMware vSAN must be assessed to determine the necessary policies. Start by assessing the
following application requirements:

n I/O performance and profile of your workloads on a per-virtual-disk basis

n Working sets of your workloads

n Hot-add of additional cache (requires repopulation of cache)

n Specific application best practice (such as block size)

After assessment, configure the software-defined storage module policies for availability and
performance in a conservative manner so that space consumed and recoverability properties are
balanced. In many cases the default system policy is adequate and no additional policies are
required unless specific requirements for performance or availability exist.

A storage policy includes several attributes. You can use them alone or combine them to provide
different service levels. By using policies, you can customize any configuration according to the
business requirements of the consuming application.

Before making design decisions, understand the policies and the objects to which they can be
applied.

VMware, Inc. 144

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-136. VMware vSAN Policy Options

Capability Use Case Default Value Maximum Value Comments

Number of failures to Redundancy 1 3 A standard

tolerate RAID 1 mirrored
configuration that
provides redundancy
for a virtual machine
disk. The higher
the value, the more
failures can be
tolerated. For N
failures tolerated, N1
copies of the disk
are created, and
2N+1 ESXi hosts
contributing storage
are required.
A higher N value
indicates that more
replicas of virtual
machines are made,
which can consume
more disk space than
expected.

Number of disk Performance 1 12 A standard RAID 0

stripes per object stripe configuration
used to increase
performance for a
virtual machine disk.
This setting defines
the number of HDDs
on which each replica
of a storage object is
striped.
If the value is
higher than 1,
you can increase
performance.
However, an increase
in system resource
usage might also
result.

VMware, Inc. 145

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-136. VMware vSAN Policy Options (continued)

Capability Use Case Default Value Maximum Value Comments

Flash read cache Performance 0% 100% Flash capacity

reservation (%) reserved as read
cache for the storage
is a percentage of
the logical object size
that is reserved for
that object.
Use this setting for
workloads only if you
must address read
performance issues.
The downside of this
setting is that other
objects cannot use a
reserved cache.
Avoid using these
reservations unless it
is necessary because
unreserved flash is
shared fairly among
all objects.

VMware, Inc. 146

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-136. VMware vSAN Policy Options (continued)

Capability Use Case Default Value Maximum Value Comments

Object space Thick provisioning 0% 100% The percentage

reservation (%) of the storage
object that is thick-
provisioned when
creating a virtual
machine. The
remaining storage
capacity is thin-
provisioned.
This setting is useful if
an object will always
use a predictable
amount of storage,
cutting back on
repeatable disk
growth operations for
all but new or non-
predictable storage
use.

Force provisioning Override policy No - Forces provisioning

to occur even if the
currently available
cluster resources
cannot satisfy the
current policy.
Force provisioning
is useful during a
planned expansion
of the vSAN
cluster, during which
provisioning of virtual
machines must
continue.
vSAN automatically
tries to bring
the object into
compliance as
resources become
available.

If you do not specify a user-configured policy, vSAN uses a default system policy of 1 failure to
tolerate and 1 disk stripe for virtual disks and virtual disk snapshots. To ensure protection for
these critical virtual machine components, policy defaults for the VM namespace and swap are set
statically and are not configurable. Configure policies according to the business requirements of
the application. By using policies, vSAN can adjust the performance of a disk on the fly.

VMware, Inc. 147

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-137. VMware vSAN Object Policy Defaults

Object Policy Comments

Virtual machine namespace Failures-to-Tolerate: 1 Configurable. Changes are not

recommended.

Swap Failures-to-Tolerate: 1 Configurable. Changes are not

recommended.

Virtual disks User-Configured Storage Policy Can be any storage policy configured
on the system

Virtual disk snapshots Uses virtual disk policy Same as virtual disk policy by default.
Changes are not recommended.

If you do not specify a user-configured policy, vSAN uses a default system policy of 1 failure to
tolerate and 1 disk stripe for virtual disks and virtual disk snapshots. To ensure protection for
these critical virtual machine components, policy defaults for the VM namespace and swap are set
statically and are not configurable. Configure policies according to the business requirements of
the application. By using policies, vSAN can adjust the performance of a disk on the fly.

Table 2-138. Design Decisions on the vSAN Storage Policy

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-010 When using a single Provides the level of You might need additional
availability zone, use the redundancy that is needed policies for third-party VMs
default VMware vSAN in the shared edge and hosted in the shared
storage policy. workload cluster. edge and workload cluster
Provides the level of because their performance
performance that is enough or availability requirements
for NSX-T Edge appliances might differ from what
and tenant workloads. the default VMware vSAN
policy supports.

SDDC-WLD-VI-SDS-011 When using two availability Provides the necessary You might need additional
zones, add the following protection for virtual policies if third-party VMs
setting to the default vSAN machines in each are to be hosted in
storage policy: availability zone, with the the shared edge and
Secondary Failures to ability to recover from an workload cluster because
Tolerate = 1 availability zone outage. their performance or
availability requirements
might differ from what
the default VMware vSAN
policy supports.

VMware, Inc. 148

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-138. Design Decisions on the vSAN Storage Policy (continued)

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-012 When using two availability Fault Domains are mapped Additional raw storage is
zones, configure two to Availability Zones required when Secondary
fault domains, one for to provide logical host Failures to Tolerate and
each availability zone. separation and ensure a Fault Domains are enabled.
Assign each host to their copy of vSAN data is
respective availability zone always available even when
fault domain. an availability zone goes
offline.

SDDC-WLD-VI-SDS-013 Leave the default virtual Creates virtual swap files None.
machine swap file as a as a sparse object on
sparse object on VMware the vSAN datastore. Sparse
vSAN. virtual swap files only
consume capacity on vSAN
as they are accessed. As
a result, you can reduce
the consumption on the
vSAN datastore if virtual
machines do not experience
memory over-commitment
which requires the use of
the virtual swap file.

vSAN Witness
When using vSAN in a stretched cluster configuration, you must deploy a witness ESXi host on
a physical server or as a virtual appliance. The vSAN witness appliance contains a special ESXi
installation that provides quorum and tiebreaker services for stretched clusters. This appliance
must be deployed in a third location that is not local to the ESXi hosts on either side of the
stretched cluster.

VMware, Inc. 149

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-139. Resource Specification of a vSAN Witness Appliance

Appliance Size Supported Capacity Number of vCPUs Memory Storage

Tiny Supports up to 2 8 GB The appliance has

10 virtual machines three virtual disks.
and 750 witness n ESXi boot disk: 12
components GB HDD
n Caching device:
10 GB SSD
n Capacity device:
15 GB HDD

Medium Supports up to 2 16 GB The appliance has

500 virtual machines three virtual disks.
and 21,000 witness n ESXi boot disk: 12
components GB HDD
n Caching Device:
10 GB SSD
n Capacity Device:
350 GB HDD

Large Supports over 500 2 32 GB The appliance has

virtual machines five virtual disks.
and 45,000 witness n ESXi boot disk: 12
components GB HDD
n Caching device:
10 GB SSD
n Capacity devices:
3 x 350 GB HDD
each

VMware, Inc. 150

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-140. Design Decisions for vSAN Witness Appliance for Multiple Availability Zones
Design
Decision ID Design Decision Design Justification Implication

SDDC-WLD- When using two The witness appliance has these features. A third
VI-SDS-014 availability zones in a n Acts as a tiebreaker if network isolation between the physically-
single region tolopogy, availability zones occurs. separate
deploy a vSAN witness location is
n Hosts all required witness components to form the
appliance in a location required.
required RAID-1 configuration on vSAN, that is, each
that is not local to the Such a
data copy at a site while the witness is at the witness
ESXi hosts in any of the location must
site.
availability zones. have a
vSphere
environment
running to
host the
witness
appliance.

When using two None.

availability zones in a
dual regions tolopogy,
deploy a vSAN witness
appliance in Region B.

SDDC-WLD- Deploy a large-size A large-size witness appliance supports more than 500 The vSphere
VI-SDS-015 witness appliance. virtual machines which is required for high availability of environment
workloads that run in the SDDC. at the witness
location must
satisfy the
resource
requirements
of the witness
appliance.

Network Design for Shared Storage for a Virtual Infrastructure Workload

Domain
In the network design for shared storage in the workload domain, you determine the network
configuration for vSAN traffic.

When determining the network configuration, you must consider the overall traffic bandwidth and
decide how to isolate storage traffic.

n Consider how much replication and communication traffic is running between ESXi hosts and
storage arrays.

n The amount of storage traffic depends on the number of VMs that are running in the cluster,
and on how write- intensive the I/O is for the applications running in the VMs.

For information on the physical network setup for vSAN traffic, and other system traffic, see
Physical Network Infrastructure Design for a Virtual Infrastructure Workload Domain.

VMware, Inc. 151

Architecture and Design for a Virtual Infrastructure Workload Domain

For information on the virtual network setup for vSAN traffic, and other system traffic, see
Distributed Port Group and VMkernel Adapter Design for a Virtual Infrastructure Workload
Domain.

vSAN Network Design

The vSAN network design includes these components.

Table 2-141. Components of the vSAN Network Design

Design Component Description

Physical NIC speed For best and most predictable performance (IOPS) for the environment, this design uses a
minimum of a 10-GbE connection, with 25-GbE recommended, for use with vSAN all-flash
configurations.

VMkernel network The vSAN VMkernel network adapter on each ESXi host is created when you enable vSAN on
adapters for vSAN the cluster. Connect the vSAN VMkernel network adapters on all ESXi hosts in a cluster to a
dedicated distributed port group including also ESXi hosts that are not contributing storage
resources to the cluster.

VLAN All storage traffic should be isolated on its own VLAN. When a design uses multiple vSAN
clusters, each cluster should use a dedicated VLAN or segment for its traffic. This approach
increases security, prevents interference between clusters and helps with troubleshooting
cluster configuration.

Jumbo frames vSAN traffic can be handled by using jumbo frames. Use jumbo frames for vSAN traffic only if
the physical environment is already configured to support them, they are part of the existing
design, or if the underlying configuration does not create a significant amount of added
complexity to the design.

Virtual switch type vSAN supports vSphere Standard Switch or vSphere Distributed Switch. The benefit of
using vSphere Distributed Switch is that it supports Network I/O Control for prioritization of
bandwidth if contention occurs.

Table 2-142. Design Decisions on Virtual Switch Configuration for vSAN

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-SDS-016 Use the existing vSphere Provides guaranteed All traffic paths are shared
Distributed Switch instance performance for vSAN over common uplinks.
for the shared edge and traffic, if there is
workload cluster. network contention, by
using existing networking
components.

SDDC-WLD-VI-SDS-017 Configure jumbo frames on n Simplifies configuration Every device in the

the VLAN dedicated to because jumbo frames network must support
vSAN traffic. are also used jumbo frames.
to improve the
performance of vSphere
vMotion and NFS
storage traffic.
n Reduces the CPU
overhead resulting high
network usage.

VMware, Inc. 152

Architecture and Design for a Virtual Infrastructure Workload Domain

vSAN Witness Network Design

When using two availability zones, to be able to communicate to the vCenter Server instance,
connect the vSAN witness appliance for the workload domain to a network that is routed to the
management network of the management domain in Availability Zone 1.

VMware Validated Design uses vSAN witness traffic separation where you can use a VMkernel
adapter for vSAN witness traffic that is different from the adapter for vSAN data traffic. In this
design, you configure vSAN witness traffic in the following way:

n On each ESXi host in both availability zones, place the vSAN witness traffic on the
management VMkernel adapter.

n On the vSAN witness appliance, use the same VMkernel adapter for both management
and witness traffic. This VMkernel adapter is connected to a network that is routed to
the management networks of the management domain and the workload domain in both
availability zones.

For information about vSAN witness traffic separation, see vSAN Witness Traffic Separation on
VMware StorageHub.

Management Network

Routed to the management networks of the management domain and the workload domain in
both availability zones. Connect the first VMkernel adapter of the vSAN witness appliance to
this network. The second VMkernel adapter on the vSAN witness appliance is not used.

Place the following traffic on this network:

n Management traffic

To be able to communicate to the vCenter Server instance, the vSAN witness appliance for
the workload domain must be routed to the management network in Availability Zone 1.

VMware, Inc. 153

Architecture and Design for a Virtual Infrastructure Workload Domain

n vSAN witness traffic

Figure 2-22. vSAN Witness Network Design

Witness Site
Physical
Upstream
Router
Witness
Appliance

VLAN: lax-m01-cl01-vds01-pg-mgmt

Availability Zone 1 Availability Zone 2

Physical Workload Domain Physical
Upstream vCenter Server Upstream
Router sfo-w01-vc01. Router
sfo.rainpole.io ESXi ESXi ESXi ESXi ESXi ESXi ESXi ESXi
Host 1 Host 2 Host 3 Host 4 Host 1 Host 2 Host 3 Host 4

VLAN: sfo-m01-cl01-vds01-pg-mgmt
VLAN: sfo-w01-cl01-vds01-pg-mgmt

VLAN: az2_sfo-w01-cl01-vds01-pg-mgmt

VLAN: sfo-w01-cl01-vds01-pg-vsan VLAN: az2_sfo-w01-cl01-vds01-pg-vsan

Table 2-143. Design Decisions on the Network Configuration of the vSAN Witness Appliance for
Multiple Availability Zones
Design
Decision ID Design Decision Design Justification Implication

SDDC-WLD- In a single region Connects the witness appliance to the vCenter server The
VI-SDS-018 topology, connect the instance and ESXi hosts in Region A. management
first VMkernel adapter networks of
of the vSAN witness both the
appliance to the management
management network in and workload
the witness site. domains in
both
In a dual regions
availability
topology, connect the
zones must be
first VMkernel adapter
routed to the
of the vSAN witness
management
appliance to the
network in the
management network in
witness site.
Region
B management domain,
that is lax-m01-cl01-
vds01-pg-mgmt.

VMware, Inc. 154

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-143. Design Decisions on the Network Configuration of the vSAN Witness Appliance for
Multiple Availability Zones (continued)
Design
Decision ID Design Decision Design Justification Implication

SDDC-WLD- Configure the vSAN Separates the witness traffic from the vSAN data traffic. The
VI-SDS-019 witness appliance to Witness traffic separation provides the following benefits: management
use the first VMkernel n Removes the requirement to have static routes from networks for
adapter, that is the the vSAN networks in both availability zones to the both the
management Interface, witness site. management
for vSAN witness traffic. and workload
n Removes the requirement to have jumbo frames
domains in
enabled on the path between both availability zones
both
and the witness site because witness traffic can use a
availability
regular MTU size of 1500 bytes.
zones must be
routed to the
management
network in the
witness site.

SDDC-WLD- Place witness traffic Separates the witness traffic from the vSAN data traffic. The
VI-SDS-020 on the management Witness traffic separation provides the following benefits: management
VMkernel adapter of all n Removes the requirement to have static routes from networks for
the ESXi hosts in the the vSAN networks in both availability zones to the both the
workload domain. witness site. management
and workload
n Removes the requirement to have jumbo frames
domains in
enabled on the path between both availability zones
both
and the witness site because witness traffic can use a
availability
regular MTU size of 1500 bytes.
zones must be
routed to the
management
network in the
witness site.

SDDC-WLD- Allocate a statically Simplifies maintenance and tracking and implements a Requires
VI-SDS-021 assigned IP address DNS configuration. precise IP
and host name to the address
management adapter management.
of the vSAN witness
appliance.

VMware, Inc. 155

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-143. Design Decisions on the Network Configuration of the vSAN Witness Appliance for
Multiple Availability Zones (continued)
Design
Decision ID Design Decision Design Justification Implication

SDDC-WLD- Configure forward and Enables connecting the vSAN witness appliance to the You must
VI-SDS-022 reverse DNS records workload domain vCenter Server by FQDN instead of IP provide DNS
for the vSAN witness address. records for
appliance assigning the the vSAN
record to the child witness
domain for the region. appliance.

SDDC-WLD- Configure time Prevents any failures in the stretched cluster configuration n An
VI-SDS-023 synchronization by using that are caused by time mismatch between the vSAN operationa
an internal NTP time witness appliance and the ESXi hosts in both availability l NTP
for the vSAN witness zones and workload domain vCenter Server. service
appliance. must be
available
in the
environme
nt.
n All
firewalls
between
the vSAN
witness
appliance
and the
NTP
servers
must allow
NTP traffic
on the
required
network
ports.

vSphere with Tanzu Detailed Design for a Virtual Infrastructure

Workload Domain
vSphere with Tanzu transforms vSphere to a platform for running Kubernetes workloads natively
on the hypervisor layer. When enabled on a vSphere cluster, vSphere with Tanzu provides the
capability to run Kubernetes workloads directly on ESXi hosts and to create upstream Kubernetes
clusters in dedicated resource pools.

Logical Design for vSphere with Tanzu for a Virtual Infrastructure Workload
Domain
The vSphere with Tanzu design consists of multiple elements that enable a fully upstream-
compliant Kubernetes cluster in your SDDC.

VMware, Inc. 156

Architecture and Design for a Virtual Infrastructure Workload Domain

You enable and configure vSphere with Tanzu on your shared edge and workload cluster in the
workload domain. NSX-T Edge nodes provide load balancing, north-south connectivity, and all
required networking for the Kubernetes services. The ESXi hosts in the shared edge and workload
cluster are prepared as NSX-T Data Center transport nodes to provide distributed routing and
firewall services to your tenant workloads.

The Kubernetes environment consists of multiple elements.

Supervisor Cluster

The Supervisor Cluster is a special kind of Kubernetes cluster that uses ESXi hosts as worker
nodes instead of Linux.

Registry Service

The Registry Service is a locally managed deployment of a VMware Harbor registry in the
Supervisor Cluster.

Tanzu Kubernetes Grid Service

The Tanzu Kubernetes Grid Service deploys Tanzu Kubernetes clusters as Photon OS
appliances on top of the Supervisor Cluster.

Deployment Specification of vSphere with Tanzu for a Virtual Infrastructure

Workload Domain
When vSphere with Tanzu is enabled on a vSphere cluster, it creates a Kubernetes control plane
inside the hypervisor layer. This layer contains multiple objects that enable the capability to run
Kubernetes workloads within ESXi.

Deployment Model for vSphere with Tanzu for a Virtual Infrastructure Workload Domain
You determine the different services to use to achieve a fully upstream-compliant Kubernetes
cluster in your SDDC.
Supervisor Cluster
A cluster that is enabled for vSphere with Tanzu is called a Supervisor Cluster. After a Supervisor
Cluster is created, a vSphere administrator can create Supervisor Namespaces. DevOps engineer
can run workloads consisting of containers running inside vSphere Pods and create Tanzu
Kubernetes Clusters.

The Supervisor Cluster uses ESXi hosts as worker nodes. This is achieved by using an additional
process Spherelet that is created on each host. Spherelet is a kubelet that is ported natively to
ESXi and allows the host to become part of the Kubernetes cluster.

VMware, Inc. 157

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-144. Design Decisions on the Supervisor Cluster for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-001 Enable vSphere with Tanzu The Supervisor cluster is You must ensure the shared
on the shared edge and required to run Kubernetes edge and workload cluster
workload cluster in the workloads natively and to is sized to support the
workload domain. deploy Tanzu Kubernetes Supervisor cluster control
Clusters in the workload plane, any additional
domain. integrated management
workloads, and any tenant
workloads.

SDDC-WLD-VI-KUB-002 Deploy the Supervisor Deploying the control You must account for the
Cluster with small size plane nodes as small size size of the control plane
control plane nodes. appliances gives you the nodes.
ability to run up to 2000
pods within your Supervisor
Cluster.
If your pod count is
higher than 2000 for the
Supervisor Cluster, you
must deploy control plane
nodes that can handle that
level of scale.

SDDC-WLD-VI-KUB-003 Use NSX-T Data Center as You can deploy a None.

provider of the software- Supervisor Cluster either by
defined networking for the using NSX-T Data Center
Supervisor Cluster. or vSphere Networking with
HA Proxy load balancing.
VMware Cloud Foundation
uses NSX-T Data
Center for software-defined
networking across the
SDDC. Deviating for
vSphere with Tanzu would
increase the operational
overhead.

Registry Service
You use a private image registry on a Supervisor Cluster through the Registry Service. The
Registry Service is a locally managed deployment of VMware Harbor registry in the Supervisor
Cluster. One cluster can have one instance of a private image registry. All namespaces on a
Supervisor Cluster appear as projects in the private registry.

VMware, Inc. 158

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-145. Design Decisions on the vSphere Registry Service for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-004 Enable the Registry Service Enabling the Registry None

for the Supervisor Cluster. Service provides an
integrated image registry
to the Supervisor Cluster
that enables multiple
functionalities:
n Federated
authentication for end
users
n Native life cycle
management of the
image registry
n Native health
monitoring of the image
registry

Tanzu Kubernetes Cluster

A Tanzu Kubernetes Cluster is a full distribution of the open-source Kubernetes container
orchestration software that is packaged, signed, and supported by VMware. Tanzu Kubernetes
Clusters are provisioned by the VMware Tanzu™ Kubernetes Grid™ Service in the Supervisor
Cluster. The cluster consists of at least one control plane node and at least one worker node.
The Tanzu Kubernetes Grid Service deploys the clusters as Photon OS appliances on top of the
Supervisor Cluster. You determine the size and number of the appliances to be deployed by using
a YAML file.

Table 2-146. Design Decisions on the Tanzu Kubernetes Cluster for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-005 Deploy a Tanzu Kubernetes For applications that None

Cluster in the Supervisor require upstream
Cluster. Kubernetes compliance, a
Tanzu Kubernetes Cluster is
required.

SDDC-WLD-VI-KUB-006 Configure a Content Library To deploy a Tanzu You must manually

for Tanzu Kubernetes Kubernetes Cluster on a configure the Content
Cluster use in the shared Supervisor Cluster, you Library.
edge and workload cluster. must configure a Content
Library in the shared edge
and workload cluster to
pull the Photon OS-based
images from VMware.

SDDC-WLD-VI-KUB-007 Use Antrea as the container Antrea is now the default New Tanzu Kubernetes
network interface (CNI) CNI for Tanzu Kubernetes Clusters are deployed with
for your Tanzu Kubernetes Clusters. Antrea as the CNI, unless
Clusters. you specify Calico.

VMware, Inc. 159

Architecture and Design for a Virtual Infrastructure Workload Domain

Sizing Compute and Storage Resources for a Virtual Infrastructure Workload Domain
Compute and storage requirements for each component are key considerations when you
consider how to size for the solution.

You size the compute and storage requirements for the vSphere with Tanzu management
workloads, Tanzu Kubernetes Cluster management workloads, NSX-T Edge nodes, and tenant
workloads deployed in either the Supervisor Cluster or a Tanzu Kubernetes Grid cluster.

Table 2-147. Compute and Storage Resource Requirements for vSphere with Tanzu

Virtual Machine Nodes Total vCPUs Total Memory Total Storage

Supervisor Cluster 3 12 48 GB 200 GB

control plane
(small nodes - up
to 2000 pods per
Supervisor cluster)

Registry Service N/A 7 7 GB 200 GB

Tanzu Kubernetes 3 (per cluster) 6 12 GB 48GB

Cluster control plane
(small nodes)

Tanzu Kubernetes 3 (per cluster) 6 12 GB 48 GB

Cluster worker nodes
(small nodes)

VMware NSX-T Edge 2 16 64 GB 400 GB

node

VMware, Inc. 160

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-148. Design Decisions on Sizing the Tanzu Kubernetes Cluster for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-008 Deploy the Tanzu Deploying three control None.

Kubernetes Cluster with a plane nodes ensures the
minimum of three control state of your Tanzu
plane nodes. Kubernetes Cluster control
plane stays healthy in the
event of a node failure.

SDDC-WLD-VI-KUB-009 Deploy the Tanzu Deploying three worker You must configure your
Kubernetes Cluster with a nodes provides a higher tenant workloads to take
minimum of three worker potential level of availability advantage of the additional
nodes. of your workloads deployed worker nodes in the
to the Tanzu Kubernetes Tanzu Kubernetes Cluster
Cluster. to provide high availability
on an application-level.

SDDC-WLD-VI-KUB-010 Deploy the Tanzu Deploying the Tanzu The size of the Tanzu
Kubernetes Cluster with Kubernetes Cluster control Kubernetes Cluster nodes
small-size nodes for control plane and worker nodes as impacts the scale of a given
plane and workers. small-size appliances meets cluster. If you must add a
the scale requirements for node to a cluster, consider
most deployments. the use bigger nodes.
If your scale requirements
are higher than what small-
nodes deployment can
provide, you must deploy
appliances that can handle
that level of scale or split
the workload to multiple
Tanzu Kubernetes Clusters.

Network Design for vSphere with Tanzu for a Virtual Infrastructure Workload
Domain
vSphere with Tanzu requires multiple networks. This section discusses networking design not
covered in the NSX-T Data Center detailed design.

You deploy all vSphere with Tanzu workloads to NSX-T overlay networks. NSX-T Edge appliances
in the shared edge and workload cluster are deployed to VLAN-backed networks.

VMware, Inc. 161

Architecture and Design for a Virtual Infrastructure Workload Domain

Figure 2-23. Network Design for vSphere with Tanzu in a Workload Domain

Tier-1 Routable
Gateway Supervisor Cluster Management Overlay Segments

Control Plane Control Plane Control Plane

Node Node Node
Tier-1
Gateway NAT
Embedded Harbor Registry Overlay Segment
Tier-0 Gateway
Active/Active Tier-1 NAT vSphere Pod vSphere Pod vSphere Pod
Gateway Containers Containers Containers
Shared
Edge and vSphere Namespace sfo-w01-ns01 Overlay Segments
Edge Compute Edge
VM 1 Cluster VM 2 Tier-1 NAT vSphere Pod vSphere Pod vSphere Pod
Gateway Containers Containers Containers

vSphere Namespace sfo-w01-tkc01

vSphere Namespace sfo-w01-tkc01 Overlay Segments

Tier-1 NAT
Gateway vSphere Pod vSphere Pod vSphere Pod
NAT Containers Containers Containers

Tanzu Kubernetes Cluster sfo-w01-tkc01 Overlay Segment

Cluster Control Cluster
Plane Nodes Worker Nodes

Table 2-149. Networks Used by vSphere with Tanzu

Network Routable / NAT Usage

Supervisor Cluster Control Plane Routable Used by the Supervisor Cluster

Network control plane nodes.

Pod Networks NAT Used by Kubernetes pods that run

in the cluster. Any Tanzu Kubernetes
Clusters instantiated in the Supervisor
Cluster also use this pool.

Service IP Pool Network NAT Used by Kubernetes applications that

need a service IP address.

Ingress IP Pool Network Routable Used by NSX-T Data Center to create

an IP pool for load balancing.

Egress IP Pool Network Routable Used by NSX-T Data Center to create

an IP pool for NAT endpoint use.

VMware, Inc. 162

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-149. Networks Used by vSphere with Tanzu (continued)

Network Routable / NAT Usage

Namespace Networks NAT When you create a namespace,

a /28 NSX-T Data Center overlay
segment and corresponding IP pool
is instantiated to service pods in that
namespace. If that IP space runs
out, an additional /28 NSX-T overlay
segment and IP pool are instantiated.

Tanzu Kubernetes Cluster Networks NAT When you create a Tanzu Kubernetes
cluster, a Tier-1 Gateway is
instantiated in NSX-T Data Center.
On that Tier-1 Gateway, a /28 NSX-T
overlay segment and IP pool is also
instantiated.

Table 2-150. Design Decisions on vSphere with Tanzu Networking

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-011 Deploy a /28 segment for Supports the Supervisor The NSX-T segment must
use by the Supervisor Cluster control plane nodes. be manually created.
Cluster control plane nodes.

SDDC-WLD-VI-KUB-012 Dedicate a /20 subnet for A single /20 subnet is Private IP space behind
pod networking. sufficient for deployments a NAT that you can
under 2000 pods. use in multiple Supervisor
Clusters.

SDDC-WLD-VI-KUB-013 Dedicate a /22 subnet for A single /22 subnet is Private IP space behind
services. sufficient for deployments a NAT that you can
under 2000 pods. use in multiple Supervisor
clusters.

SDDC-WLD-VI-KUB-014 Dedicate a /24 or larger A /24 subnet is sufficient for This subnet must be
subnet on your corporate most deployments under routable to the rest of the
network for ingress 2000 pods, but you must corporate network.
endpoints. evaluate your ingress needs
before deployment.

SDDC-WLD-VI-KUB-015 Dedicate a /24 or larger A /24 subnet is sufficient for This subnet must be
subnet on your corporate most deployments under routable to the rest of the
network for egress 2000 pods, but you must corporate network.
endpoints. evaluate your egress needs
before deployment.

Monitoring and Alerting Design for vSphere with Tanzu for a Virtual
Infrastructure Workload Domain
During normal operation of a Kubernetes cluster and the running applications, a number of
components are instantiated. Monitor the health of the vSphere with Tanzu environment for

VMware, Inc. 163

Architecture and Design for a Virtual Infrastructure Workload Domain

identifying failures in the control plane and worker nodes by using vRealize Log Insight and
vRealize Operations Manager.

n vRealize Log Insight saves log queries and alerts, and you can use dashboards for efficient
monitoring.

n vRealize Operations Manager provides native alerts, capacity management, and custom views
and dashboards. The vSphere adapter includes metrics for:

n Supervisor Cluster Namespaces

n vSphere Pods

n Tanzu Kubernetes Clusters running in a Supervisor Cluster

Information Security and Access Design for vSphere with Tanzu for a Virtual
Infrastructure Workload Domain
You design authentication access, controls, and certificate management for vSphere with Tanzu
according to industry standards and the requirements of your organization.

vSphere with Tanzu Authentication and Access Control

You integrate vSphere with Tanzu with vCenter Single Sign-On for authentication. You can use the
configured identity sources for vCenter Single Sign-On, such as Active Directory, in the Supervisor
cluster. You can assign permissions to users on a given Supervisor cluster object, such as a
Namespace.

VMware, Inc. 164

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-151. Design Decisions on Authentication and Access Control for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-016 Create a ug-sddc-devops- Necessary to provide role- You must manually create
admin security group in based access control in the and maintain the security
Active Directory for DevOps Supervisor Cluster. group in Active Directory.
administrators. Add users
who need edit permissions
within a namespace to
the group and grant Can
Edit permissions to the
namespace for that group.
If you require
different permissions
per namespace, create
additional groups.

SDDC-WLD-VI-KUB-017 Create a ug-sddc-devops- Necessary for role-based You must manually create
ro security group in access control within the and maintain the security
Active Directory for DevOps Supervisor cluster. group within Active
administrators. Add users Directory.
who need read-only
permissions in a namespace
to the group, and grant Can
View permissions to the
namespace for that group.
If you require
different permissions
per namespace, create
additional groups.

Certificate Management
By default, vSphere with Tanzu uses a self-signed Secure Sockets Layer (SSL) certificate. This
certificate is not trusted by end-user devices or Web browsers.

As a best practice, replace self-signed certificates with certificates that are signed by a third-party
or enterprise Certificate Authority (CA).

VMware, Inc. 165

Architecture and Design for a Virtual Infrastructure Workload Domain

Table 2-152. Design Decisions on Certificate Management for vSphere with Tanzu

Decision ID Design Decision Design Justification Design Implication

SDDC-WLD-VI-KUB-018 Replace the default self- Ensures that the Replacing and managing
signed certificate for communication between certificates is an operational
the Supervisor Cluster administrators and overhead.
management interface with the Supervisor Cluster
a CA-signed certificate. management interface is
encrypted by using a
trusted certificate.

SDDC-WLD-VI-KUB-019 Replace the default self- Ensures that Replacing and managing
signed certificate for default communication between certificates is an operational
ingress with a CA-signed end users and applications overhead.
certificate. that use NSX-T Data Center
load balancers for ingress
traffic is encrypted by using
a trusted certificate.

SDDC-WLD-VI-KUB-020 Use a SHA-2 or higher The SHA-1 algorithm is Not all certificate authorities
algorithm when signing considered less secure and support SHA-2.
certificates. has been deprecated.

VMware, Inc. 166