Dell EMC PowerEdge MX SmartFabric Configuration

and Troubleshooting Guide

Abstract
This document provides the steps for configuring and troubleshooting the
Dell EMC PowerEdge MX networking switches in SmartFabric mode. It
includes examples for ethernet connections to Dell EMC Networking,
Cisco Nexus, and Fibre Channel networks.

This document replaces the Dell EMC PowerEdge MX SmartFabric

Mode Deployment Guide, which is now deprecated.

September 2019

Dell EMC Configuration and Troubleshooting Guide

Revisions
Date Description
September 2019 OpenManage Enterprise-Modular 1.10.00 and SmartFabric OS10.5.0.1 updates, Fibre
Channel connectivity, additional troubleshooting information
May 2019 Initial Release

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any software described in this publication requires an applicable software license.

© 2019 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, Dell EMC and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be trademarks of their respective owners.

Dell believes the information in this document is accurate as of its publication date. The information is subject to change without notice.

2 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

Table of contents
Revisions.............................................................................................................................................................................2
1 Introduction ...................................................................................................................................................................8
1.1 Typographical conventions ...............................................................................................................................10
1.2 Attachments ......................................................................................................................................................10
2 SmartFabric Services for PowerEdge MX: An overview ............................................................................................11
2.1 Dell EMC SmartFabric OS10 ............................................................................................................................11
2.2 Operating modes ..............................................................................................................................................11
2.2.1 Full Switch mode ..............................................................................................................................................12
2.2.2 SmartFabric mode ............................................................................................................................................12
2.3 Changing operating modes ..............................................................................................................................14
2.4 MX9116n Fabric Switching Engine (FSE): virtual ports ...................................................................................15
2.5 Virtual Link Trunking .........................................................................................................................................16
2.6 Fibre Channel (FC) connectivity ......................................................................................................................17
2.6.1 NPIV Proxy Gateway (NPG) .............................................................................................................................17
2.6.2 Direct Attached (F_port) ...................................................................................................................................18
2.6.3 FCoE (FSB) ......................................................................................................................................................18
2.7 Networks and automated QoS .........................................................................................................................19
2.8 Server templates, virtual identities, networks, and deployment .......................................................................20
2.8.1 Templates .........................................................................................................................................................20
2.8.2 Virtual identities and identity pools ...................................................................................................................21
2.8.3 Deployment.......................................................................................................................................................21
3 SmartFabric mode requirements, guidelines, and restrictions ...................................................................................22
3.1 Create multi-chassis management group .........................................................................................................22
3.2 Upstream network requirements ......................................................................................................................22
3.2.1 Physical connectivity ........................................................................................................................................22
3.2.2 Spanning Tree Protocol ....................................................................................................................................23
3.3 VLAN scaling guidelines ...................................................................................................................................23
3.4 Configuring port speed and breakout ...............................................................................................................24
3.5 Switch slot placement for SmartFabric mode ...................................................................................................24
3.5.1 Two MX9116n Fabric Switching Engines or FSEs in different chassis ............................................................24
3.5.2 Two MX5108n Ethernet switches in the same chassis ....................................................................................25
3.5.3 Two MX9116n Fabric Switching Engines or FSEs in the same chassis ..........................................................26
3.6 Switch-to-Switch cabling ...................................................................................................................................26
3.6.1 VLT backup link ................................................................................................................................................27
3.7 NIC teaming guidelines ....................................................................................................................................27

3 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 3.8 Maximum Transmission Unit (MTU) behavior ..................................................................................................28
3.9 Other restrictions and guidelines ......................................................................................................................28
4 Creating a SmartFabric ..............................................................................................................................................30
4.1 Physically cable MX chassis and upstream switches .......................................................................................30
4.2 Define VLANs ...................................................................................................................................................30
4.2.1 Define VLANs for FCoE ....................................................................................................................................31
4.3 Create the SmartFabric ....................................................................................................................................32
4.4 Configure uplink port speed or breakout, if needed .........................................................................................33
4.5 Create Ethernet uplink ......................................................................................................................................34
4.6 Configure Fibre Channel universal ports ..........................................................................................................36
4.7 Create Fibre Channel uplinks ..........................................................................................................................36
4.8 Configuring the upstream switch and connect uplink cables ...........................................................................37
5 Deploying a server .....................................................................................................................................................38
5.1 Server preparation ............................................................................................................................................38
5.1.1 Reset server CNAs to factory defaults (if required) ..........................................................................................38
5.1.2 Configure NIC partitioning on CNAs .................................................................................................................38
5.1.3 Configure other server settings ........................................................................................................................40
5.2 Create a server template ..................................................................................................................................40
5.3 Create identity pools .........................................................................................................................................42
5.4 Associate server template with networks – no FCoE .......................................................................................42
5.5 Associate Server Template with networks - FCoE ...........................................................................................43
5.6 Deploy a server template ..................................................................................................................................44
6 SmartFabric operations ..............................................................................................................................................46
6.1 Viewing the fabric .............................................................................................................................................46
6.2 Editing the fabric ...............................................................................................................................................48
6.3 Editing uplinks...................................................................................................................................................49
6.4 Editing VLANs on a deployed server ................................................................................................................50
6.5 Managing Fibre Channel Zoning on the MX9116n FSE ..................................................................................51
6.6 Connecting non-MX Ethernet Devices to the fabric .........................................................................................53
6.7 Delete SmartFabric ...........................................................................................................................................53
6.8 Module replacement process in SmartFabric ...................................................................................................54
6.8.1 Physically replace the IOM ...............................................................................................................................54
6.8.2 Identify the master IOM in a fabric ....................................................................................................................54
6.8.3 Execute process of Module Replacement with Linux command ......................................................................55
7 Switch operations .......................................................................................................................................................56
7.1 Switch management page overview .................................................................................................................56

4 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 7.1.1 Switch overview ................................................................................................................................................56
7.1.2 Hardware tab ....................................................................................................................................................58
7.1.3 Firmware tab .....................................................................................................................................................59
7.1.4 Alerts tab ...........................................................................................................................................................59
7.1.5 Settings tab .......................................................................................................................................................60
7.2 Configuring Ethernet ports ................................................................................................................................62
7.3 Upgrade Dell EMC SmartFabric OS10 .............................................................................................................64
8 Validating the SmartFabric deployment .....................................................................................................................67
8.1 View the MCM group topology .........................................................................................................................67
8.2 View the SmartFabric status .............................................................................................................................68
8.3 View port status ................................................................................................................................................70
8.4 CLI commands..................................................................................................................................................71
8.4.1 show switch-operating-mode ............................................................................................................................71
8.4.2 show discovered-expanders .............................................................................................................................71
8.4.3 show unit-provision ...........................................................................................................................................71
8.4.4 show lldp neighbors ..........................................................................................................................................71
8.4.5 show qos system ..............................................................................................................................................73
8.4.6 show policy-map ...............................................................................................................................................73
8.4.7 show class-map ................................................................................................................................................73
8.4.8 show vlt domain-id ............................................................................................................................................73
8.4.9 show vlt domain-id vlt-port-detail ......................................................................................................................74
8.4.10 show interface port channel summary .........................................................................................................74
9 SmartFabric troubleshooting ......................................................................................................................................75
9.1 Troubleshooting errors encountered for port group breakout ..........................................................................75
9.2 Troubleshooting Spanning Tree Protocol (STP) ..............................................................................................77
9.2.1 Verify if STP is enabled on upstream switches ................................................................................................77
9.2.2 Verify if type of STP is the same on MX and upstream switches .....................................................................77
9.3 Verify VLT/vPC configuration on upstream switches .......................................................................................78
9.4 Discovery of FEM and compute sleds ..............................................................................................................78
9.5 Troubleshooting uplink errors ...........................................................................................................................79
9.5.1 Toggle auto negotiation ....................................................................................................................................79
9.5.2 Set uplink ports to administratively up ..............................................................................................................80
9.5.3 Verify MTU size ................................................................................................................................................80
9.5.4 Verify auto negotiation settings on upstream switches ....................................................................................80
9.5.5 Verify LACP ......................................................................................................................................................81
9.6 Troubleshooting FC/FCoE ................................................................................................................................83

5 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 9.7 SmartFabric Services – Troubleshooting commands.......................................................................................85
9.7.1 show smartfabric personality ............................................................................................................................85
9.7.2 show smartfabric cluster ...................................................................................................................................85
9.7.3 show smartfabric cluster member.....................................................................................................................85
9.7.4 show smartfabric details ...................................................................................................................................86
9.7.5 show smartfabric uplinks ..................................................................................................................................86
9.7.6 show smartfabric networks ...............................................................................................................................87
9.7.7 show smartfabric validation-error .....................................................................................................................87
9.7.8 show smartfabric nodes ....................................................................................................................................87
10 Uplink configuration scenarios ...................................................................................................................................89
10.1 Scenario 1 - SmartFabric deployment with Dell EMC PowerSwitch Z9100-ON upstream switches ...............89
10.1.1 Dell EMC PowerSwitch Z9100-ON switch configuration..............................................................................90
10.1.2 Dell EMC PowerSwitch Z9100-ON validation ..............................................................................................92
10.2 Scenario 2 - SmartFabric connected to Cisco Nexus 3232C switches ............................................................94
10.2.1 Cisco Nexus 3232C switch configuration .....................................................................................................95
10.2.2 Configuration validation ................................................................................................................................97
10.3 Scenario 3: Connect MX9116n FSE to Fibre Channel storage - NPIV Proxy Gateway mode .....................100
10.3.1 Configuration validation ..............................................................................................................................101
10.4 Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC Direct Attach .......................................104
10.4.1 Configuration validation ..............................................................................................................................105
10.5 Scenario 5: Connect MX5108n to Fibre Channel storage - FSB ...................................................................107
10.5.1 SmartFabric configuration steps ................................................................................................................108
10.6 Scenario 6: Configure Boot from SAN ............................................................................................................109
10.6.1 Configure NIC Boot Device ........................................................................................................................109
10.6.2 Configure BIOS settings .............................................................................................................................111
10.6.3 Connect FCoE LUN....................................................................................................................................112
10.6.4 Set up install media connection .................................................................................................................112
10.6.5 Set up OS driver install media using Lifecycle Controller ..........................................................................112
A Hardware used in this document ..............................................................................................................................114
A.1 Dell EMC PowerSwitch S3048-ON.................................................................................................................114
A.2 Dell EMC PowerSwitch Z9100-ON .................................................................................................................114
A.3 Dell EMC PowerSwitch S4148U-ON ..............................................................................................................114
A.4 Dell EMC PowerSwitch Z9264F-ON...............................................................................................................114
A.5 Dell EMC Unity 500F storage array ................................................................................................................115
A.6 Cisco Nexus 3232C ........................................................................................................................................115
B Dell EMC Unity information ......................................................................................................................................116

6 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 B.1 Determine Unity 500F storage array FC WWPNs ..........................................................................................116
B.2 CNA FCoE port WWPNs ................................................................................................................................117
B.3 Configure Unity FC storage ............................................................................................................................118
B.3.1 Create a storage pool .....................................................................................................................................118
B.3.2 Add hosts ........................................................................................................................................................119
B.3.3 Create LUNs and configure host access ........................................................................................................119
C Additional information ...............................................................................................................................................121
C.1 Delete MCM group..........................................................................................................................................121
C.2 Reset chassis using RACADM .......................................................................................................................121
C.3 Reset SmartFabric OS10 switch to factory defaults .......................................................................................121
C.4 Reset Cisco Nexus 3232C to factory defaults ................................................................................................121
C.5 Connect to IO Module console port via RACADM ..........................................................................................122
D Validated components ..............................................................................................................................................123
D.1 Scenarios 1 and 2 ...........................................................................................................................................123
D.1.1 Dell EMC PowerSwitches ...............................................................................................................................123
D.1.2 Dell EMC PowerEdge MX7000 chassis and components .............................................................................123
D.1.3 Cisco Nexus switches .....................................................................................................................................124
D.2 Scenarios 3 and 4 ...........................................................................................................................................124
E Technical resources .................................................................................................................................................125
F Support and feedback ..............................................................................................................................................126

7 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

1 Introduction
Our vision at Dell EMC is to be the essential infrastructure company from the edge, to the core, and to the
cloud. Dell EMC Networking ensures modernization for today’s applications and for the emerging cloud-native
world. Dell EMC is committed to disrupting the fundamental economics of the market with an open strategy
that gives you the freedom of choice for networking operating systems and top-tier merchant silicon. The Dell
EMC strategy enables business transformations that maximize the benefits of collaborative software and
standards-based hardware, including lowered costs, flexibility, freedom, and security. Dell EMC provides
further customer enablement through validated deployment guides which demonstrate these benefits while
maintaining a high standard of quality, consistency, and support.

The Dell EMC PowerEdge MX is a unified, high-performance data center infrastructure. PowerEdge MX
provides the agility, resiliency, and efficiency to optimize a wide variety of traditional and new, emerging data
center workloads and applications. With its kinetic architecture and agile management, PowerEdge MX
dynamically configures compute, storage, and fabric, increases team effectiveness, and accelerates
operations. The responsive design delivers the innovation and longevity that customers need for their IT and
digital business transformations.

As part of the PowerEdge MX platform, Dell EMC SmartFabric OS10 includes SmartFabric Services.
SmartFabric Services is a network automation and orchestration solution that is fully integrated with the MX
Platform.

Dell EMC PowerEdge MX7000 chassis

This document provides information about SmartFabric OS10 SmartFabric Services running on the
PowerEdge MX platform. This document also provides examples for the deployment of two PowerEdge
MX7000 chassis and the setup and configuration of SmartFabric Services. In SmartFabric mode, switches

8 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 operate as a Layer 2 I/O aggregation fabric and are managed through the Open Manage Enterprise - Modular
(OME-M) graphical user interface or console.

This guide also demonstrates connectivity with different upstream switch options, including:

• Dell EMC PowerSwitch Z9100-ON

• Cisco Nexus 3232C
• Fibre Channel connectivity methods: NPG, Direct Attach, and FSB modes

Note: The examples in document assume that the MX7000 chassis are configured in a Multi-Chassis
Management group and that no errors have been found. Additionally, this guide assumes the reader has a basic
understanding of the PowerEdge MX platform.

Four important terminologies and their definitions are as follows:

Scalable Fabric – This is exclusive to the MX7000 platform. This is an architecture comprised of the Dell
EMC Networking MX9116n Fabric Switching Engine (FSE) and Dell EMC Networking MX7116n Fabric
Expander Module (FEM) allowing a fabric to span up to ten MX7000 chassis. This creates a single network
fabric enabling efficient east/west traffic flows between participating chassis. Scalable Fabric is supported in
both SmartFabric and Full Switch modes.

SmartFabric mode - SmartFabric mode leverages SmartFabric Services (see below) to create a Layer 2
network leveraging one to ten MX7000 chassis. Switches operating in SmartFabric mode are administered
through the OpenManage Enterprise - Modular (OME-M) GUI interfaces that provide complete lifecycle
management of the network fabric.

Full Switch mode – When operating in Full Switch mode, the switch can perform any functionality supported
by the version of SmartFabric OS10 running on the switch. Most of the configuration is performed using the
CLI, not the OME-M GUI.

SmartFabric Services (SFS) – In PowerEdge MX, SFS technology provides the underlying network
automation and orchestration to support all automated network operations. SFS is the underlying technology
for all Dell EMC SmartFabric OS10 automation efforts including PowerEdge MX, Isilon back-end storage
networking, VxRail network automation, and so on.

Table 1 outlines what this document is and is not. Also, this guide assumes a basic understanding of the
PowerEdge MX platform.

Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide - is/is not
This guide is This guide is not/does not
A reference for the most used features of A guide for all features of the MX7000 platform
SmartFabric operating mode
A secondary reference to the Release Notes Take precedence over the Release Notes

Note: For a general overview and details of PowerEdge MX networking concepts, see the Dell EMC PowerEdge
MX Network Architecture Guide.

9 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

1.1 Typographical conventions
The CLI and GUI examples in this document use the following conventions:

Monospace Text CLI examples

Underlined Monospace Text CLI examples that wrap the page

Italic Monospace Text Variables in CLI examples

Bold Monospace Text Commands entered at the CLI prompt, or to highlight information in CLI
output

Bold text UI elements and information entered in the GUI

1.2 Attachments
This document in .pdf format includes one or more file attachments. To access attachments in Adobe Acrobat
Reader, click the icon in the left pane halfway down the page, then click the icon.

10 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

2 SmartFabric Services for PowerEdge MX: An overview

2.1 Dell EMC SmartFabric OS10

The networking market is transitioning from a closed, proprietary stack to open hardware supporting a variety
of operating systems. Dell EMC SmartFabric OS10 is designed to allow multi-layered disaggregation of the
network functionality. While OS10 contributions to Open Source provide users freedom and flexibility to pick
their own third-party networking, monitoring, management and orchestration applications, SmartFabric OS10
bundles industry hardened networking stack featuring standard L2 and L3 protocols over a standard and well
accepted CLI interface.

Dell EMC SmartFabric OS10 High Level Architecture

2.2 Operating modes

The Dell EMC Networking MX9116n Fabric Switching Engine (FSE) and MX5108n Ethernet Switch operate in
one of two modes:

• Full Switch mode (Default) – All switch-specific SmartFabric OS10 capabilities are available
• SmartFabric mode – Switches operate as a Layer 2 I/O aggregation fabric and are managed
through the Open Manage Enterprise-Modular (OME-M) console

The following SmartFabric OS10 CLI commands have been added specifically for the PowerEdge MX
platform:

11 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 • show switch-operating-mode – displays the current operating mode (SmartFabric or
Full Switch) of a supported switch
• show discovered-expanders – displays the MX7116n FEMs attached to the MX9116n
FSEs
• show unit-provision – displays or configures the unit ID and service tag of a MX7116n
FEM attached to a MX9116n FSE

Note: For more information, see the SmartFabric OS10 User Guide for PowerEdge MX I/O Modules on the
Support for Dell EMC Networking MX9116n - Manuals and documents and Support for Dell EMC Networking
MX5108- Manuals and documents web pages.

2.2.1 Full Switch mode

In Full Switch mode, all SmartFabric OS10 features and functions supported by the hardware are available to
the user. In other words, the switch operates the same way as any other SmartFabric OS10 switch.
Configuration is primarily done using the CLI, however, the following items can be configured or managed
using the OME-M GUI:

1. Initial switch deployment: Configure hostname, password, SNMP, NTP, etc.

2. Set ports administratively up or down, configure MTU
3. Monitor health, logs, alerts, and events
4. Update the SmartFabric OS10 software
5. View physical topology
6. Switch power management

Full Switch mode is typically used when a desired feature or function is not available when operating in
SmartFabric mode. For more information about Dell EMC SmartFabric OS10 operations, see Dell EMC
Networking OS Info Hub.

2.2.2 SmartFabric mode

A SmartFabric is a logical entity that consists of a collection of physical resources, such as servers and
switches, and logical resources such as networks, templates, and uplinks. The OpenManage Enterprise –
Modular (OME-M) console provides a method to manage these resources as a single unit.

In the PowerEdge M1000e and FX2 platforms, I/O Aggregation (IOA) was implemented to simplify the
process to connect blade servers to upstream networks, so server administrators and generalists could
manage uplinks, downlinks, and VLAN assignments without needing to be fluent with the CLI.

SmartFabric Services mode builds on this IOA functionality providing:

1. Data center modernization

• I/O Aggregation
• Plug-and-play fabric deployment
• Single interface to manage all switches in the fabric

2. Lifecycle management

• Fabric-wide SmartFabric OS10 updates

• Automated or user enforced roll back to last well-known state

12 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 3. Fabric automation

• Physical topology compliance

• Server networking managed via templates
• Automated QoS assignment per VLAN
• Automated storage networking

4. Failure remediation

• Dynamically adjusts bandwidth across all inter-switch links in the event of a link failure
• Automatically detects fabric misconfigurations or link level failure conditions
• Automatically heals the fabric on failure condition removal

Note: In SmartFabric mode, MX series switches operate entirely as a Layer 2 network fabric. Layer 3 protocols
are not supported.

When operating in SmartFabric mode, access to certain CLI commands is restricted to SmartFabric OS10
show commands and can run the following subset of CLI configuration commands:

• clock – Configure clock parameters

• end – Exit to the EXEC mode
• exit – Exit from the current mode
• fc alias – Set Fiber Channel name
• fc zone – Set Fiber Channel zone name
• fc zoneset – Set Fiber Channel zoneset name
• help – Display available commands
• hostname – Set the system hostname
• host-description – Define the description for the host
• interface – Configure or select an interface
• ip nameserver – Configure nameserver
• ip ssh server – Configure SSH server
• ip telnet server – Configure Telnet server
• login concurrent-session – Start concurrent session and login
• login statistics – Enable timeframe for the login session
• logging – Configure system logging
• management route – Configure the IPV4/IPv6 management route
• no – Delete or disable commands in configuration mode
• ntp – Configure the network time protocol
• snmp-server – Configure the SNMP server
• tacacs-server – Configure the TACACS server
• username – Create or modify user credentials
• spanning-tree commands:

- disable – Disable spanning tree globally

- mac-flush-timer – Set the time used to flush MAC address entries
- mode – Enable a spanning-tree mode, such as RSTP

13 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 - rstp – Configure rapid spanning-tree protocol (RSTP) mode
- vlan – Configure spanning-tree on a VLAN range

Table 2 outlines the differences between the two operating modes and apply to both the MX9116n FSE and
the MX5108n switches.

IOM operating mode differences

Full Switch mode SmartFabric mode
Configuration changes are persistent during power Only the configuration changes made using the
cycle events. OS10 commands below are persistent across power
cycle events. All other CLI configuration commands
are disabled.

clock
fc alias
fc zone
fc zoneset
hostname
host-description
interface
ip nameserver
ip ssh server
ip telnet server
login concurrent-session
login statistics
logging
management route
ntp
snmp-server
tacacs-server
username
spanning-tree
vlan

All switch interfaces are assigned to VLAN 1 by Layer 2 bridging is disabled by default. Interfaces
default and are in the same Layer 2 bridge domain. must join a bridge domain (VLAN) before being able
to forward frames.
All configuration changes are saved in the running Verify configuration changes using feature-specific
configuration by default. To display the current show commands, such as show interface and
configuration, use the show running- show vlan, instead of show running-
configuration command. configuration.

2.3 Changing operating modes

In both Full Switch and SmartFabric modes, all configuration changes you make using the OME-M GUI are
retained when you switch modes. Dell EMC recommends using the graphical user interface for switch
configuration in SmartFabric mode and the SmartFabric OS10 CLI for switch configuration in Full Switch
mode.

By default, a switch is in Full Switch mode. When that switch is added to a fabric, it automatically changes to
SmartFabric mode. When you change from Full Switch to SmartFabric mode, all Full Switch CLI
configurations are deleted except for the subset of CLI commands supported in SmartFabric mode.

14 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 To change a switch from SmartFabric to Full Switch mode, you must delete the fabric. At that time, all
SmartFabric GUI configuration changes are deleted except for the configurations supported by the subset of
SmartFabric CLI commands (hostname, SNMP settings, etc.) and the changes made to port interfaces,
except for admin state (shutdown/no shutdown), MTU, speed, and auto-negotiation mode.

Note: There is no CLI command to switch between operating modes. Delete the fabric to change from
SmartFabric to Full Switch mode.

The CLI command show switch-operating-mode displays the currently configured operating mode
of the switch. This information is also available on the switch landing page in the OME-M GUI.

2.4 MX9116n Fabric Switching Engine (FSE): virtual ports

A virtual port is a logical switch port that connects to a downstream server and has no physical hardware
location on the switch. Virtual ports are created when an MX9116n Fabric Switching Engine (FSE) on-boards
an MX7116n Fabric Expander Module (FEM). The onboarding process consists of discovery and
configuration.

Note: If the servers in the chassis have dual-port NICs, only QSFP28-DD port 1 on the FEM needs to be
connected. Do not connect QSFP28-DD port 2.

To verify the auto-discovered Fabric Expander Modules, enter the show discovered-expanders
command.

OS10# show discovered-expanders

Service-tag Model Type Chassis-service-tag Chassis-slot Port-group Virtual-
Slot-Id
-----------------------------------------------------------------------------------------
403RPK2 MX7116n Fabric 1 SKY003Q A2 1/1/1 71
Expander Module

If the FSE is in SmartFabric mode, the attached FEM is automatically configured and virtual ports on the
Fabric Expander Module and a virtual slot ID are created and mapped to 8x25GbE breakout interfaces in
FEM on the Fabric Engine.

An FSE in Full Switch mode automatically discovers the FEM when these conditions are met:

• The FEM is connected to the FSE by attaching a cable between the QSFP28-DD ports on both
devices
• The interface for the QSFP28-DD port-group connected to the FSE is in 8x25GbE FEM mode
• At least one blade server is inserted into the MX7000 chassis containing the FEM

Note: If the FSE is in Full Switch mode, you must manually configure the unit ID of the FEM. See the OS10
Enterprise Edition User Guide — PowerEdge MX I/O Modules for implementation.

Once the FSE discovers the FEM, it creates virtual ports by mapping each 8x25GbE FEM breakout interface
in port groups 1 to 10 to a FEM virtual port. Table 3 shows an example of this mapping.

15 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Virtual port mapping
FSE QSFP28-DD FSE 25G FEM unit ID
FEM service tag FEM virtual ports
port group interfaces (virtual slot ID)
12AB3456 portgroup1/1/1 1/1/17:1 71 1/71/1
1/1/17:2 1/71/2
1/1/17:3 1/71/3
1/1/17:4 1/71/4
1/1/18:1 1/71/5
1/1/18:2 1/71/6
1/1/18:3 1/71/7
1/1/18:4 1/71/8

When a QSFP28-DD port group is mapped to a FEM, in the show interface status output, the eight
interfaces display dormant instead of up until a virtual port starts to transmit server traffic:

OS10# show interface status

--------------------------------------------------------------------------------
Port Description Status Speed Duplex Mode Vlan Tagged-Vlans
--------------------------------------------------------------------------------
...
Eth 1/1/17:1 dormant
Eth 1/1/17:2 dormant
Eth 1/1/17:3 dormant
Eth 1/1/17:4 dormant
Eth 1/1/18:1 dormant
Eth 1/1/18:2 dormant
Eth 1/1/18:3 dormant
Eth 1/1/18:4 dormant
...

You can also use the show interface command to display the Fabric Engine physical port-to-Fabric
Expander virtual port mapping, and the operational status of the line:

OS10# show interface ethernet 1/1/30:3

Ethernet 1/1/30:3 is up, line protocol is dormant
Interface is mapped to ethernet1/77/7

Note: If you move a FEM by cabling it to a different QSFP28-DD port on the Fabric Engine, all software
configurations on virtual ports are maintained. Only the QSFP28-DD breakout interfaces that map to the virtual
ports change.

2.5 Virtual Link Trunking

Virtual Link Trunking (VLT) aggregates two identical physical switches to form a single logical extended
switch. However, each of the VLT peers has its own control and data planes and can be configured
individually for port, protocol, and management behaviors. Though the dual physical units act as a single

16 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 logical unit, the control and data plane of both switches remain isolated, ensuring high availability and high
resilience for all its connected devices. This differs from the legacy stacking concept, where there is a single
control plane across all switches in the stack, creating a single point of failure.

With the critical need for high availability in modern data centers and enterprise networks, VLT plays a vital
role connecting with rapid convergence, seamless traffic flow, efficient load balancing, and loop free
capabilities.

With the instantaneous synchronization of MAC and ARP entries, both the nodes remain Active-Active and
continue to forward the data traffic seamlessly.

VLT is required when operating in SmartFabric mode.

For more information on VLT, see the Virtual Link Trunking chapter in the OS10 Enterprise Edition User
Guide - PowerEdge MX I/O Modules and Virtual Link Trunking (VLT) in Dell EMC OS10 Enterprise Edition
Best Practices and Deployment Guide.

2.6 Fibre Channel (FC) connectivity

PowerEdge MX Ethernet I/O modules support Fibre Channel, or FC connectivity in three different ways:
Direct Attach (also called F_port), NPIV Proxy Gateway (NPG), and FIP Snooping Bridge (FSB). The method
to implement depends on the existing infrastructure and application requirements. Consult your Dell EMC
representative for more information.

Configuring FC connectivity in SmartFabric mode is simple and is almost identical across the three
connectivity types.

2.6.1 NPIV Proxy Gateway (NPG)

The most common connectivity method, NPIV Proxy Gateway mode (NPG) is used when connecting
PowerEdge MX to a storage area network that hosts a storage array. NPG mode is very simple to implement
as there is very little configuration that needs to be done. The NPG switch converts FCoE from the server to
native FC and aggregates the traffic into an uplink. The NPG switch is effectively transparent to the FC SAN,
which “sees” the hosts themselves. This mode is supported only on the MX9116n FSE.

FC Switch FC Switch
Spine 1 Spine 2

FC SAN A
FC SAN B

Controller A Controller B

MX9116n VLT MX9116n Unity 500F

(Leaf 1) (Leaf 2) UnityUnity
500F 500F
MX7000 MX7000 Note: Brocade 6510 switches were used in
chassis 1 chassis 2 validating this brownfield example. Other FC
switches may be used as well.
FC (NPG) network to Dell EMC Unity

17 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 For more information on configuration and deployment, see Section 10.3.

2.6.2 Direct Attached (F_port)

Direct Attached mode, or F_port, is used when FC storage needs to be directly connected to the switch. The
switch supports the required services such as name server and zoning that are typical of FC switches.

This example demonstrates Fibre Channel directly attaching to the Dell EMC Unity 500F storage array.
MX9116n FSE universal ports 44:1 and 44:2 are required for FC connections and operate in F_port mode,
which allows for an FC storage array to be connected directly to the MX9116n FSE. The uplink type enables
F_port functionality on the MX9116n unified ports, converting FCoE traffic to native FC traffic and passing that
traffic to a directly attached FC storage array.

This mode is supported only on the MX9116n FSE.

Spine 1 Spine 2 Unity

Unity 500F500F
Controller A Controller B

FC SAN A
FC SAN B

MX9116n VLT MX9116n

(Leaf 1) (Leaf 2)

MX7000 MX7000
chassis 1 chassis 2
Fibre Channel (F_Port) direct connect network to Dell EMC Unity

For more information on configuration and deployment, see Section 10.4.

2.6.3 FCoE (FSB)

FCoE Transit, or FSB mode is used when connecting PowerEdge MX to an upstream switch that accepts
FCoE and converts it to native FC, such as the Dell EMC PowerSwitch S4148U. This mode is typically used
when an existing FCoE infrastructure is in place that PowerEdge MX must connect to. In Figure 5, uplink type
enables NPG FC Gateway functionality on the MX9116n FSE unified ports, converting FCoE traffic to native
FC traffic on PowerSwitch S4148U-ON and passing that traffic to an external FC switch.

When operating in FSB mode, the switch snoops FIP packets on FCoE-enabled VLANs and discovers the
following information:

• End nodes (ENodes)

• Fibre Channel forwarders (FCFs)
• Connections between ENodes and FCFs
• Sessions between ENodes and FCFs

Using the discovered information, the switch installs ACL entries that provide security and point-to-point link
emulation.

18 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 This mode is supported on both the MX9116n FSE and the MX5108n Ethernet Switch.

S4148U-ON S4148U-ON
(NPG mode) (NPG mode)
ToR switch 1 VLT ToR switch 2 FCoE SAN A
FC SAN A

FCoE SAN B
FC SAN B

FC Switch FC Switch

MX5108n MX5108n
FSB mode VLT
FSB mode
(Leaf 2) Controller A Controller B
(Leaf 1)

MX7000 chassis Unity 500F

UnityUnity
500F 500F
FCoE (FSB) Network to Dell EMC Unity through NPG switch

For more information on configuration and deployment, see Section 10.5.

2.7 Networks and automated QoS

In addition to assigning VLANs to server profiles, SmartFabric automates QoS settings based on the Network
Type specified. Figure 6 shows that when defining a VLAN, one of 11 options are pre-defined.

Network types available in SmartFabric mode

Table 4 lists the network types and related settings. The QoS group is the numerical value for the queues
available in SmartFabric mode. Available queues include 2 through 5. Queues 1, 6, and 7 are reserved.

Note: In SmartFabric mode, an administrator cannot change the default weights for the queues.

19 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Network types and default QoS settings
Network type Description QoS group

General Purpose (Bronze) Used for low priority data traffic 2

General Purpose (Silver) Used for standard/default priority data traffic 3

General Purpose (Gold) Used for high priority data traffic 4

General Purpose (Platinum) Used for extremely high priority data traffic 5

Cluster Interconnect Used for cluster heartbeat VLANs 5

Used for hypervisor management connections such as the

Hypervisor Management 5
ESXi management VLAN

Storage - iSCSI Used for iSCSI VLANs 5

Storage - FCoE Used for FCoE VLANs 5

Used for VLANs supporting storage data replication such 5

Storage - Data Replication
as for VMware VSAN

Used for VLANs supporting vMotion and similar 5

VM Migration
technologies

VMware FT Logging Used for VLANs supporting VMware Fault Tolerance 5

2.8 Server templates, virtual identities, networks, and deployment

For detailed information on templates, identities, and deployment, see the OpenManage Enterprise - Modular
documentation and the technical paper PowerEdge MX7000: Templates and Profiles.

2.8.1 Templates
A template is a set of system configuration settings referred to as attributes. A template may contain a small
set of attributes for a specific purpose, or all the attributes for a full system configuration. Templates allow for
multiple servers to be configured quickly and automatically without the risk of human error.

Networks (VLANs) are assigned to NICs as part of the server template. When the template is deployed, those
networks are programmed on the fabric for the servers associated with the template.

Note: Network assignment through template only functions for servers connected to a SmartFabric. If a template
with network assignments is deployed to a server connected to a switch in Full Switch mode, the network
assignments are ignored.

OME-M GUI provides options for creating templates:

• Most frequently, templates are created by getting the current system configuration from a server that
has been configured to the exact specifications required (referred to as a “Reference Server”).
• Templates may be cloned or copied and edited.
• A template can be created by importing a Server Configuration Profile (SCP) file. The SCP file may
be from a server or exported by OpenManage Essentials, OpenManage Enterprise, or OME-M.
• OME-M comes prepopulated with several templates for specific purposes.

20 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

2.8.2 Virtual identities and identity pools
Some of the attributes included in a template are referred to as identity attributes. Identity attributes identify a
device and distinguish it from all other devices on the network. Since identity attributes must uniquely identify
a device, it is imperative that each device has a unique network identity. Otherwise, devices won’t be able to
communicate with each other over the network.

Devices come with unique manufacturer-assigned identity values preinstalled, such as a factory-assigned
MAC address. Those identities are fixed and never change. However, devices can assume a set of alternate
identity values, called a “virtual identity”. A virtual identity functions on the network using that identity, as if the
virtual identity was its factory-installed identity. The use of virtual identity is the basis for stateless operations.

OME-M console uses identity pools to manage the set of values that can be used as virtual identities for
discovered devices. It controls the assignment of virtual identity values, selecting values for individual
deployments from pre-defined ranges of possible values. This allows the customer to control the set of values
which can be used for identities. The customer doesn’t have to enter all needed identity values with every
deployment request, or remember which values have or have not been used. Identity pools make
configuration deployment and migration much easier to manage.

Identity pools are used in conjunction with template deployment and profile operations. They provide sets of
values that can be used for virtual identity attributes for deployment. After a template is created, an identity
pool may be associated with it. Doing this directs the identity pool to get identity values whenever the
template is deployed to a target device. The same identity pool can be associated with, or used by, any
number of templates. Only one identity pool can be associated with a template.

Each template will have specific virtual identity needs, based on its configuration. For example, one template
may have iSCSI configured, so it needs the appropriate virtual identities for iSCSI operations. Another
template may not have iSCSI configured, but may have FCoE configured, so it will need virtual identities for
FCoE operations but not for iSCSI operations, etc.

2.8.3 Deployment
Deployment is the process of applying a full or partial system configuration on a specific target device. In
OME-M, templates are the basis for all deployments. Templates contain the system configuration attributes
that get sent to the target server, then the iDRAC on the target device applies the attributes contained in the
template and reboots the server if necessary. Often, templates contain virtual identity attributes. As mentioned
above, identity attributes must have unique values on the network. Identity Pools facilitate the assignment and
management of unique virtual identities.

21 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

3 SmartFabric mode requirements, guidelines, and restrictions
Before deploying a SmartFabric, ensure that the following requirements, guidelines, and restrictions are
followed. Failure to do so may impact your network.

3.1 Create multi-chassis management group

For a scalable fabric that uses more than one MX chassis, the chassis must be in a Multi-Chassis
Management (MCM) Group. For more information on how to create MCM Group, refer to Dell EMC
OpenManage Enterprise-Modular Edition for PowerEdge MX7000 Chassis.

Note: SmartFabric mode can be enabled on a single chassis having two MX9116n FSEs or two MX5108n
switches. For a SmartFabric implemented using a single chassis, creating an MCM group is not mandatory but
recommended. The chassis must be in an MCM group for a SmartFabric containing more than one MX chassis.

3.2 Upstream network requirements

3.2.1 Physical connectivity

All physical Ethernet connections within an uplink from a SmartFabric are automatically grouped into a single
LACP LAG. Because of this, all ports on the upstream switches must also be in a single LACP LAG. Failure to
do so may create network loops.

A minimum of one physical uplink from each MX switch to each upstream switch is required and that the
uplinks must be connected in a “mesh” or “bowtie” design.

Note: The upstream switch ports must be in a single LACP LAG as shown in the figure below. Creating multiple
LAGs within a single uplink results in a network loop.

Recommended upstream network connectivity

22 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

3.2.2 Spanning Tree Protocol
By default, SmartFabric OS10 uses Rapid per-VLAN Spanning Tree Plus (RPVST+) across all switching
platforms including PowerEdge MX networking IOMs. SmartFabric OS10 also supports RSTP.

Note: Dell EMC recommends using RSTP instead of RPVST+ when more than 64 VLANs are required in a fabric
to avoid performance problems.

Caution should be taken when connecting an RPVST+ to an existing RSTP environment. RPVST+ creates a
single topology per VLAN with the default VLAN, typically VLAN 1, for the Common Spanning Tree (CST) with
RSTP.

For non-native VLANs, all bridge protocol data unit (BPDU) traffic is tagged and forwarded by the upstream,
RSTP-enabled switch with the associated VLAN. These BPDUs use a protocol-specific multicast address.

Any other RPVST+ tree attached to the RSTP tree might processes these packets accordingly leading to the
potential of unexpected trees.

Note: When connecting to an existing environment that is not using RPVST+, Dell EMC recommends changing to
the existing spanning tree protocol before connecting a SmartFabric OS10 switch. This ensures same type of
Spanning Tree is run on the SmartFabric OS10 MX switches and the upstream switches.

To switch from RPVST+ to RSTP, use the spanning-tree mode rstp command:

MX9116N-A1(config)# spanning-tree mode rstp

MX9116N-A1(config)# end

To validate the STP configuration, use the show spanning-tree brief command:

MX9116N-A1#show spanning-tree brief

Spanning tree enabled protocol rstp with force-version rstp
Executing IEEE compatible Spanning Tree Protocol
Root ID Priority 0, Address 4c76.25e8.f2c0
Root Bridge hello time 2, max age 20, forward delay 15
Bridge ID Priority 32768, Address 2004.0f00.cd1e
Configured hello time 2, max age 20, forward delay 15
Flush Interval 200 centi-sec, Flush Invocations 95
Flush Indication threshold 0 (MAC flush optimization is disabled)

Note: STP is required. Operating a SmartFabric with STP disabled creates network loops and is not supported.

3.3 VLAN scaling guidelines

Because SmartFabric mode provides network automation capabilities that Full Switch mode does not, the
number of recommended VLANs differs between the modes. Table 5 provides the recommended maximum
number of VLANs per fabric, uplink, and server port.

Note: These are recommendations, not enforced maximums.

23 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Recommended maximum number of VLANs in SmartFabric mode
OS10 Version Parameter Value

Recommended Max VLANs per Fabric 256

10.5.0.1 Recommended Max VLANs per Uplink 256

Recommended Max VLANs per server port 64

Recommended Max VLANs per Fabric 128

10.4.0.R3S
Recommended Max VLANs per Uplink 128
10.4.0.R4S
Recommended Max VLANs per server port 32

3.4 Configuring port speed and breakout

If you need to change the default port speed and/or breakout configuration of an uplink port, you must do that
prior to creating the uplink.

For example, the QSFP28 interfaces that belong to port groups 13, 14, 15, and 16 on MX9116n FSE are
typically used for uplink connections. By default, the ports are set to 1x100GbE. The QSFP28 interface
supports the following Ethernet breakout configurations:

• 1x 100GbE – One 100GbE interface

• 1x 40GbE – One 40GbE interface
• 2x 50GbE – Breakout a QSFP28 port into two 50GbE interfaces
• 4x 25GbE – Breakout a QSFP28 port into four 25GbE interfaces
• 4x 10GbE – Breakout a QSFP28 port into four 10GbE interfaces

The MX9116n FSE also supports Fibre Channel (FC) capabilities via Universal Ports on port-groups 15 and
16. For more information on configuring FC storage on the MX9116n FSE, see Section 10.3 and 10.4.

For more information on interface breakouts, see OS10 User Guide - PowerEdge MX I/O Modules Release
10.4.0E(R3S).

3.5 Switch slot placement for SmartFabric mode

SmartFabric mode supports three specific switch placement options. Attempts to use placements different
than described here is not supported and may result in unpredictable behavior and/or data loss.

Note: The cabling shown in this section, Section 3.5, is the VLTi connections between the MX switches.

3.5.1 Two MX9116n Fabric Switching Engines or FSEs in different chassis

This is the recommended placement when creating a SmartFabric on top of a Scalable Fabric Architecture.
Placing the FSE modules in different chassis provides redundancy in the event of a chassis failure. This
configuration supports placement in Chassis1 Slot A1 and Chassis 2 Slot A2 or Chassis1 Slot B1 and
Chassis 2 Slot B2. A SmartFabric cannot include a switch in Fabric A and a switch in Fabric B.

24 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 IOM placement – 2 x MX9116n in different chassis

3.5.2 Two MX5108n Ethernet switches in the same chassis

The MX5108n Ethernet Switch is only supported in single chassis configurations, with the switches in either
slots A1/A2 or slots B1/B2. A SmartFabric cannot include a switch in Fabric A and a switch in Fabric B.

IOM placement – 2 x MX5108n in the same chassis

25 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

3.5.3 Two MX9116n Fabric Switching Engines or FSEs in the same chassis
This placement should only be used in environments with a single chassis, with the switches in either slots
A1/A2 or slots B1/B2. A SmartFabric cannot include a switch in Fabric A and a switch in Fabric B.

IOM placement – 2 x MX9116n in the same chassis

3.6 Switch-to-Switch cabling

When operating in SmartFabric mode, each switch pair runs a VLT interconnect (VLTi) between them. For the
MX9116n FSE, QSFP28-DD port groups 11 and 12 (eth1/1/37-1/1/40) are used.

For the MX5108n, ports 9 and 10 are used. Port 10 operates at 40GbE instead of 100GbE because all VLTi
links must run at the same speed.

Note: The VLTi ports are not user selectable and the connection topology is enforced by the SmartFabric engine.

MX9116n SmartFabric VLTi cabling

MX5108n SmartFabric VLTi cabling

26 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

3.6.1 VLT backup link
A pair of cables is used to provide redundancy for the VLTi link. A third redundancy mechanism, a VLT
backup link is automatically created when the SmartFabric is created. This link exchanges VLT heartbeat
information between the two switches to avoid a split-brain scenario should the external VLTi links go down.
Based on the node liveliness information, the VLT LAG/port is in up state in the primary VLT peer and in down
state in the secondary VLT peer. When only the VLTi link fails, but the peer is alive, the secondary VLT peer
shuts down the VLT ports. When the node in primary peer fails, the secondary becomes the primary peer.

To see the status of VLT backup link, run show vlt domain-id backup-link.

For example:
OS10# show vlt 255 backup-link
VLT Backup Link
------------------------
Destination : fde1:53ba:e9a0:de14:2204:fff:fe00:a267
Peer Heartbeat status : Up
Heartbeat interval : 30
Heartbeat timeout : 90
Destination VRF : default

3.7 NIC teaming guidelines

While NIC teaming is not required, it is generally suggested for redundancy unless a specific implementation
recommends against it.

There are two main kinds of NIC teaming:

• Switch dependent: Also referred to as LACP, 802.3ad, or Dynamic Link Aggregation, this teaming
method uses the LACP protocol to understand the teaming topology. This teaming method provides
Active-Active teaming and requires the switch to support LACP teaming.
• Switch independent: This method uses the operating system and NIC device drivers on the server
to team the NICs. Each NIC vendor may provide slightly different implementations with different pros
and cons.

NIC Partitioning (NPAR) can impact how NIC teaming operates. Based on restrictions implemented by the
NIC vendors related to NIC partitioning, certain configurations will preclude certain types of teaming.

The following restrictions are in place for both Full Switch and SmartFabric modes:

• If NPAR is NOT in use, both Switch Dependent (LACP) and Switch Independent teaming methods
are supported
• If NPAR IS in use, only Switch Independent teaming methods are supported. Switch Dependent
teaming is NOT supported

If Switch Dependent (LACP) teaming is used, the following restrictions are in place:

• The iDRAC shared LAN on motherboard (LOM) feature can only be used if the “Failover” option on
the iDRAC is enabled
• If the host OS is Windows, the LACP timer MUST be set to “slow” (also referred to as “normal”)

1. Microsoft Windows 2012 R2, see Instructions

27 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 2. Microsoft Windows 2016, see Instructions

Refer to the network adapter or operating system documentation for detailed NIC teaming instructions.

Note: If using VMware ESXi and LACP, it is recommended to use VMware ESXi 6.7.0 Update 2.

Table 6 shows the options that MX Platform provides for NIC Teaming.

NIC Teaming Options on MX Platform

No Teaming No NIC Bonding/Teaming or Switch Independent
Teaming
LACP Teaming LACP (Also called 802.3ad or Dynamic Link
Aggregation)
Other Other (Not recommended. Can have performance
impact on link management)

Note: LACP Fast timer is not currently supported.

3.8 Maximum Transmission Unit (MTU) behavior

The default MTU for SmartFabric OS10 interfaces is 1532 bytes. When a SmartFabric is created, the default
MTU for the switch is set to Jumbo (9216 bytes). This introduces the following behaviors:

1. If the MTU is not specifically set on a specific interface, the MTU will be 9216 bytes.
2. If the MTU has been specifically set on a specific interface, the MTU will be the value that has
been specified.
3. If a FCoE VLAN is assigned to an interface, the MTU will be set to 2500 bytes even if the
MTU had been manually set to a different value before the FCoE VLAN was assigned. It is
recommended that you set the MTU back to 9216 bytes after the FCoE VLAN has been
assigned

Please see Section 7.2 for instructions on setting the MTU.

3.9 Other restrictions and guidelines

The following additional restrictions and guidelines are in place when operating in SmartFabric mode:

1. Interconnecting switches in Slots A1/A2 with switches in Slots B1/B2 regardless of chassis is not
supported.
2. When operating with multiple chassis, switches in Slots A1/A2 or Slots B1/B2 in one chassis must be
interconnected only with other Slots A1/A2 or Slots B1/B2 switches respectively. Connecting switches
that reside in Slots A1/A2 in one chassis with switches in Slots B1/B2 in another is not supported.
3. Uplinks must be symmetrical. If one switch in a SmartFabric has two uplinks, the other switch must have
two uplinks of the same speed.
4. You cannot have a pair of switches in SmartFabric mode uplink to another pair of switches in SmartFabric
mode. A SmartFabric can uplink to a pair of switches in Full Switch mode.
5. VLANs 4001 to 4020 are reserved for internal switch communication and must not be assigned to an
interface.

28 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 6. In SmartFabric mode, although you can use the CLI to create VLANs 1 to 4000 and 4021 to 4094, you
cannot assign interfaces to them. For this reason, do not use the CLI to create VLANs in SmartFabric
mode.
7. VLAN 1 is automatically created as the Default/Native VLAN, but it is not required to be used. See
Section 4.2 for more information.
8. When using LACP NIC teaming, the LACP timer must be set to slow.
9. If you assign OME-M to a specific VLAN (VLAN 500, for example), do not create VLAN 500 on the fabric.

29 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

4 Creating a SmartFabric
The general steps required to create a SmartFabric are:

1. Physically cable the MX chassis and upstream switches.

2. Define the VLANs.
3. Create the SmartFabric.
4. If needed, configure uplink port speed and breakout.
5. Create the Ethernet uplink.
6. Configure the upstream switch and connect uplink cables.

These steps make the following assumptions:

• All MX7000 chassis and management modules are cabled correctly and in a Multi-Chassis
Management group.
• The VLTi cables between switches have been connected.
• Open Manage Enterprise - Modular is at version 1.10.00 and OS10 is version 10.5.0.1

Note: All server, network, and chassis hardware has been updated to the latest firmware. See Appendix D for the
minimum recommended firmware versions.

4.1 Physically cable MX chassis and upstream switches

The first step in creating the SmartFabric is to cable the MX chassis and upstream switches.

• For Management Module cabling, see Dell EMC PowerEdge MX Networking Architecture Guide.
• For VLTi cabling of different IOM placements, see Figure 8, Figure 9, and Figure 10.

For information on cabling the MX chassis to the upstream switches, see the example topologies in Section
10 in this document.

For further information on cabling PowerEdge MX in general, see Dell EMC PowerEdge MX Networking
Architecture Guide.

4.2 Define VLANs

Before creating the SmartFabric, the initial set of VLANs should be created. The first VLAN to be created
should be the default, or native VLAN, typically VLAN 1. The default VLAN must be created for any untagged
traffic to cross the fabric.

Note: VLAN1 will be created as a Default VLAN when the first fabric is created.

To define VLANs using the OME-M console, perform the following steps:

1. Open the OME-M console.

2. From the navigation menu, click Configuration > Networks.
3. In the Network pane, click Define.
4. In the Define Network window, complete the following:
• Enter a name for the VLAN in the Name box. In this example, VLAN0010 was used.
• Optionally, enter a description in the Description box. In this example, the description was
entered as “Company A General Purpose”.

30 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 • Enter the VLAN number in the VLAN ID box. In this example, 10 was entered.
• From the Network Type list, select the desired network type. In this example, General
Purpose (Bronze) was used.
• Click Finish.

Defined VLAN list

Figure 13 shows VLAN 1 and VLAN 10 after being created using the steps above.

4.2.1 Define VLANs for FCoE

Note: Define VLANs for FCoE, if implementing Fibre Channel configurations as per requirement.

A standard Ethernet uplink carries assigned VLANs on all physical uplinks. When implementing FCoE, traffic
for SAN path A and SAN path B must be kept separate. The storage arrays have two separate controllers
which creates two paths SAN path A and SAN path B connected to MX9116n FSE. For storage traffic to be
redundant, two separate VLANs are created for that traffic.

Using the same process as above, create two additional VLANs for FCoE traffic.

VLAN attributes
Name Description Network Type VLAN ID SAN

FC A1 FCOE A1 Storage - FCoE 30 A

FC A2 FCOE A2 Storage - FCoE 40 B

Figure 14 Shows the two new VLANs.

31 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Defined VLAN list

Note: To create VLANs for FCoE, From the Network Type list, select Storage – FCoE, and then click Finish.
VLANs to be used for FCoE must be configured as the Storage – FCoE network type.

4.3 Create the SmartFabric

To create a SmartFabric using the OME-M console, perform the following steps:

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.
3. In the Fabric pane, click Add Fabric.
4. In the Create Fabric window, complete the following:
• Enter a name for the fabric in the Name box. In this example, SmartFabric was entered.
• Optionally, enter a description in the Description box. In this example, the description was
entered as “SmartFabric using MX9116n/MX7116n in Fabric A”.
• Click Next.
• From the Design Type list, select the appropriate type. In this example, “2x MX9116n Fabric
Switching Engine in different chassis” was selected.
• From the Chassis-X list, select the first MX7000 chassis.
• From the Switch-A list, select Slot-IOM-A1.
• From the Chassis-Y list, select the second MX7000 chassis to join the fabric.
• From the Switch-B list, select Slot-IOM-A2.
• Click Next.
• On the Summary page, verify the proposed configuration and click Finish.

Note: From the Summary window a list of the physical cabling requirements can be printed.

32 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 SmartFabric deployment design window

The SmartFabric deploys. This process can take several minutes to complete. During this time all related
switches will be rebooted, and the operating mode changed to SmartFabric mode.

Note: After the fabric is created, the fabric health will be critical until at least one uplink is created.

Figure 16 shows the new SmartFabric object and some basic information about the fabric.

SmartFabric post-deployment without defined uplinks

4.4 Configure uplink port speed or breakout, if needed

If the uplink ports need to be reconfigured to a different speed or breakout setting from the default, you must
do that before creating the actual uplink.

To configure the Ethernet breakout on port groups using OME-M Console, perform the following steps:

1. Open the OME-M console.

2. From the navigation menu, click Devices > I/O Modules.
3. Select the switch you want to manage. In this example, a MX9116n FSE in slot IOM-A1 is selected.
4. Choose Hardware > Port Information.
5. In the Port Information pane, choose the desired port-group. In this example port-group1/1/13 is
selected.

33 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Note: Prior to choosing the breakout type, you must set the Breakout Type to HardwareDefault and then set the
desired configuration. If the desired breakout type is selected prior to setting HardwareDefault, an error will occur.

6. Choose Configure Breakout. In the Configure Breakout dialog box, select HardwareDefault.
7. Click Finish.

First set the breakout type to HardwareDefault

8. Once the job is completed, choose Configure Breakout. In the Configure Breakout dialog box,
select the required Breakout Type. In this example, the Breakout Type for port-group1/1/13 is
selected as 1x40GE. Click Finish.

Select the desired breakout type

9. Configure the remaining breakout types on additional uplink port groups as needed.

4.5 Create Ethernet uplink

Note: To change the port speed or breakout configuration, see Section 4.4 and make those changes before
creating the uplinks.

After initial deployment, the new fabric shows Uplink Count as ‘zero’ and shows a warning ( ). The lack of a
fabric uplink results in a failed health check ( ). To create the uplink, follow these steps:

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.

34 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 3. Click on the fabric name. In this example, SmartFabric is selected.
4. In the Fabric Details pane, click Uplinks.
5. Click on the Add Uplinks button.
6. In the Add Uplink window complete the following:
• Enter a name for the uplink in the Name box. In this example, Uplink01 is entered.
• Optionally, enter a description in the Description box.
• From the Uplink Type list, select the desired type of uplink. In this example,
Ethernet is selected.
• Click Next.
• From the Switch Ports list, select the uplink ports on both the Mx9116n FSEs. In
this example, ethernet 1/1/41 and ethernet 1/1/42 are selected for both MX9116n
FSEs.
• From the Tagged Networks list, select the desired tagged VLANs. In this example,
VLAN0010 is selected.
• From the Untagged Network list, select the untagged VLAN. In this example,
VLAN0001 is selected.

Create Ethernet uplink

7. Click Finish.

At this point, SmartFabric creates the uplink object and the status for the fabric changes to OK .

35 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Note: VLAN1 will be assigned to Untagged Network by default.

4.6 Configure Fibre Channel universal ports

Note: Configure Fibre Channel universal ports, if implementing Fibre Channel configurations as per requirement.

On the MX9116n FSE, port-group 1/1/15 and 1/1/16 are universal ports capable of connecting to FC devices
at a variety of speeds depending on the optic being used. In this example, we are configuring the universal
port speed as 4x16G FC. To enable FC capabilities, perform the following steps on each MX9116n FSE.

Note: Port-group 1/1/16 is used for FC connections in this example.

1. Open the OME-M console.

2. From the navigation menu click Devices, then click I/O Modules.
3. In the Devices panel, click to select the IOM to configure.
4. In the IOM panel click Hardware, then click Port Information.

Note: See SmartFabric mode - MX Port-Group Configuration Errors video for more information on configuration
errors.

5. Click the port-group 1/1/16 check box, then click Configure breakout.
6. In Configure breakout panel, select HardwareDefault as the breakout type.
7. Click Finish.
8. To set the port group 1/1/16 to 4X16GFC, select the port-group 1/1/16 check box, then click
Configure breakout.
9. In Configure breakout panel, select 4X16GFC as the breakout type.
10. Click Finish.

Note: When enabling Fibre Channel ports, they are set administratively down by default. Select the ports and
click Toggle Admin State button. Click Finish to administratively set the ports to up.

4.7 Create Fibre Channel uplinks

Note: Create Fibre Channel Uplinks VLANs for FCoE, if implementing Fibre Channel configurations as per
requirement.

Note: The steps in this section allow you to connect to an existing FC switch via NPG mode, or directly attach a
FC storage array. The uplink type is the only setting within the MX chassis that distinguishes between the two
configurations.

To create uplinks, follow these steps:

1. Open the OME-M console.

2. From the navigation menu click Devices, then click Fabric.
3. Click the SmartFabric fabric name.
4. In the Fabric Details panel, click Uplinks, then click the Add Uplinks button.

36 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 5. From the Add Uplinks window, use the information in Table 8 to enter an uplink name in the Name
box.
6. Optionally, enter a description in the Description box.
7. From the Uplink Type list, select Type, then click Next. In this example, FCoE as Uplink type is
selected. Choose Uplink type as per your configuration from FC Gateway, FC Direct Attach or FCoE
options.
8. From the Switch Ports list, select the FC ports as defined in Table 8. Select appropriate port for the
connected uplink.
9. From the Tagged Networks list, select VLAN defined in Table 8, then click Finish. SmartFabric creates
the uplink object and the status for the fabric changes to OK.

For the example shown here in Section 10.3 and 10.4, The uplink attributes are defined here.

Uplink attributes
Uplink Name Description Ports VLAN (Tagged)

FC Uplink for Switch model

FCoE A1 30
switch in Slot A1 dependent

FC Uplink for Switch model

FCoE A2 40
switch in Slot A2 dependent

Note: Do not assign the same FCoE VLAN to both switches. They must be kept separate.

4.8 Configuring the upstream switch and connect uplink cables

The upstream switch ports must be configured in a single LACP LAG. This document provides six example
configurations. The first three example configurations are for ethernet uplinks and other three scenarios are
for storage.

• Scenario 1: SmartFabric deployment with Dell EMC PowerSwitch Z9100-ON upstream switches
• Scenario 2: SmartFabric connected to Cisco Nexus 3232C switches
• Scenario 3: Connect MX9116n FSE to Fibre Channel Storage – NPIV Proxy gateway mode
• Scenario 4: Connect MX9116n FSE to Fibre Channel Storage – FC Direct attach mode
• Scenario 5: Connect MX5108n to Fibre Channel Storage - FSB mode
• Scenario 6: Configure Boot from SAN

37 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

5 Deploying a server
Before beginning, ensure that all server firmware, especially the NIC/CNA, has been updated to the latest
version.

5.1 Server preparation

The examples in this guide use a reference server of a Dell EMC PowerEdge MX740c compute sled with
QLogic (model QL41262HMKR) Converged Network Adapters (CNAs) installed. CNAs are required to
achieve FCoE connectivity. Use the steps below to prepare each CNA by setting them to factory defaults (if
required) and configuring NIC partitioning (NPAR) if needed. Not every implementation requires NPAR.

Note: iDRAC steps in this section may vary depending on hardware, software and browser versions used. See
the Installation and Service Manual for your PowerEdge server for instructions on connecting to the iDRAC.
From the link, select your server, then Manuals and documents.

5.1.1 Reset server CNAs to factory defaults (if required)

Note: If you are running the 14.10.xx firmware version, you must first update the firmware version to the newest
release.Because when a Qlogic QL41262 is reset to default, and incorrect value for Energy Efficient Ethernet
(EEE) is set to "Balanced" mode. The correct value for this should be "Disabled", and this is set correctly at
factory. This only occurs on 14.10.xx firmware.
The results of this setting causes the FCOE link not to establish even after the reboots.
To avoid this, you can extract Winfwnx2.exe, Tcl85.dll, QLMAPla.dll files from Windows Firmware DUP, and
change the Energy Efficient Ethernet parameter from 1 to 0.
To achieve that use the utility to run the following commands:
cfg -show eee_power_mode to show EEE mode.
cfg -eee_power_mode 0 to change the mode.
This will resolve the issue.

Reset the CNAs to their factory defaults using the steps in this section. Resetting CNAs to factory default is
only necessary if the CNAs installed have been modified from their factory default settings.

1. From the OME-M console, select the server to use to access the storage.
2. Launch the server Virtual Console.
3. From the Virtual Console, select Next Boot then BIOS Setup.
4. Reboot the server.
5. From the System Setup Main Menu, select Device Settings.
6. From the Device Settings page, select the first CNA port.
7. From the Main Configuration page, click the Default button.
8. Click Yes to load the default settings, and then click OK.
9. Click Finish. Notice if a message indicates a reboot is required for changes to take effect.
10. Click Yes to save changes, then click OK.
11. Repeat the steps in this section for each CNA port listed on the Device Settings page.

If required per step 9, reboot the system and return to System Setup to configure NIC partitioning.

5.1.2 Configure NIC partitioning on CNAs

In this example, each QLogic CNA port is partitioned into one Ethernet and one FCoE partition.

38 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Note: This is only done on CNA ports that carry converged traffic. In this example, these are the two 25GbE
QLogic CNA ports on each server that attach to the fabric internally.

If the system is already in System Setup from the previous section, skip to step 4.

1. Using a web browser, connect to the iDRAC server and launch the Virtual Console.
2. From the Virtual Console, click Next Boot menu then select BIOS Setup.
3. Select the option to reboot the server.
4. On the System Setup Main Menu, select Device Settings.
5. Select the first CNA port.
6. Select Device Level Configuration.
7. Set the Virtualization Mode to NPAR, if not already set, and then click Back.
8. Select NIC Partitioning Configuration, Partition 1 Configuration, and click to set the NIC +
RDMA Mode to Disabled.
9. Click Back.

CNA partition 1 configuration

10. Select Partition 2 Configuration and set the NIC Mode to Disabled.
11. Set the FCoE Mode to Enabled, then click Back.

CNA partition 2 configuration

12. If present, select Partition 3 Configuration and set all modes to Disabled, then click Back.
13. If present, select Partition 4 Configuration and set all modes to Disabled, then click Back.
14. Click Back, and then Finish.
15. When prompted to save changes, click Yes and then click OK in the Success window.

39 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 16. Select the second CNA port and repeat steps in this section for port 2
17. After configuring port 2, click Finish, then Finish.
18. Click Yes to exit and reboot.

5.1.3 Configure other server settings

Depending on application and infrastructure requirements, also configure other server’s settings such as
BIOS, RAID, iDRAC as appropriate.

5.2 Create a server template

A server template contains parameters extracted from a server and allows these parameters to be quickly
applied to multiple compute sleds. A server template contains all server settings for a specific deployment
type including BIOS, iDRAC, RAID, NIC/CNA, and so on. The template is captured from a reference server
and can then be deployed to multiple servers at the same time. The server template allows an administrator
to associate VLANs to compute sleds.

The templates contain settings for the following categories:

• Local access configuration

• Location configuration
• Power configuration
• Chassis network configuration
• Slot configuration
• Setup configuration

Before creating the template, select a server to be the reference server and configure the hardware to the
exact settings required for the implementation.

Note: In SmartFabric mode, you must use a template to deploy a server and to configure networking.

To create a server template, follow these steps:

1. Open the OME-M console.

2. From the navigation menu, click Configuration, then click Deploy.
3. From the center panel, click Create Template, then click From Reference Device to open
the Create Template window.
4. In the Template Name box, enter a name. In this example, “MX740c with FCOE CNA ” is
entered.

40 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Create Template dialog box

5. Optionally, enter a description in the Description box, then click Next.

6. In the Device Selection section, click Select Device.
7. From the Select Devices window, choose the server previously configured, then click Finish.
8. From the Elements to Clone list, select all the elements, and then click Finish.

Select the elements to clone

A job starts and the new server template displays on the list. When complete, the Completed successfully
status displays.

41 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

5.3 Create identity pools
Dell EMC recommends the use of identity pools. Virtual identity pools are used in conjunction with server
templates to automate network onboarding of compute sleds. Follow these steps to create an ID pool:

1. Open the OME-M console.

2. From the navigation menu, click Configure, then click Identity Pools.
3. In the Network panel, click Create. The Create Identity Pool window displays.
4. Type Ethernet CNA into the Pool Name box.
5. Optionally, enter a description in the Description box.
6. Click Next.
7. Click to select the Include Ethernet Virtual MAC Addresses option.
8. In the Starting MAC Address box, type a unique MAC address (e.g.
06:3C:F9:A4:CC:00).
9. Type 255 in the Number of Virtual MAC Identities box, click Next, then click Next
again.
10. Select the Include FCoE Identity option if using FCoE
11. In the Starting MAC Address box, type a unique MAC address (e.g.
06:3C:F9:A4:CD:00).
12. Type 255 in the Number of FCoE identities box for FCoE scenarios.

Include FCoE Identity

13. Click Finish, then click Finish again.

5.4 Associate server template with networks – no FCoE

After successfully creating a new template, associate the template with a network:

1. From the Deploy pane, select the template to be associated with VLANs. In this example, MX740c
with Intel mezzanine server template is selected.
2. Click Edit Network.
3. In the Edit Network window, complete the following:
• Optionally, from the Identity Pool list, choose the desired identity pool. In this example,
Ethernet ID Pool is selected
• Optionally, from the NIC Teaming option, choose the desired NIC Teaming option.

42 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 • The NIC teaming option can be selected from No Teaming, LACP teaming and Other as
mentioned in Table 6.
• For both ports, from the Untagged Network list, select the untagged VLAN. In this example,
VLAN0001 is selected.
• For both ports, from the Tagged Network list, select the tagged VLAN. In this example,
VLAN0010 is selected.
• Click Finish.

Figure 25 shows the associated networks for the server template.

Server template network settings

5.5 Associate Server Template with networks - FCoE

After successfully creating a new template, associate the template with a network:

1. From the Deploy pane, select the template to be associated with VLANs. In this example, MX740c
with FCoE CNA server template is selected.
2. Click Edit Network.
3. In the Edit Network window, complete the following:
• To choose FCoE VLANs, From identity pool list, Choose Ethernet CNA.
• For NIC in Mezzanine 1A Port 1, from the Tagged Network list, Choose FC A1.
• For NIC in Mezzanine 1A Port 2, from the Tagged Network list, Choose FC A2.
• Click Finish.

Figure 26 shows the associated networks for the server template.

43 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Server Template Network Settings

5.6 Deploy a server template

To deploy the server template, complete the following steps:

1. From the Deploy pane, select the template to be deployed. In this example, MX740c with FCOE CNA
server template is selected.
2. Click Deploy Template.
3. In the Deploy Template window, complete the following:

• Click the Select button to choose which slots or compute sleds to deploy the template to.
• Select the Do not forcefully reboot the host OS option.
• Click Next. Choose Run Now. Click Finish.

The interfaces on the switch are updated automatically. SmartFabric configures each interface with an
untagged VLAN and any tagged VLANs. Additionally, SmartFabric deploys associated QoS settings. See
Section 2.7 for more information.

To monitor the deployment progress, go to Monitor > Jobs > Select Job > View Details. This will show the
progress of Deployment of Server Template.

44 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Job displaying Deployment of Server Template

45 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

6 SmartFabric operations

6.1 Viewing the fabric

A SmartFabric is viewed using OME-M. The green check mark adjacent to the fabric name informs that the
status of the fabric is healthy. In this example, the fabric created is named Fabric01.

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.
3. To view the Fabric components, select the fabric. This can also be achieved by clicking the View
Details button the right.

SmartFabric

Fabric components include Uplinks, Switches, Servers and ISL Links. Uplinks connect the MX9116n
switches with upstream switches. In this example, the uplink is named as Uplink1.

Uplinks

46 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Switches are the I/O Modules that are part of the fabric. In this example, the fabric has two MX9116n
switches.

Note: Fabric Expander Modules are transparent and therefore do not appear on the Fabric Details page.

Switches

Servers are the compute sleds that are part of the fabric. In this example, two PowerEdge MX740c compute
sleds are part of the fabric.

Servers

ISL Links are the VLT interconnects between the two switches. The ISL links should be connected on port
groups 11 and 12 on MX9116n switches, and ports 9 and 10 on MX5108n switches. This is a requirement
and failure to connect the defined ports will result in a fabric validation error.

47 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 ISL Links

6.2 Editing the fabric

A fabric has four components:

• Uplinks
• Switches
• Servers
• ISL Links

Editing the fabric discussed in this section includes editing the fabric name and description.

To edit the name of the fabric that was created, follow the steps below:

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.
3. On the right, Click the Edit button.

Edit fabric name and description

48 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 4. In the Edit Fabric dialog box, change the name and description as desired. Click Finish.

Edit Fabric dialog box

6.3 Editing uplinks

Editing the uplinks on the created fabric is done using the following steps:

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.
3. Select the fabric.
4. Select the Uplink to edit and click Edit. In this example, Uplink1 is selected.
5. In the Edit Uplink dialog box, modify the Name and Description as desired.

Note: The uplink type cannot be modified once the fabric is created. If the uplink type needs to be changed after
the fabric is created, delete the uplink and create a new uplink with the desired uplink type.

Edit Uplink dialog box

6. Click Next.

49 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 7. Edit the uplink ports on the MX switches that connects to the upstream switches. In this example,
ports 41 and 42 on the MX9116n switches that connects to upstream switches are displayed.

Note: Care should be taken to modify the uplink ports on both MX switches. Select the IOM to display the
respective uplink switch ports.

Edit uplink ports and VLAN networks

8. If desired, modify the tagged and untagged VLANs.

9. Click Finish.

6.4 Editing VLANs on a deployed server

The OME-M Console is used to add/remove VLANs on the deployed servers in a SmartFabric. The following
illustrates the steps to add/remove VLANs on the deployed servers.

1. Open OME-M Console.

2. From the navigation menu, click Devices > Fabric.
3. Select the fabric.
4. Select Servers from the left pane.

50 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Add/remove VLANs

5. Choose the desired server. In this example PowerEdge MX740C with service tag 8XQP0T2 is
selected.
6. Choose Edit Networks.
7. Choose NIC teaming from LACP, No Teaming and Other options.
8. Modify the VLAN selections as required by defining the tagged and untagged VLANs.
9. Select VLANs on Tagged and Untagged Network for each Mezzanine card port.
10. Click Save.

Modify VLANs

Note: At this time, only one server can be selected at a time in the GUI.

6.5 Managing Fibre Channel Zoning on the MX9116n FSE

When a storage array is directly connected to the MX9116n FSE, Fibre Channel Zones can be used to
improve security and performance.

Preparation of servers will be same as mentioned in Section 5. Determine the FC WWPNs for the compute
sleds and storage array as discussed in Appendix B.

These examples assume that the storage array has been successfully connected to the MX9116n FSE’s FC
uplinks and there are no errors.

Below are examples of the steps and commands to configure FC Zoning. For more information on Dell EMC
SmartFabric OS10 Fibre Channel capabilities and commands, see the Dell EMC SmartFabric OS10 User
Guide.

Step 1: Configure FC aliases for server and storage adapter WWPNs.

51 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 A FC Alias is a human defined name that references a WWN. This allows users to refer to those devices by
the easy to remember alias instead of the long WWN. In this example, aliases for two MX740c compute sleds
and a Dell EMC Unity storage array is defined.

The WWNs for the servers are obtained via OME-M console.

MX9116n-A1 MX9116n-A2

configure terminal configure terminal

fc alias mx740c-1p1 fc alias mx740c-1p2

member wwn 20:01:00:0E:1E:09:A2:3A member wwn 20:01:00:0E:1E:09:A2:3A

fc alias mx740c-2p1 fc alias mx740c-2p2

member wwn 20:01:00:0E:1E:09:B8:F6 member wwn 20:01:00:0E:1E:09:B8:F7

fc alias SpA-0 fc alias SpA-1

member wwn 50:06:01:66:47:E0:1B:19 member wwn 50:06:01:67:47:E0:1B:19

fc alias SpB-0 fc alias SpB-1

member wwn 50:06:01:6E:47:E0:1B:19 member wwn 50:06:01:6F:47:E0:1B:19

Step 2: Create FC zones

Server and storage adapter WWPNs, or their aliases, are combined into zones to allow communication
between devices in the same zone. Dell EMC recommends single-initiator zoning. In other words, no more
than one server HBA port per zone. For high availability, each server HBA port should be zoned to at least
one port from SP A and one port from SP B. In this example, one zone is created for each server HBA port.
The zone contains the server port and the two storage processor ports connected to the same MX9116n FSE.

MX9116n-A1 MX9116n-A2

fc zone mx740c-1p1zone fc zone mx740c-1p2zone

member alias-name mx740c-1p1 member alias-name mx740c-1p2
member alias-name SpB-0 member alias-name SpB-1
member alias-name SpA-0 member alias-name SpA-1

fc zone mx740c-2p1zone fc zone mx740c-2p2zone

member alias-name mx740c-2p1 member alias-name mx740c-2p2
member alias-name SpB-0 member alias-name SpB-1
member alias-name SpA-0 member alias-name SpA-1

Step 3: Create a zoneset

A zone set is a collection of zones. A zone set named zoneset1 is created on each switch, and the zones are
added to it.

MX9116n-A1 MX9116n-A2

fc zoneset zoneset1 fc zoneset zoneset1

member mx740c-1p1zone member mx740c-1p2zone

52 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 member mx740c-2p1zone member mx740c-2p2zone
exit exit

write memory write memory

6.6 Connecting non-MX Ethernet Devices to the fabric

As of SmartFabric OS10.5.0.1 and OME-M 1.10.00, PowerEdge MX Ethernet switches allow the connection
of non-MX devices such as rack servers to the fabric as long as that device provides a physical interface
supported by the switch. Once connected, VLANs must be assigned to each port that the device is connected
to. This capability does not allow non-MX devices to support FCoE.

To connect a non-MX device to a switch running in SmartFabric mode, perform the following steps:

1. Open the OME-M Console. To Configure Breakout on port-group, refer to Section 4.4.
2. Once the Breakout on Port-group is done, select the port. Click Edit VLANs. Make sure the port is
not in use for any other purpose.
3. Click Edit VLANs and select Default VLAN 1 as shown as Untagged Network in the example below
and choose any other VLAN for Tagged Network.

Choose VLANs under Edit VLANs

4. Click Finish.
5. Repeat these steps for any other port or IOM

6.7 Delete SmartFabric

To remove the SmartFabric using the OME-M console, perform the following steps:
1. Open the OME-M console.
2. From the navigation menu, click Devices > Fabric.
3. Select SmartFabric.
4. Click the Delete button.
5. In the delete fabric dialog box click Yes.

All participating switches reboot to Full Switch mode.

53 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Note: Any configuration not completed by the OME-M console is lost when switching between IOM operating
modes.

6.8 Module replacement process in SmartFabric

Dell EMC PowerEdge MX platform gives you an option to replace the I/O Modules in a SmartFabric mode in
case of any errors or failures. MX9116n FSE and MX5108n can be replaced with another same type of I/O
Module. In case of errors or failures, replace the old IOM with the new IOM using Linux command.

For Module Replacement process, follow the steps:

1. Physically replace the IOM.

2. Identify the master IOM in a fabric.
3. Execute the process of module replacement from the master IOM using the module replacement
command.

6.8.1 Physically replace the IOM

Follow the below instructions to physically replace IOM:

• Identify the faulty IOM to replace. Disconnect the cables connected to faulty IOM.
• Insert the new IOM in the same slot as the failed IOM. The IOM should be in SmartFabric and must
be up and running.
• The model of the IOM should be same.
• The new IOM should have the same version of SmartFabric OS10 as the old IOM.
• Confirm if the new IOM has been recognized by OME-M before proceeding further.

6.8.2 Identify the master IOM in a fabric

To identify the master IOM in the fabric, run sfs_master_details.py command on the Linux shell prompt
the IOM. To run the above command, follow the instructions below:

1. Login to the Linux shell

OS10# system bash

admin@MX9116n-A1#:~$

2. Go to the bin directory on SmartFabric OS10 by running cd /opt/dell/os10/bin command. This

directory will have all the python scripts to run for the module replacement process.
• Identify the master IOM by running sfs_master_details.py.

admin@MX9116N-A1:/opt/dell/os10/bin$sfs_master_details.py

• If you run the command on master IOM, the output will be I am the Master. If you run the
command on other member IOMs it will return the IPv6 address of master IOM.

For Example:

The console displays the following output on Master IOM:

admin@MX9116N-A1:/opt/dell/os10/bin$sfs_master_details.py
I am the Master

54 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 The console displays the following output on Member IOM:

admin@MX9116N-A1:/opt/dell/os10/bin$sfs_master_details.py
Master IP Address: fde1:53ba:e9a0:de14:2204:fff:fe01:cd90

• To login to master IOM from member IOM, run ssh admin@Master IP Address command.

6.8.3 Execute process of Module Replacement with Linux command

To replace an old IOM with the new IOM and on board the new IOM, run the following command in the same
bin directory on SmartFabric OS10.

admin@MX9116N-A1:/opt/dell/os10/bin$ sfs_node_replace.py NewServiceTag

OldServiceTag

NewServiceTag – Service tag for the new IOM

OldServiceTag – Service tag for the old IOM

You can find the Service tag by running show license status command on IOM.

The system displays the following error when the IOM is not part of the SmartFabric:

Enter the Username for the Admin: admin

Password:

No Fabric found for specified nodes. Please recheck and issue this command
again.

If IOM is part of the Fabric, the module replacement process will be initiated.

For Example:

admin@MX9116N-A1#:~$ sfs_node_replace.py C23RPK2 8XRK0T2

Enter the Username for the Admin: admin
Password:

% Total % Received % Xferd Average Speed Time Time Time Current

Dload Upload Total Spent Left Speed
100 132 100 79 100 43 632 370 --:--:-- --:--:-- --:--:-- 636

Node replacement work-flow is initiated, the node C23RPK2 will reboot into
Fabric mode.

55 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

7 Switch operations
PowerEdge MX switches can be managed using the OME-M console. From the Switch Management page,
you can view activity, health, and alerts, as well as perform operations such as power control, firmware
update, and port configuration. Some of these operations can also be performed in Full Switch mode.

7.1 Switch management page overview

To get to the switch management page, follow these steps:

1. Open the OME-M console

2. From the navigation menu, click Devices > I/O Modules.
3. Select the desired switch. In this example, MX9116n FSE IOM-A1 is selected.

IOM Overview page on OME-M

7.1.1 Switch overview

The Overview page provides a convenient location to view pertinent data on the IOM such as:

• Chassis information
• Recent Alerts
• Recent Activity
• IOM Subsystems

56 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 • Environment

The Power Control drop-down button provides three options:

• Power Off: Turns off the IOM

• Power Cycle: Power cycles the IOM
• System Reseat: Initiates a cold reboot of the IOM

Power Control button

The Blink LED drop down button provides an option to turn on or turn off the ID LED on the IOM. To turn on
the ID LED, choose:

• Blink LED > Turn On

This activates a blinking blue LED and provides easy identification. To turn off the blinking ID LED, choose:

• Blink LED > Turn Off

Blink LED button

57 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

7.1.2 Hardware tab
The Hardware tab provides information on the following IOM hardware:

• FRU
• Device Management Info
• Installed Software
• Port Information

Hardware Tab

The Port Information provides useful operations such as:

• Configuring port-group breakout

• Toggling the admin state of ports
• Configuring MTU of ports
• Toggling Auto Negotiation

Port Information

58 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

7.1.3 Firmware tab
The Firmware tab provides options to manage the firmware on the IOM. The Dell Update Package (DUP) file
is used to update the firmware of the IOM.

Firmware Tab

7.1.4 Alerts tab

The Alert tab provides information on alerts and notifies the administrator. The advanced filter option can be
leveraged to quickly filter out alerts. Various operations can be performed on an alert or several alerts such
as:

• Acknowledge
• Unacknowledged
• Ignore
• Export
• Delete

Alerts Tab

59 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

7.1.5 Settings tab
The Settings tab provides options to configure the following settings for the IOMs:

• Network
• Management
• Monitoring
• Advanced Settings

Settings Tab

The Network option includes configuring IPv4, IPv6, DNS Server and Management VLAN settings.

Network Settings

60 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 The Management option includes setting the hostname and linuxadmin password.

Note: Although the GUI has the field name listed as Root Password, it denotes the linuxadmin password. For
logging on to the CLI of the MX switch, use default credentials with username as admin and password as admin.

Management Settings

Monitoring provides options for SNMP settings.

Monitoring Settings

The Advanced Settings tab offers the option for time configuration replication and alert replication. Select the
Replicate Time Configuration from Chassis check box to replicate the time settings configured in the
chassis to the IOM. Select the Replicate Alert Destination Configuration from Chassis check box to
replicate the alert destination settings configured in the chassis to the IOM.

61 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Advanced Settings

7.2 Configuring Ethernet ports

The MX switches can be accessed using the OME-M console. Various operations such as port breakout,
altering the MTU size, enabling/disabling auto negotiation etc. Follow the below steps to gain insight into
modifying various entities.

Note: In SmartFabric mode, configuring interfaces via the CLI should not be performed. Use the OME-Modular
GUI

1. From the switch management page, choose Hardware > Port Information.

IOM Overview page on OME-M

62 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Port information

2. To configure MTU, select the port listed under the respective port-group.
3. Click Configure MTU. Enter MTU size in bytes.
4. Click Finish.

Configure MTU

5. To configure Auto Negotiation, select the port listed under the respective port-group. Click Toggle
AutoNeg. This will change the Auto Negotiation of the port to Disabled/Enabled. Click Finish.

Enable/Disable Auto Negotiation

6. To configure the administrative state (shut/no shut) of a port, select the port listed under the
respective port-group. Click Toggle Admin State. This will toggle the port’s administrative state to
Disabled/Enabled state.
7. Click Finish.

63 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Toggle Admin State

7.3 Upgrade Dell EMC SmartFabric OS10

Upgrading the IOMs in the fabric can be done using the OME-M console. The upgrade is carried out using a
Dell Update Package (DUP). A DUP is a self-contained package format that updates a single element on a
system. Using DUPs, you can update a wide range of system components simultaneously and apply scripts
to similar sets of Dell systems to levels. The OS10 DUP is available for download from Support for Dell EMC
Products.

Download DUP file for MX9116n FSE

Note: If an IOM is in SmartFabric mode, all switches that are part of the fabric will be updated. Do not select both
switches in the fabric to be updated.

If an IOM is in Full Switch mode, the firmware upgrade is completed only on the specific IOMs selected.

To upgrade the IOMs that are part of a fabric, follow the steps below:

64 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 1. From the switch management page of one of the switches in the fabric, choose Firmware > Update
Firmware. In the Update Firmware dialog box, browse and select the appropriate DUP file.

Switch management page

Update firmware dialog box

2. Once the file is uploaded, select the check box next to the file and click Next.
3. Select Update Now and then click Finish.

65 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Schedule update

The firmware upgrade job can be monitored by navigating to Monitor > Jobs > Select Job > View Details.

View the job for more details.

66 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

8 Validating the SmartFabric deployment

8.1 View the MCM group topology

The OME-M console can be used to show the physical cabling of the SmartFabric.

1. Open the OME-M console.

2. In the left pane click View Topology.
3. Click the lead chassis and then click Show Wiring.
4. The icons can be clicked to show cabling.

Figure 61 shows the current wiring of the SmartFabric.

SmartFabric cabling

67 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

8.2 View the SmartFabric status
The OME-M console can be used to show the overall health of the SmartFabric.

1. Open the OME-M console.

2. From the navigation menu, click Devices > Fabric.
3. Select SmartFabric1 to expand the details of the fabric.

Figure 62 shows the details of the fabric.

Fabric status details

The Overview tab shows the current inventory, including switches, servers, and interconnects between the
MX9116n FSEs in the fabric. Figure 63 shows the SmartFabric switch in a healthy state. Figure 64 shows the
participating servers in a healthy state.

SmartFabric switch inventory

68 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 SmartFabric server inventory

Figure 65 shows the Topology tab and the VLTi created by the SmartFabric mode.

SmartFabric overview fabric diagram

Figure 66 displays the wiring diagram table from the Topology tab.

SmartFabric topology wiring diagram table

69 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

8.3 View port status
The OME-M console can be used to show the port status. In this example, the figure displays ports for an
MX9116n FSE.

1. Open the OME-M console.

2. From the navigation menu, click Devices > I/O Modules.
3. Select an IOM and click the View Details button to the right of the inventory screen. The IOM
overview for that device, displays.
4. From the IOM Overview, click Hardware.
5. Click to select the Port Information tab.

Figure 67 shows ethernet 1/1/1, 1/1/3, 1/71/1, and 1/72/1 in the correct operational status (Up). These
interfaces correspond to the MX740c compute sleds in slots 1 and 2 in both chassis. The figure also shows
the VLT connection (port channel 1000) and the uplinks (port channel 1) to the Z9100-ON leaf switches.

IOM Port Information

70 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

8.4 CLI commands

8.4.1 show switch-operating-mode

Use the show switch-operating-mode command to display the current operating mode:

C140A1# show switch-operating-mode

Switch-Operating-Mode : Smart Fabric Mode

8.4.2 show discovered-expanders

The show discovered-expanders command is only available on the MX9116n FSE and displays the
MX7116n FEMs service tag attached to the MX9116n FSEs and the associated port-group and virtual slot.

C140A1# show discovered-expanders

Service Model Type Chassis Chassis-slot Port-group Virtual
tag service-tag Slot-Id
--------------------------------------------------------------------------
D10DXC2 MX7116n 1 SKY002Z A1 1/1/1 71
FEM

8.4.3 show unit-provision

The show unit-provision command is only available on the MX9116n FSE and displays the unit ID
and the provision and discovered name of the MX7116n FEM attached to the MX9116n FSE.

C140A1# show unit-provision

Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+---------------------------------+-------|
1 | 71 | D10DXC2 | D10DXC2 | up |

8.4.4 show lldp neighbors

The show lldp neighbors command shows information about LLDP neighbors. The iDRAC in
PowerEdge MX compute sleds produce LLDP topology packets that contain specific information that the
SmartFabric Services engine uses to determine the physical network topology regardless if a switch is in Full
Switch or SmartFabric modes. For servers connected to switches in SmartFabric mode, the iDRAC LLDP
topology feature is required. Without it, the fabric will not recognize the compute sled and the user will not be
able to deploy networks to the sled.

The iDRAC MAC address can be verified by selecting iDRAC Settings > Overview > Current Network
Settings from the iDRAC GUI of a compute sled. An example is shown as follows:

IOM Port Information

71 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Alternately the iDRAC MAC information can be obtained from the System Information on the iDRAC
Dashboard page.

IOM Port Information

Subsequently, viewing the LLDP neighbors shows the iDRAC MAC address in addition to the NIC MAC
address of the respective mezzanine card.

C160A1# show lldp neighbors

Loc PortID Rem Host Name Rem Port Id Rem Chassis Id
--------------------------------------------------------------------------------
ethernet1/1/1 Not Advertised 98:03:9b:65:73:b2 98:03:9b:65:73:b4
ethernet1/1/1 iDRAC-8XQP0T2 8XQP0T2 NIC.Mezzanine.1A-1-1 d0:94:66:87:ab:40

---- OUTPUT TRUNCATED -----

In the example deployment validation of LLDP neighbors, Ethernet1/1/1, ethernet 1/1/3, and
ethernet 1/1/71-1/1/72 represent the two MX740c sleds in one chassis. The first entry is the iDRAC
for the compute sled. The iDRAC uses connectivity to the mezzanine card to advertise LLDP information. The
second entry is the mezzanine card itself.

Ethernet 1/71/1 and ethernet 1/71/2 represent the MX740c compute sleds connected to the
MX7116n FEM in the other chassis.

Ethernet range ethernet1/1/37-1/1/40 are the VLTi interfaces for the SmartFabric. Last,
ethernet1/1/41-1/1/42 are the links in a port channel connected to the Z9100-ON leaf switches.

C140A1# show lldp neighbors

Loc PortID Rem Host Name Rem Port Id Rem Chassis Id
----------------------------------------------------------------------------
ethernet1/1/1 iDRAC-CBMP9N2 CBMP9N2 NIC.Mezzanine.1A-1-1 d0:94:66:2a:07:2f
ethernet1/1/1 Not Advertised 24:6e:96:9c:e3:50 24:6e:96:9c:e3:50
ethernet1/1/3 iDRAC-1S35MN2 1S35MN2 NIC.Mezzanine.1A-1-1 d0:94:66:29:fa:f4
ethernet1/1/3 Not Advertised 24:6e:96:9c:e5:48 24:6e:96:9c:e5:48
ethernet1/1/37 C160A2 ethernet1/1/37 20:04:0f:00:a1:9e

72 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 ethernet1/1/38 C160A2 ethernet1/1/38 20:04:0f:00:a1:9e
ethernet1/1/39 C160A2 ethernet1/1/39 20:04:0f:00:a1:9e
ethernet1/1/40 C160A2 ethernet1/1/40 20:04:0f:00:a1:9e
ethernet1/1/41 Z9100-Leaf1 ethernet1/1/3 4c:76:25:e8:f2:c0
ethernet1/1/42 Z9100-Leaf2 ethernet1/1/3 4c:76:25:e8:e8:40
ethernet1/71/1 Not Advertised 24:6e:96:9c:e5:d8 24:6e:96:9c:e5:d8
ethernet1/71/1 iDRAC-CF52XM2 CF52XM2 NIC.Mezzanine.1A-1-1 d0:94:66:29:fe:b4
ethernet1/71/2 Not Advertised 24:6e:96:9c:e5:da 24:6e:96:9c:e5:da
ethernet1/71/2 iDRAC-1S34MN2 1S34MN2 NIC.Mezzanine.1A-1-1 d0:94:66:29:ff:27

8.4.5 show qos system

The show qos system command displays the QoS configuration applied to the system. The command is
useful to verify the service policy created manually or automatically by a SmartFabric deployment.

C140A1# show qos system

Service-policy (input): PM_VLAN
ETS Mode : off

8.4.6 show policy-map

Using the service policy from show qos system, the show policy-map type qos PM_VLAN
command displays QoS policy details including associated class maps, for example, CM10, and QoS queue
settings, qos-group 2.

C140A1# show policy-map type qos PM_VLAN

Service-policy (qos) input: PM_VLAN
Class-map (qos): CM10
set qos-group 2

8.4.7 show class-map

The command show class-map displays details for all the configured class-maps. For example, the
association between CM10 and VLAN 10 is shown.

C140A1# show class-map

Class-map (application): class-iscsi
Class-map (qos): class-trust
Class-map (qos): CM10(match-any)
Match: mac vlan 10
Class-map (qos): CM2(match-any

8.4.8 show vlt domain-id

The show vlt domain-id command validates the VLT configuration status. The role of one switch in
the VLT pair is primary (not shown), and its peer switch is assigned the secondary role. The VLT domain ID of
255 is automatically configured in SmartFabric mode. The VLTi link Status and VLT Peer Status must both be
up. SmartFabric automatically configures the VLTi as port channel 1000.

MX9116n-1# show vlt 255

Domain ID : 255
Unit ID : 1
Role : secondary

73 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Version : 1.0
Local System MAC address : 20:04:0f:00:b8:1e
VLT MAC address : 20:04:0f:00:b8:1e
IP address : fda5:74c8:b79e:1::1
Delay-Restore timer : 90 seconds
Peer-Routing : Disabled
Peer-Routing-Timeout timer : 0 seconds
VLTi Link Status
port-channel1000 : up

VLT Peer Unit ID System MAC Address Status IP Address Version

--------------------------------------------------------------------------------
2 20:04:0f:00:9d:1e up fda5:74c8:b79e:1::2 1.0

8.4.9 show vlt domain-id vlt-port-detail

The show vlt domain-id vlt-port-detail command shows the VLT port channel status for both
VLT peers. The VLT in this example is connected to the Cisco ACI vPC. It is automatically configured in port
channel 1, and it consists of two ports on each switch.

MX9116n-1# show vlt 255 vlt-port-detail

vlt-port-channel ID : 1
VLT Unit ID Port-Channel Status Configured ports Active ports
-------------------------------------------------------------------------------
* 1 port-channel1 up 2 2
2 port-channel1 up 2 2

8.4.10 show interface port channel summary

The show interface port-channel summary command shows the LAG number (VLT port
channel 1 in this example), the mode, status and ports used in the port channel.

MX9116n-1# show interface port-channel summary

LAG Mode Status Uptime Ports
1 L2-HYBRID up 00:29:20 Eth 1/1/43 (Up)
Eth 1/1/44 (Up)

74 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

9 SmartFabric troubleshooting
This section provides information on errors that might be encountered while working with a SmartFabric.
Troubleshooting and remediation actions are also included to assist in resolving errors.

9.1 Troubleshooting errors encountered for port group breakout

The creation of a SmartFabric involves executing specific steps in a recommended order. The SmartFabric
deployment consists of four broad steps all completed using the OME-M console:

1. Create the VLANs to be used in the fabric.

2. Select switches and create the fabric based on the physical topology desired.
3. Create uplinks from the fabric to the existing network and assign VLANs to those uplinks.
4. Create and deploy the appropriate server templates to the compute sleds.

For cases where breakout of port groups is required, the breakout must be configured after the SmartFabric
creation and before adding the uplinks.

CREATE VLANs

PORT BREAKOUT >

CREATE FABRIC
HARDWARE DEFAULT

PORT BREAKOUT >

ADD UPLINKS SELECT DESIRED
BREAKOUT TYPE

CREATE AND DEPLOY

SERVER TEMPLATES

EXECUTE WHEN BREAKOUT NOT REQUIRED

EXECUTE BASED ON BREAKOUT REQUIREMENT

Recommended order of steps for port breakout while creating SmartFabric

You may encounter the following errors if the recommended order of steps is not followed:

• Configuration of the breakout requires you to create the SmartFabric first. When attempting to
configure breakout before creating a SmartFabric, the following error displays:

75 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Error: I/O Module is not in fabric mode

• Configuration of the breakout requires you to select the HardwareDefault breakout type first. If the
breakout type is directly selected without first selecting HardwareDefault, the following error displays:

Error: interface fanout type is not hardware default

• Once the uplinks are added, they are most often associated with tagged or untagged VLANs. When
attempting to configure the breakout on the uplink port-groups after adding uplinks associated with
VLANs to the fabric, the following error displays:

Error: Breakout failed

76 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

9.2 Troubleshooting Spanning Tree Protocol (STP)
Spanning Tree Protocol (STP) prevent loops in the network. Loops can occur when multiple redundant parts
are available between the switches. To prevent the network from going down due to loops, various flavors of
STP are available. Initial introduction of STP evolved into various types. It is essential to ensure that network
loops are prevented by using appropriate type of STP on the upstream switches as well as the MX9116n
FSEswitches that are part of the MX SmartFabric.

9.2.1 Verify if STP is enabled on upstream switches

STP is required when connecting a SmartFabric to the upstream network. Turning off Spanning Tree in the
upstream switches will result in network loops and may cause downtime. Enable the appropriate STP type on
the upstream switches.

9.2.2 Verify if type of STP is the same on MX and upstream switches

Check the upstream switch if STP is enabled and verify that the type of STP matches the type of STP running
on the MX switches. By default, the MX switches run RPVST+ as shown below:

OS10# show spanning-tree brief

Spanning tree enabled protocol rapid-pvst
VLAN 1
Executing IEEE compatible Spanning Tree Protocol
---- OUTPUT TRUNCATED -----

• The following example shows the STP on the upstream switches, Cisco Nexus 3232C, is configured
to run MST:

Nexus-3232C-Leaf1(config)# do show spanning-tree summary

Switch is in mst mode (IEEE Standard)
Root bridge for: MST0000
Port Type Default is disable
---- OUTPUT TRUNCATED -----

• The recommended course of action is to change the STP type to RPVST+ on the upstream Cisco
Nexus switches.

Nexus-3232C-Leaf1(config)# spanning-tree mode rapid-pvst

Nexus-3232C-Leaf1(config)# do show spanning-tree summary
Switch is in rapid-pvst mode
--- OUTPUT TRUNCATED -----

• Another course of action in the above case can be to change the spanning tree type on the MX
switches operating in SmartFabric mode to match the STP type on the upstream switches. This can
be done using the SmartFabric OS10 CLI. The options available on the type of STP are as follows:

OS10(config)# spanning-tree mode ?

<rstp/mst/rapid-pvst> STP Protocol type

77 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

9.3 Verify VLT/vPC configuration on upstream switches
Configuring a single VLT domain with Dell EMC upstream switches or a single vPC domain with Cisco
upstream switches is required. Creating two VLT/vPC domains may cause a network loop. See Scenario 1
and Scenario 2 for the topology used in the deployment example.

The following example shows a mismatch of the VLT domain IDs on VLT peer switches. To resolve this issue,
ensure that a single VLT domain is used across the VLT peers.

Z9100-Leaf1# show vlt 1

Domain ID : 1
Unit ID : 1
Role : primary
Version : 1.0
Local System MAC address : 4c:76:25:e8:f2:c0

Z9100-Leaf2# show vlt 30

Domain ID : 30
Unit ID : 1
Role : primary
Version : 1.0

The following example shows a mismatch of the vPC domain IDs on vPC peer switches. To resolve this
issue, ensure that a single vPC domain is used across the vPC peers.

Nexus-3232C-Leaf1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 1
Peer status : peer link is down
vPC keep-alive status : peer is alive, but domain IDs do not match
---- OUTPUT TRUNCATED -----

3232C-Leaf2# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 255
Peer status : peer link is down
vPC keep-alive status : peer is alive, but domain IDs do not match
---- OUTPUT TRUNCATED -----

9.4 Discovery of FEM and compute sleds

The following can be verified if server or FEM discovery doesn’t happen:

• If there is no link indicated on the FSE port, toggle the auto-negotiation settings for that port.
• Ensure that the compute sled is properly seated in the compute slot in the MX chassis.
• Make sure that the compute sled is turned on.

78 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 • Ensure that the drivers and firmware for BIOS, iDRAC, NICs and/or CNAs on the compute sleds are up to
date.
• Verify the Topology LLDP settings. This can be verified by selecting iDRAC Settings > Connectivity on
the compute sled’s iDRAC GUI. Ensure that this setting is set to Enabled as shown in the figure below.

Ensure Topology LLDP is enabled

9.5 Troubleshooting uplink errors

There might be additional settings enabled or disabled after uplinks are added to the fabric.

9.5.1 Toggle auto negotiation

Enabling or disabling auto negotiation from the OME-M console can bring up the uplinks connecting to the
upstream switches. For example, when deploying the SmartFabric with the Cisco Nexus 3232C (see Scenario
2), disable auto negotiation on uplink ports on the MX switches to bring the link up.

The OME-M console is used to disable/enable auto negotiation ports on MX switches. The following steps
illustrate turning disabling auto negotiation on ports 41 and 42 of a MX9116n.

1. From switch management page, choose Hardware > Port Information.

2. Select the ports on which auto negotiation needs to be disabled. In this example, ports 1/1/41 and
port 1/1/42 are selected.
3. Click Toggle AutoNeg > Finish.

79 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Toggle AutoNeg dialog box

9.5.2 Set uplink ports to administratively up

The uplink ports on the switch might be administratively down. Enabling the uplink ports can be carried out
from the OME-M console. The uplink ports can be administratively down when a port group breakout
happens, especially for FC breakouts.

The OME-M console can be used to disable/enable the ports on MX switches. The following steps illustrate
turning setting the administrative state on ports 41 and 42 of an MX9116n.

1. From switch management page, choose Hardware > Port Information.

2. Select the ports. In this example, ports 1/1/41 and port 1/1/42 are selected.
3. Click Toggle Admin State > Finish.

Toggle Administrative port state

9.5.3 Verify MTU size

It is recommended to keep the same MTU size on ports connecting MX switches and the ports on the
upstream switches and server NICs. To set the MTU size from the OME-M console, see Section 7.2.

9.5.4 Verify auto negotiation settings on upstream switches

Verify the auto negotiation settings on the upstream switches. In case of where auto negotiation settings are
modified, the links might not come up. Change the auto negotiation on upstream switches to resolve the
issue.

For example, if the auto negotiation was disabled on the Cisco Nexus upstream switches, the setting can be
turned on. To enable the auto-negotiation on an ethernet interface on Cisco Nexus switches, follow the below
steps:

switch# configure terminal

switch(config)# interface ethernet interface-number

80 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 switch(config-if)# negotiate auto

The following example shows interface ethernet 1/2 that has auto negotiation enabled on the interface:

Nexus-3232C-Leaf1(config-if)# do show int eth 1/2

Ethernet1/2 is down (XCVR not inserted)
admin state is down, Dedicated Interface
Hardware: 40000/100000 Ethernet, address: 00fe.c8ca.f367 (bia 00fe.c8ca.f36c)
MTU 1500 bytes, BW 100000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
auto-duplex, auto-speed
Beacon is turned off
Auto-Negotiation is turned on, FEC mode is Auto
---- OUTPUT TRUNCATED -----

9.5.5 Verify LACP

The interface status of the upstream switches can provide valuable information for the link being down. The
following example shows interfaces 1 and 3 on upstream Cisco Nexus switches as members of port channel
1:

3232C-Leaf2# show interface status

--------------------------------------------------------------------------------
Port Name Status Vlan Duplex Speed Type
--------------------------------------------------------------------------------
mgmt0 -- connected routed full 1000 --
Eth1/1 To MX Chassis 1 suspended trunk full 100G QSFP-100G-SR4
Eth1/2 -- xcvrAbsen routed auto auto --
Eth1/3 To MX Chassis 2 suspended trunk full 100G QSFP-100G-SR4
---- OUTPUT TRUNCATED -----

Checking interface 1 reveals that the ports are not receiving the LACP PDUs as shown in the following
example:

3232C-Leaf2# show int eth 1/1

Ethernet1/1 is down (suspended(no LACP PDUs))
admin state is up, Dedicated Interface
Belongs to Po1
---- OUTPUT TRUNCATED -----

Note: In Dell EMC PowerSwitches, use show interface status command to view the interfaces and
associated status information. Use show interface ethernet interface number to view the interface
details.

In this example, the errors listed above occurred because an uplink was not created on the fabric.

81 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Fabric details

Fabric topology with no uplinks

The resolution is to add the uplinks and verify that the fabric turns healthy.

82 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Healthy fabric

9.6 Troubleshooting FC/FCoE

The following points can be verified while troubleshooting FC or FCoE errors:

• Ensure that the firmware and drivers are up to date on the CNAs.
• Check the storage guide to ensure that the CNAs are supported by the storage used in the
deployment. For qualified support matrix, see elab navigator and Dell EMC Storage Compatibility
Matrix for SC Series, PS Series and FS Series.
• Verify that port group breakout mode is appropriately configured.
• Ensure that the FC port-groups broken out on the unified ports in MX9116n switches are made
administratively up once the ports are changed from Ethernet to FC.
• MX9116n switches operating in SmartFabric mode support various commands to verify the
configuration. Use the following commands to verify FC configurations from MX9116n CLI:

OS10# show fc
alias Show FC alias
ns Show FC NS Switch parameters
statistics Show FC Switch parameters
switch Show FC Switch parameters
zone Show FC Zone
zoneset Show fc zoneset

Use the following commands to verify FCoE configurations from MX9116n CLI:

OS10# show fcoe

enode Show FCOE enode information
fcf Show FCOE fcf information
sessions Show FCOE session information
statistics Show FCOE statistics information
system Show FCOE system information
vlan Show FCOE vlan information

83 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Make sure that the FC ports are up, for example:

OS10# show interface status | grep 1/43

Fc 1/1/43:1 up 16G auto -
Fc 1/1/43:2 up 16G auto -
Fc 1/1/43:3 down 0 auto –
Fc 1/1/43:4 down 0 auto –

The show vfabric command output provides a variety of information including the default zone mode, the
active zone set, and interfaces that are members of the vfabric.

OS10# show vfabric

Fabric Name New vfabric
Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
FKA-Adv-Period Enabled,8
Config-State ACTIVE
Oper-State UP
==========================================
Switch Config Parameters
==========================================
Domain ID 1
==========================================
Switch Zoning Parameters
==========================================
Default Zone Mode: Allow
Active ZoneSet: None
==========================================
Members
fibrechannel1/1/44:1
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2

The show fcoe sessions command shows active FCoE sessions. The output includes MAC addresses,
Ethernet interfaces, the FCoE VLAN ID, FC IDs, and WWPNs of logged-in CNAs.

Note: Due to the width of the command output, each line of output is shown on two lines below.

OS10# show fcoe sessions

Enode MAC Enode Interface FCF MAC FCF interface VLAN
FCoE MAC FC-ID PORT WWPN PORT WWNN
--------------------------------------------------------------------------------
-------------------------------------------------------------------------

84 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:00

Note: For more information on FC and FCoE, see the Dell EMC SmartFabric OS10 User Guide.

9.7 SmartFabric Services – Troubleshooting commands

The following commands allow user to view various SmartFabric Services configuration information. These
commands can also be used as troubleshooting purpose on SmartFabric OS10.

These commands are available in OS10.5.0.1

9.7.1 show smartfabric personality

show smartfabric personality command is used on a node to view the personality of SmartFabric
Services configured. The possible values can be PowerEdge MX, Isilon, VxRail, L3 fabric.

9.7.2 show smartfabric cluster

show smartfabric cluster command is used to see if node is part of the cluster. This displays the
cluster information of the node such as node role, service, virtual ip address and node domain. It can also be
used to verify role of the node as either Backup or Master.

OS10# show smartfabric cluster

----------------------------------------------------------
CLUSTER DOMAIN ID : 50
VIP : fde1:53ba:e9a0:de14:0:5eff:fe00:150
ROLE : MASTER
SERVICE-TAG : CBJXLN2
----------------------------------------------------------

9.7.3 show smartfabric cluster member

show smartfabric cluster member command is used to see the member details of the cluster. This
displays the cluster member information such as service-tag, ip address, status, role, type of each node and
service tag of the chassis where the node belongs to.

OS10# show smartfabric cluster member

Service-tag IP Address Status Role
Type Chassis-Service-Tag Chassis-Slot
--------------------------------------------------------------------------------
------------------------------------------
CBJXLN2 fde1:53ba:e9a0:de14:2204:fff:fe00:cde7 ONLINE MASTER
MX5108n SKY002Z A1

85 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 BZTQPK2 fde1:53ba:e9a0:de14:2204:fff:fe00:19e5 ONLINE BACKUP
MX5108n SKY002Z B1
6L59XM2 fde1:53ba:e9a0:de14:2204:fff:fe00:3de5 ONLINE BACKUP
MX5108n SKY002Z B2
F13RPK2 fde1:53ba:e9a0:de14:2204:fff:fe00:a267 ONLINE BACKUP
MX5108n SKY003Z A2

9.7.4 show smartfabric details

show smartfabric details command is used to see the all configured fabric details. This displays
which nodes are part of the fabric, status of the fabric, and design type associated with the fabric.

OS10# show smartfabric details

----------------------------------------------------------
Name : Fabric 1
Description :
ID : 74b3d3a4-7804-4c15-b6d3-5e4e7c364f82
DesignType : 2xMX9116n_Fabric_Switching_Engines_in_different_chassis
Validation Status: VALID
VLTi Status : VALID
Placement Status : VALID
Nodes : CBJXLN2, F13RPK2
----------------------------------------------------------

9.7.5 show smartfabric uplinks

show smartfabric uplinks command is used to verify the uplinks configured across the nodes in the
fabric. This displays name, description, id, media type, native vlan, configured interfaces and network profile
associated with fabric.

OS10# show smartfabric uplinks

----------------------------------------------------------
Name : FCoE Path A
Description :
ID : 1b328dc2-b99c-466e-b87c-b84c9c342225
Media Type : FC
Native Vlan : 0
Untagged-network :
Networks : 6a161bae-788f-4d65-8b0c-69b404c477dc
Configured-Interfaces : CBJXLN2:fibrechannel1/1/44:1,
CBJXLN2:fibrechannel1/1/44:2
----------------------------------------------------------
----------------------------------------------------------
Name : Uplink1
Description :
ID : d493fee2-9680-41c7-989d-cf0347aab4fd
Media Type : ETHERNET
Native Vlan : 1

86 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Untagged-network :
Networks : e6189b88-7f19-4b05-98b5-0c05ff7ff8c8, 284dae93-b91f-
4593-9cff-c8521cd7ae90
Configured-Interfaces : CBJXLN2:ethernet1/1/42:1, F13RPK2:ethernet1/1/41:1,
F13RPK2:ethernet1/1/42:1, CBJXLN2:ethernet1/1/41:1
----------------------------------------------------------
----------------------------------------------------------
Name : FCoE Path B
Description :
ID : 0f7ad3a2-e59e-4a07-9a74-4e57558f0a4d
Media Type : FC
Native Vlan : 0
Untagged-network :
Networks : e2c35ec5-c177-46f1-9a69-75d8b202d739
Configured-Interfaces : F13RPK2:fibrechannel1/1/44:1,
F13RPK2:fibrechannel1/1/44:2
----------------------------------------------------------

9.7.6 show smartfabric networks

show smartfabric networks command is used to view the various network profile configured. The
command displays the vlans configured, QoS Priority and network type for each network profile.

OS10# show smartfabric networks

Name Type QosPriority Vlan
--------------------------------------------------------------------------------
FCoE A1 STORAGE_FCOE PLATINUM 998
VLAN1 GENERAL_PURPOSE BRONZE 1
FCoE A2 STORAGE_FCOE PLATINUM 999
VLAN10 GENERAL_PURPOSE SILVER 10
UPLINK VLAN GENERAL_PURPOSE SILVER 2491

9.7.7 show smartfabric validation-error

show smartfabric validation-error displays all the information on the validation errors such as
category, sub-category, recommended action, severity, timestamp and recommended link to each error.

9.7.8 show smartfabric nodes

show smartfabric nodes command is used to view the details of the nodes that are part of the cluster.
This command helps the user to view the status of a node, chassis details the node belongs to.

OS10# show smartfabric nodes

Service-Tag Type Status Mode
Chassis-Service Chassis-Slot
Tag
--------------------------------------------------------------------------------
---------------------------------------------
F13RPK2 MX9116n ONLINE FABRIC SKY003Z A2
110DXC2 MX7116n NOT-APPLICABLE SKY002Z A2

87 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 CBJXLN2 MX9116n ONLINE FABRIC SKY002Z A1
6L59XM2 MX5108n ONLINE FULL-SWITCH SKY002Z B2
D10DXC2 MX7116n NOT-APPLICABLE SKY003Z A1
BZTQPK2 MX5108n ONLINE FULL-SWITCH SKY002Z B1

88 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10 Uplink configuration scenarios

10.1 Scenario 1 - SmartFabric deployment with Dell EMC PowerSwitch

Z9100-ON upstream switches
Figure 80 shows the production topology using a pair of Dell EMC PowerSwitch Z9100-ONs as upstream
switches. This section walks through configuring the Z9100-ON as well as validating the Z9100-ON
configuration.

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33SFP+ Stack ID

Z9100-ON Leaf 1 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34SFP+

EST

VLTi
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33SFP+ Stack ID

Z9100-ON Leaf 2 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34SFP+

EST

MX7116n FEM
MX9116n FSE

VLTi

MX9116n FSE
MX7116n FEM

VLTi (100 GbE)

Fabric uplink (100 GbE)
FSE to FEM (200 GbE)
FSE VLTi (200 GbE)

SmartFabric with Dell EMC PowerSwitch Z9100-ON leaf switches

Note: See Dell EMC PowerEdge MX Network Architecture Guide for more information on QSFP28-DD cables.

89 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.1.1 Dell EMC PowerSwitch Z9100-ON switch configuration
The following section outlines the configuration commands issued to the Dell EMC PowerSwitch Z9100-ON
switches. The switches start at their factory default settings per Appendix C.3.

Note: The MX IOMs run Rapid Per-VLAN Spanning Tree Plus (RPVST+) by default. RPVST+ runs RSTP on each
VLAN while RSTP runs a single instance of spanning tree across the default VLAN. The Dell EMC PowerSwitch
Z9100-ON used in this example runs SmartFabric OS10 and has RPVST+ enabled by default. See Spanning
Tree Protocol recommendations for more information.

There are 4 steps to configure the Z9100-ON upstream switches:

1. Set the switch hostname and management IP address.

2. Configure the VLT between the switches.
3. Configure the VLANs.
4. Configure the port channels to connect to the MX switches.

Use the following commands to set the hostname, and to configure the OOB management interface and
default gateway.

Z9100-ON Leaf 1 Z9100-ON Leaf 2

configure terminal configure terminal

hostname Z9100-Leaf1 hostname Z9100-Leaf2

interface mgmt 1/1/1 interface mgmt 1/1/1

no ip address dhcp no ip address dhcp
no shutdown no shutdown
ip address 100.67.162.35/24 ip address 100.67.162.34/24

management route 0.0.0.0/0 100.67.162.254 management route 0.0.0.0/0 100.67.162.254

Note: Use spanning-tree {vlan vlan-id priority priority-value} command to set the bridge
priority for the upstream switches. The bridge priority ranges from 0 to 61440 in increments of 4096. For example,
to make Z9100-ON Leaf 1 as the root bridge for VLAN 10, enter the command spanning-tree vlan 10
priority 4096.

Configure the VLT between switches using the following commands. VLT configuration involves setting a
discovery interface range and discovering the VLT peer in the VLTi.

Z9100-ON Leaf 1 Z9100-ON Leaf 2

interface range ethernet1/1/29-1/1/31 interface range ethernet1/1/29-1/1/31
description VLTi description VLTi
no shutdown no shutdown
no switchport no switchport

vlt-domain 1 vlt-domain 1
backup destination 100.67.162.34 backup destination 100.67.169.35
discovery-interface ethernet1/1/29-1/1/31 discovery-interface ethernet1/1/29-1/1/31

90 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Configure the required VLANs on each switch. In this deployment example, the VLAN used is VLAN 10.

Z9100-ON Leaf 1 Z9100-ON Leaf 2

interface vlan10 interface vlan10
description “Company A General Purpose” description “Company A General Purpose”
no shutdown no shutdown

Configure the port channels that connect to the downstream switches. The LACP protocol is used to create
the dynamic LAG. Trunk ports allow tagged VLANs to traverse the trunk link. In this example, the trunk is
configured allow VLAN 10.

Z9100-ON Leaf 1 Z9100-ON Leaf 2

interface port-channel1 interface port-channel1
description "To MX Chassis" description "To MX Chassis"
no shutdown no shutdown
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan10 switchport trunk allowed vlan10
vlt-port-channel 1 vlt-port-channel 1

interface ethernet1/1/1 interface ethernet1/1/1

description "To MX Chassis-1" description "To MX Chassis-1"
no shutdown no shutdown
no switchport no switchport
channel-group 1 mode active channel-group 1 mode active

interface ethernet1/1/3 interface ethernet1/1/3

description "To MX Chassis-2" description "To MX Chassis-2"
no shutdown no shutdown
no switchport no switchport
channel-group 1 mode active channel-group 1 mode active

end end
write memory write memory

91 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.1.2 Dell EMC PowerSwitch Z9100-ON validation
This section contains validation commands for the Dell EMC PowerSwitch Z9100-ON leaf switches.

10.1.2.1 show vlt

The show vlt command validates the VLT configuration status when the VLTi Link Status is up. The role of
one switch in the VLT pair is primary, and its peer switch (not shown) is assigned the secondary role.

Z9100-Leaf1# show vlt 1

Domain ID : 1
Unit ID : 1
Role : primary
Version : 1.0
Local System MAC address : 4c:76:25:e8:f2:c0
VLT MAC address : 4c:76:25:e8:f2:c0
IP address : fda5:74c8:b79e:1::1
Delay-Restore timer : 90 seconds
Peer-Routing : Disabled
Peer-Routing-Timeout timer : 0 seconds
VLTi Link Status
port-channel1000 : up

VLT Peer Unit ID System MAC Address Status IP Address

Version
--------------------------------------------------------------------------------
2 4c:76:25:e8:e8:40 up fda5:74c8:b79e:1::2 1.0

10.1.2.2 show lldp neighbors

The show lldp neighbors command provides information about connected devices. In this case,
ethernet1/1/1 and ethernet1/1/3 connect to the two MX9116n FSEs, C160A2 and C140A1. The
remaining links, ethernet1/1/29 and ethernet 1/1/31, represent the VLTi connection.

Z9100-Leaf1# show lldp neighbors

Loc PortID Rem Host Name Rem Port Id Rem Chassis Id
----------------------------------------------------------------
ethernet1/1/1 C160A2 ethernet1/1/41 20:04:0f:00:a1:9e
ethernet1/1/3 C140A1 ethernet1/1/41 20:04:0f:00:cd:1e
ethernet1/1/29 Z9100-Leaf2 ethernet1/1/29 4c:76:25:e8:e8:40
ethernet1/1/31 Z9100-Leaf2 ethernet1/1/31 4c:76:25:e8:e8:40

10.1.2.3 show spanning-tree brief

The show spanning-tree brief command validates that STP is enabled on the leaf switches. All
interfaces are forwarding (FWD), as shown in the Sts column.

Z9100-Leaf1# show spanning-tree brief

Spanning tree enabled protocol rapid-pvst
VLAN 1
Executing IEEE compatible Spanning Tree Protocol
Root ID Priority 32768, Address 2004.0f00.a19e
Root Bridge hello time 2, max age 20, forward delay 15
Bridge ID Priority 32769, Address 4c76.25e8.f2c0

92 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Configured hello time 2, max age 20, forward delay 15
Flush Interval 200 centi-sec, Flush Invocations 432
Flush Indication threshold 0 (MAC flush optimization is disabled)
Interface Designated
Name PortID Prio Cost Sts Cost Bridge ID PortID
--------------------------------------------------------------------------------
port-channel1 128.2517 128 50 FWD 0 32768 2004.0f00

Interface
Name Role PortID Prio Cost Sts Cost Link-type Edge
--------------------------------------------------------------------------------
port-channel1 Root 128.2517 128 50 FWD 0 AUTO No

VLAN 10
Executing IEEE compatible Spanning Tree Protocol
Root ID Priority 32778, Address 4c76.25e8.e840
Root Bridge hello time 2, max age 20, forward delay 15
Bridge ID Priority 32778, Address 4c76.25e8.f2c0
Configured hello time 2, max age 20, forward delay 15
Flush Interval 200 centi-sec, Flush Invocations 5
Flush Indication threshold 0 (MAC flush optimization is disabled)
Interface Designated
Interface Designated
Name PortID Prio Cost Sts Cost Bridge ID PortID
--------------------------------------------------------------------------------
port-channel1 128.2517 128 50 FWD 1 32768 2004.0f00
Interface
Name Role PortID Prio Cost Sts Cost Link-type Edge
--------------------------------------------------------------------------------

port-channel1 Root 128.2517 128 50 FWD 1 AUTO No

93 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.2 Scenario 2 - SmartFabric connected to Cisco Nexus 3232C switches
Figure 81 shows the production topology using a pair of Cisco Nexus 3232C as leaf switches. This section
configures the Cisco Nexus 3232Cs and creating a SmartFabric with the corresponding uplinks.

S1 / 33 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
1
2

Nexus 3232C Leaf 1

3
BCN
4
STS

ENV

LS
S2 / 34

vPC Peer-Link
S1 / 33 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
1
2

Nexus 3232C Leaf 2

3
BCN
4
STS

ENV

LS
S2 / 34

MX7116n FEM
MX9116n FSE

VLTi

MX9116n FSE
MX7116n FEM

Fabric uplink (100 GbE)

FSE VLTi (200 GbE)
FSE to FEM (200 GbE)
vPC (100 GbE)

SmartFabric with Cisco Nexus 3232C leaf switches

Note: See Dell EMC PowerEdge MX Network Architecture Guide for more information on QSFP28-DD cables.

94 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.2.1 Cisco Nexus 3232C switch configuration
The following section outlines the configuration commands issued to the Cisco Nexus 3232C leaf switches.

Note: While this configuration example is specific to the Cisco Nexus 3232C switch, the same concepts apply to
other Cisco Nexus and IOS switches.

The switches start at their factory default settings, as described in Appendix C.4.

Note: The MX IOMs run Rapid per-VLAN Spanning Tree Plus (RPVST+) by default. Ensure the Cisco and Dell
switches are configured to use compatible STP protocols. The mode of STP on the Cisco switch can be set using
the command spanning-tree mode, which is shown below. See Spanning Tree Protocol recommendations for
more information. In this deployment example, default VLAN is VLAN 1 and the created VLAN is VLAN 10. See
Cisco Nexus 3000 Series NX-OS configuration guide for more details.

There are 4 steps to configure the 3232C upstream switches:

1. Set switch hostname, management IP address, enable features and spanning tree
2. Configure vPC between the switches
3. Configure the VLANs
4. Configure the downstream port channels to connect to the MX switches

Enter the following commands to set the hostname, enable required features, and enable RPVST spanning
tree mode. Configure the management interface and default gateway.

Cisco Nexus 3232C Leaf 1 Cisco Nexus 3232C Leaf 2

configure terminal configure terminal

hostname 3232C-Leaf1 hostname 3232C-Leaf2

feature vpc feature vpc

feature lldp feature lldp
feature lacp feature lacp

spanning-tree mode rapid-pvst spanning-tree mode rapid-pvst

interface mgmt0 interface mgmt0

vrf member management vrf member management
ip address 100.67.162.201/24 ip address 100.67.162.200/24

vrf context management vrf context management

ip route 0.0.0.0/0 100.67.162.254 ip route 0.0.0.0/0 100.67.162.254

Enter the following commands to create a virtual port channel (vPC) domain and assign the keepalive
destination to the peer switch management IP. Then create a port channel for the vPC peer link and assign
the appropriate switchport interfaces.

95 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Cisco Nexus 3232C Leaf 1 Cisco Nexus 3232C Leaf 2
vpc domain 255 vpc domain 255
peer-keepalive destination 100.67.162.200 peer-keepalive destination 100.67.162.201

interface port-channel255 interface port-channel255

switchport switchport
switchport mode trunk switchport mode trunk
vpc peer-link vpc peer-link

interface Ethernet1/29 interface Ethernet1/29

description vPC Interconnect description vPC Interconnect
switchport switchport
switchport mode trunk switchport mode trunk
channel-group 255 mode active channel-group 255 mode active
no shutdown no shutdown

interface Ethernet1/31 interface Ethernet1/31

description vPC Interconnect description vPC Interconnect
switchport switchport
switchport mode trunk switchport mode trunk
channel-group 255 mode active channel-group 255 mode active
no shutdown no shutdown

Enter the following commands to configure the port channels to connect to the downstream MX9116n FSEs.
Then, exit configuration mode and save the configuration.

Cisco Nexus 3232C Leaf 1 Cisco Nexus 3232C Leaf 2

interface port-channel1 interface port-channel1
description To MX Chassis description To MX Chassis
switchport switchport
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 1,10 switchport trunk allowed vlan 1,10
vpc 255 vpc 255

interface Ethernet1/1 interface Ethernet1/1

description To MX Chassis 1 description To MX Chassis 1
switchport switchport
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 1,10 switchport trunk allowed vlan 1,10
channel-group 1 mode active channel-group 1 mode active
no shutdown no shutdown

interface Ethernet1/3 interface Ethernet1/3

description To MX Chassis 2 description To MX Chassis 2
switchport switchport
switchport mode trunk switchport mode trunk
switchport trunk allowed vlan 1,10 switchport trunk allowed vlan 1,10
channel-group 1 mode active channel-group 1 mode active
no shutdown no shutdown

end end
copy running-configuration startup-configuration copy running-configuration startup-configuration

96 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Note: If the connections to the MX switches do not come up, see Section 9 for troubleshooting steps.

Trunk ports on switches allow tagged traffic to traverse the links. All flooded traffic for the a VLAN will be sent
across trunk ports to all the switches even if those switches do not have associated VLAN. This takes up the
network bandwidth with unnecessary traffic. VLAN or VTP Pruning is the feature that can be used to eliminate
this unnecessary traffic by pruning the VLANs.

Pruning restricts the flooded traffic to only those trunk ports with associated VLANs to optimize the usage of
network bandwidth. If the existing environment is configured for Cisco VTP or VLAN pruning, ensure that the
Cisco upstream switches are configured appropriately. See See Cisco Nexus 3000 Series NX-OS
configuration guides for additonal information.

Note: Do not use switchport trunk allow vlan all on the Cisco interfaces. VLANs must be explicitly assigned to the
interface

10.2.2 Configuration validation

This section covers the validation of the Cisco Nexus 3232C leaf switches. For information about the Dell
EMC Networking MX switch validation commands, see Section 10.1.2.

10.2.2.1 show vpc

The show vpc command validates the vPC configuration status. The peer adjacency should be OK, with the
peer should show as alive. The end of the command shows which VLANs are active across the vPC.

NX3232C-Leaf1# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 255

Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 inconsistency reason : Consistency Check Not Performed
vPC role : secondary, operational primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Disabled
Delay-restore status : Timer is off.(timeout = 30s)
Delay-restore SVI status : Timer is off.(timeout = 10s)

vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po255 up 1,10

97 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
255 Po1 up success success 1,10

10.2.2.2 show vpc consistency-parameters

The show vpc consistency-parameters command displays the configured values on all interfaces
in the vPC. The displayed configurations are only those configurations that limit the vPC peer link and vPC
from coming up.

NX3232C-Leaf1# show vpc consistency-parameters vpc 255

Legend:
Type 1 : vPC will be suspended in case of mismatch

Name Type Local Value Peer Value

------------- ---- ---------------------- -----------------------
STP Port Type 1 Normal Port Normal Port
STP Port Guard 1 Default Default
STP MST Simulate PVST 1 Default Default
lag-id 1 [(1000, [(1000,
20-4-f-0-cd-1e, 1, 0, 20-4-f-0-cd-1e, 1, 0,
0), (7f9b, 0), (7f9b,
0-23-4-ee-be-ff, 80ff, 0-23-4-ee-be-ff, 80ff,
0, 0)] 0, 0)]
mode 1 active active
delayed-lacp 1 disabled disabled
Speed 1 100 Gb/s 100 Gb/s
Duplex 1 full full
Port Mode 1 trunk trunk
Native Vlan 1 1 1
MTU 1 1500 1500
Dot1q Tunnel 1 no no
Switchport Isolated 1 0 0
vPC card type 1 N9K TOR N9K TOR
Allowed VLANs - 1,10 1,10
Local suspended VLANs - - -

98 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.2.2.3 show lldp neighbors
The show lldp neighbors command provides information about lldp neighbors. In this example,
Eth1/1 and Eth1/3 are connected to the two MX9116n FSEs, C160A2 and C140A1. The remaining
links, Eth1/29 and Eth1/31, represent the vPC connection.

NX3232C-Leaf1(config)# show lldp neighbors

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
S3048-ON mgmt0 120 PBR ethernet1/1/45
C160A2 Eth1/1 120 PBR ethernet1/1/41
C140A1 Eth1/3 120 PBR ethernet1/1/41
NX3232C-Leaf2 Eth1/29 120 BR Ethernet1/29
NX3232C-Leaf2 Eth1/31 120 BR Ethernet1/31
Total entries displayed: 5

10.2.2.4 show spanning-tree summary

The show spanning-tree summary command validates that STP is enabled on the leaf switches. All
interfaces are shown as forwarding.

NX3232C-Leaf1# show spanning-tree summary

Switch is in rapid-pvst mode
Root bridge for: VLAN0010
Port Type Default is disable
Edge Port [PortFast] BPDU Guard Default is disabled
Edge Port [PortFast] BPDU Filter Default is disabled
Bridge Assurance is enabled
Loopguard Default is disabled
Pathcost method used is short
STP-Lite is disabled

Name Blocking Listening Learning Forwarding STP Active

---------------------- -------- --------- -------- ---------- ----------
VLAN0001 0 0 0 2 2
VLAN0010 0 0 0 2 2
---------------------- -------- --------- -------- ---------- ----------
2 vlans 0 0 0 4 4

99 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.3 Scenario 3: Connect MX9116n FSE to Fibre Channel storage -
NPIV Proxy Gateway mode
This chapter discusses a method for connecting the MX9116n FSE to a FC storage array connected to
existing FC switches using the NPIV Proxy Gateway (NPG) mode for the connection. NPG mode allows for
larger SAN deployments that aggregate I/O traffic at the NPG switch.

FC Switch FC Switch
Spine 1 Spine 2

FC SAN A
FC SAN B

Controller A Controller B

MX9116n VLT MX9116n Unity 500F

(Leaf 1) (Leaf 2) UnityUnity
500F 500F
MX7000 MX7000 Note: Brocade 6510 switches were used in
chassis 1 chassis 2 validating this brownfield example. Other FC
switches may be used as well.

FC (NPG) network to Dell EMC Unity

This scenario shows attachment to a brownfield FC switch infrastructure. Configuration of the existing FC
switches is beyond the scope of this document.

Note: The MX5108n Ethernet Switch does not support this feature.

This example assumes that an existing SmartFabric has been created and is fully operational. For instructions
on creating a SmartFabric, see Section 4.3.

To configure NPG mode on an existing SmartFabric, the following steps are completed using the OME-M
console:

1. Connect the MX9116n FSE to the FC SAN. Note that the cables do NOT “criss-cross” between
the switches
2. Define FCoE VLANs to use in the fabric. For instructions, see Section 4.2.1 for defining VLANs.
3. Create Identity Pools if desired. see Section 5.3 for more information on how to create identity
pools.
4. Configure the physical switch ports for FC operation. See Section 4.6 for instructions.
5. Create the FC Gateway uplinks. For instruction, see Section 4.7 on creating Uplinks.
6. Create and deploy the appropriate server templates to the compute sleds. See Section 5.2 to 5.6
for more information.

Once the server operating system loads the FCoE driver, the WWN will appear on the fabric and on the FC
SAN. At that point, your system is now ready to connect to Fibre Channel storage. See Appendix B for setting
up storage logical unit numbers (LUNs).

100 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.3.1 Configuration validation
10.3.1.1 show fc ns switch
The show fc ns switch command shows all device ports logged into the fabric. In this deployment, four
ports are logged in to each switch: two storage ports and two CNA ports.

C140A1# show fc ns switch

Total number of devices = 3

Switch Name 10:00:20:04:0f:00:cd:1e
Domain Id 1
Switch Port fibrechannel1/1/44:1
FC-Id 01:00:00
Port Name 50:06:01:61:47:e4:1b:19
Node Name 50:06:01:60:c7:e0:1b:19
Class of Service 8
Symbolic Port Name UNITY::::SPA13::FC::::::
Symbolic Node Name UNITY::::SPA::FC::::::
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

Switch Name 10:00:20:04:0f:00:cd:1e

Domain Id 1
Switch Port ethernet1/71/1
FC-Id 01:01:00
Port Name 20:01:06:c3:f9:a4:cd:03
Node Name 20:00:06:c3:f9:a4:cd:03
Class of Service 8
Symbolic Port Name
Symbolic Node Name
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

Switch Name 10:00:20:04:0f:00:cd:1e

Domain Id 1
Switch Port ethernet1/1/1
FC-Id 01:02:00
Port Name 20:01:f4:e9:d4:73:d0:0c
Node Name 20:00:f4:e9:d4:73:d0:0c
Class of Service 8
Symbolic Port Name QLogic qedf v8.24.8.0
Symbolic Node Name QLogic qedf v8.24.8.0
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

101 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.3.1.2 show fcoe sessions
The show fcoe sessions command shows active FCoE sessions. The output includes MAC addresses,
Ethernet interfaces, the FCoE VLAN ID, FC IDs, and WWPNs of logged-in CNAs.

Note: Due to the width of the command output, each line of output is shown on two lines below.

C140A1# show fcoe sessions

Enode MAC Enode Interface FCF MAC FCF interface VLAN
FCoE MAC FC-ID PORT WWPN PORT WWNN
--------------------------------------------------------------------------------
-------------------------------------------------------------------------
06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:0c

10.3.1.3 show vfabric

The show vfabric command output provides a variety of information including the default zone mode, the
active zone set, and interfaces that are members of the vfabric.

C140A1# show vfabric

Fabric Name New vfabric
Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
FKA-Adv-Period Enabled,8
Config-State ACTIVE
Oper-State UP
==========================================
Switch Config Parameters
==========================================
Domain ID 1
==========================================
Switch Zoning Parameters
==========================================
Default Zone Mode: Allow
Active ZoneSet: None
==========================================
Members
fibrechannel1/1/44:1
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2

10.3.1.4 show fc switch

The show fc switch command verifies the switch mode (e.g. F_Port) for FC traffic.

102 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 C140A1# show fc switch
Switch Mode : FPORT
Switch WWN : 10:00:e4:f0:04:6b:04:42

103 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.4 Scenario 4: Connect MX9116n FSE to Fibre Channel storage - FC
Direct Attach
This chapter discusses a method for connecting a FC storage array directly to the MX9116n FSE.

As you can see, the steps to configure NPG mode or FC Direct Attach mode on the MX9116n FSE is different
only by selecting which type of uplink is desired.

Spine 1 Spine 2 Unity

Unity 500F500F
Controller A Controller B

FC SAN A
FC SAN B

MX9116n VLT MX9116n

(Leaf 1) (Leaf 2)

MX7000 MX7000
chassis 1 chassis 2
Fibre Channel (F_Port) direct connect network to Dell EMC Unity

This example shows directly attaching a Dell EMC Unity 500F storage array to the MX9116n FSE using
universal ports 44:1 and 44:2.

Note: The MX5108n Ethernet Switch does not support this feature

This example assumes that an existing SmartFabric has been created and is fully operational. For instructions
on creating a SmartFabric, see Section 4.3.

To configure NPG mode on an existing SmartFabric, the following steps are completed using the OME-M
console:

1. Connect the storage array to the MX9116n FSE. Note that each storage controller is
connected to each MX9116n FSE. Define FCoE VLANs to use in the fabric. For instructions,
see Section 4.2.1 for defining VLANs.
2. Create Identity Pools if desired. See Section 5.3 for more information on how to create
identity pools.
3. Configure the physical switch ports for FC operation. See Section 4.6 for instructions.
4. Create the FC Direct Attached uplinks. See Section 4.7 on creating Uplinks.
5. Create and deploy the appropriate server templates to the compute sleds. See Section 5.2 to
5.6 for more information.
6. Configure zones and zonesets. See Section 6.5 for instructions.

Once the server operating system loads the FCoE, the WWN will appear on the fabric and on the FC SAN. At
that point, your system is now ready to connect to Fibre Channel storage. See Appendix B for setting up
storage logical unit numbers (LUNs).

104 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.4.1 Configuration validation
10.4.1.1 show fc ns switch
The show fc ns switch command shows all device ports logged into the fabric. In this deployment, four
ports are logged in to each switch: two storage ports and two CNA ports.

C140A1# show fc ns switch

Total number of devices = 3

Switch Name 10:00:20:04:0f:00:cd:1e
Domain Id 1
Switch Port fibrechannel1/1/44:1
FC-Id 01:00:00
Port Name 50:06:01:61:47:e4:1b:19
Node Name 50:06:01:60:c7:e0:1b:19
Class of Service 8
Symbolic Port Name UNITY::::SPA13::FC::::::
Symbolic Node Name UNITY::::SPA::FC::::::
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

Switch Name 10:00:20:04:0f:00:cd:1e

Domain Id 1
Switch Port ethernet1/71/1
FC-Id 01:01:00
Port Name 20:01:06:c3:f9:a4:cd:03
Node Name 20:00:06:c3:f9:a4:cd:03
Class of Service 8
Symbolic Port Name
Symbolic Node Name
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

Switch Name 10:00:20:04:0f:00:cd:1e

Domain Id 1
Switch Port ethernet1/1/1
FC-Id 01:02:00
Port Name 20:01:f4:e9:d4:73:d0:0c
Node Name 20:00:f4:e9:d4:73:d0:0c
Class of Service 8
Symbolic Port Name QLogic qedf v8.24.8.0
Symbolic Node Name QLogic qedf v8.24.8.0
Port Type N_PORT
Registered with NameServer Yes
Registered for SCN Yes

105 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.4.1.2 show fcoe sessions
The show fcoe sessions command shows active FCoE sessions. The output includes MAC addresses,
Ethernet interfaces, the FCoE VLAN ID, FC IDs, and WWPNs of logged-in CNAs.

Note: Due to the width of the command output, each line of output is shown on two lines below.

C140A1# show fcoe sessions

Enode MAC Enode Interface FCF MAC FCF interface VLAN
FCoE MAC FC-ID PORT WWPN PORT WWNN
--------------------------------------------------------------------------------
-------------------------------------------------------------------------
06:c3:f9:a4:cd:03 Eth 1/71/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:01:00 01:01:00 20:01:06:c3:f9:a4:cd:03 20:00:06:c3:f9:a4:cd:03
f4:e9:d4:73:d0:0c Eth 1/1/1 20:04:0f:00:ce:1d ~ 30
0e:fc:00:01:02:00 01:02:00 20:01:f4:e9:d4:73:d0:0c 20:00:f4:e9:d4:73:d0:0c

10.4.1.3 show vfabric

The show vfabric command output provides a variety of information including the default zone mode, the
active zone set, and interfaces that are members of the vfabric.

C140A1# show vfabric

Fabric Name New vfabric
Fabric Type FPORT
Fabric Id 1
Vlan Id 30
FC-MAP 0xEFC00
Vlan priority 3
FCF Priority 128
FKA-Adv-Period Enabled,8
Config-State ACTIVE
Oper-State UP
==========================================
Switch Config Parameters
==========================================
Domain ID 1
==========================================
Switch Zoning Parameters
==========================================
Default Zone Mode: Allow
Active ZoneSet: None
==========================================
Members
fibrechannel1/1/44:1
ethernet1/1/1
ethernet1/71/1
ethernet1/71/2

10.4.1.4 show fc switch

The show fc switch command verifies the switch mode (e.g. F_Port) for FC traffic.

106 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 C140A1# show fc switch
Switch Mode : FPORT
Switch WWN : 10:00:e4:f0:04:6b:04:42

10.5 Scenario 5: Connect MX5108n to Fibre Channel storage - FSB

This chapter provides instructions for connecting either the MX5108n or MX9116n to a Fibre Channel SAN via
native FCoE uplinks. This connection type would be used in an environment where an existing switch such as
the Dell EMC PowerSwitch S4148U has the capability to accept native FCoE and connect to native FC.

Dell EMC SmartFabric OS10 uses a FIP Snooping Bridge (FSB) to detect and manage FCoE traffic and
discovers the following information:

• End nodes (E_Nodes)

• Fibre Channel forwarder (FCF)
• Connections between E_Nodes and FCFs
• Sessions between E_Nodes and FCFs

Using the discovered information, the switch installs ACL entries that provide security and point-to-point link
emulation to ensure that FCoE traffic is handled appropriately.

Note: The examples in this chapter use the Dell EMC Networking MX5108n. The same instructions may also be
applied and used with the MX9116n.

The FSB switch can connect to an upstream switch operating in NPG mode:

S4148U-ON S4148U-ON
(NPG mode) (NPG mode)
ToR switch 1 VLT ToR switch 2 FCoE SAN A
FC SAN A

FCoE SAN B
FC SAN B

FC Switch FC Switch

MX5108n MX5108n
FSB mode VLT
FSB mode
(Leaf 2) Controller A Controller B
(Leaf 1)

MX7000 chassis Unity 500F

UnityUnity
500F 500F

FCoE (FSB) Network to Dell EMC Unity through NPG mode switch

Or operating in F_Port mode:

107 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 S4148U-ON S4148U-ON
(F_port mode) (F_port mode)
ToR switch 1 VLT ToR switch 2 FCoE SAN A
FC SAN A

FCoE SAN B
FC SAN B

Controller A Controller B
MX5108n MX5108n
FSB mode VLT
FSB mode Unity500F
500F
(Leaf 1) (Leaf 2) UnityUnity
500F

MX7000 chassis

FCoE (FSB) Network to Dell EMC Unity through F_port mode switch

Note: See the Dell EMC SmartFabric OS10 Documentation for configuring NPG mode globally on the S4148U-
ON switches.

10.5.1 SmartFabric configuration steps

This example assumes that an existing SmartFabric has been created and is fully operational. For instructions
on creating a SmartFabric, see Section 4.3.

1. To configure FCoE mode on an existing SmartFabric, the following steps are completed using the
OME-M console:Connect the MX switch to the S4148U. Note that the cables do NOT “criss-cross”
between the switches
2. Define FCoE VLANs to use in the fabric. For instructions, see Section 4.2.1 for defining VLANs.
3. Create Identity Pools if desired. See Section 5.3 for more information.
4. Create the FCoE uplinks. See Section 4.7 on creating Uplinks.
5. Create and deploy the appropriate server templates to the compute sleds. See Section 5.2 to 5.6 for
more information.
6. Configure the S4148U switch. See Dell EMC Networking Fibre Channel Deployment with S4148U-
ON in F_port Mode for more information

Once the server operating system loads the FCoE driver, the WWN displays on the fabric and on the FC
SAN. At that point, your system is now ready to connect to Fibre Channel storage. See Appendix B for setting
up storage logical unit numbers (LUNs).

To validate the configuration, use the same commands as Section 10.3.1.

108 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

10.6 Scenario 6: Configure Boot from SAN
The host OS of MX Server can boot from a remote FC storage array utilizing the IOMs. Booting to an OS
through FC direct attach (F_port), FC (NPG), and FCoE (FSB) scenarios are supported.

Boot from SAN

FC storage (3 methods)

Direct Attached (F_port) FC (NPG) FCoE (FSB)

MX9116n MX9116n MX5108n

MX7000 MX7000 MX7000
Server
Server
Server
Server
Server Server

Boot from SAN

Figure 87 shows the example topology used in this chapter to demonstrate Boot from SAN. The required
steps are provided to configure NIC partitioning, system BIOS, an FCoE LUN, and an OS install media device
required for Boot from SAN.

S4148u S4148u
(NPG mode) (NPG mode)
ToR switch 1 VLT ToR switch 2 FCoE SAN A
FC SAN A

FCoE SAN B
FC SAN B

FC Switch FC Switch
MX5108n MX5108n
FSB mode VLT
FSB mode
(Leaf 2) Controller A Controller B
(Leaf 1)

MX7000 chassis Unity 500F

UnityUnity
500F500F
FCoE Boot from SAN

Note: See the OS10 User Guide document for configuring NPG mode globally on the S4148U-ON switches.

10.6.1 Configure NIC Boot Device

In this section, each QLogic CNA port is partitioned into one Ethernet and one FCoE partition.

Note: This is only done on CNA ports that carry converged traffic. In this example, these are the two 25GbE
QLogic CNA ports on each server that attach to the switches internally through an orthogonal connection.

1. Connect to the server's iDRAC in a web browser and launch the virtual console.
2. In the virtual console, select BIOS Setup from the Next Boot menu.
3. Reboot the server.

109 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 4. On the System Setup Main Menu, select Device Settings.
5. Select the first CNA port.
6. Select Device Level Configuration.
7. Set Virtualization Mode to NPAR (if not already set), and click Back.

Virtualization Mode to NPAR

8. Choose NIC Partitioning Configuration.

9. Select Partition 1 Configuration.
10. Set NIC + RDMA Mode to Disabled.

Set the value of NIC + RDMA mode

11. Click Back to return.

12. Select Partition 2 Configuration.
13. Set FCoE Mode to Enabled as shown.

FCoE mode to Enabled

110 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 14. Click Back and select Back to go to Main Configuration Page.
15. Select NIC Configuration, then set the Boot Protocol to UEFI FCoE, and then click Back.

Set the value of Boot Protocol to UEFI FCoE

16. If present, select Partition 3 Configuration in NIC Partitioning Configuration. Set all modes to
Disabled and then click Back.
17. If present, select Partition 4 Configuration in NIC Partitioning Configuration. Set all modes to
Disabled and then click Back.
18. Select FCoE Configuration.
19. Set the Virtual LAN ID (30 is used in this example).
20. Set Connect 1 to Enabled.
21. Set the World Wide Port Name Target 1 to the connected port on Unity.

FCoE configuration

22. Click Back and then click Finish.

23. When prompted, answer Yes to save changes and click OK in the Success window
24. Select the second CNA port and repeat the steps in this section for port 2.
25. Click Finish to exit to the System Setup Main Menu.

10.6.2 Configure BIOS settings

To allow boot from SAN, perform the following steps in the system BIOS settings to disable the PXE BIOS.

1. Select System BIOS from the System Setup Main Menu.

111 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 2. Select Network Settings.
3. Click Disable for all PXE Devices.
4. Click Back.
5. Click Finish, click Finish again, then select Yes to exit and reboot.

Note: As previously documented, this server configuration may be used to generate a template to deploy to other
servers with identical hardware. When a template is not used, repeat the steps in this chapter for each MX server
sled that requires access to the FC storage.

10.6.3 Connect FCoE LUN

The server should be provisioned to connect to an FCoE boot LUN before moving on to Section 10.6.4.
Follow the procedures in Appendix B to configure and connect to an FCoE LUN. Once connected, continue
to the steps below to complete the Boot from SAN configuration.

10.6.4 Set up install media connection

Note: The steps in this section were completed using the iDRAC Java Virtual Console.

1. Connect to the server’s iDRAC in a web browser and launch the virtual console.
2. In the virtual console, from the Virtual Media menu, select Virtual Media.
3. In the virtual console, from the Virtual Media menu, select Map CD/DVD.
4. Click Browse to find the location of the OS install media then click Map Device.
5. In the virtual console, from the Next Boot menu, select Lifecycle Controller.
6. Reboot the server.

10.6.5 Set up OS driver install media using Lifecycle Controller

Some Operating System’s install media do not contain the necessary FCoE drivers to boot from a FCoE LUN.
Use this procedure to create an internal OS install media device. For VMware ESXi, refer to the Dell
customized media instructions provided on the Dell EMC Support website.

1. In Lifecycle Controller, select OS Deployment, then select Deploy OS.

2. From the Select an Operating System screen, verify that Boot mode is set to UEFI.
3. Select an OS to be installed to the boot LUN.

Lifecycle Controller OS deployment menu

4. Click Next.
5. Click the Manual Install check box, then click Next.
6. Click Next on the Insert OS Media screen.

112 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 7. Click Finish when prompted on the Reboot System screen.
8. System reboots to Virtual Media. Press any key to boot install media when prompted.
9. Follow the OS prompts to install the OS to the FCoE storage LUN.

113 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

A Hardware used in this document
This section covers the rack-mounted networking switches used in the examples in this guide.

For detailed information on Hardware components related to MX Platform please see the Dell EMC
PowerEdge MX Networking Architecture Guide.

A.1 Dell EMC PowerSwitch S3048-ON

The Dell EMC PowerSwitch S3048-ON is a 1-Rack Unit (RU) switch with forty-eight 1GbE BASE-T ports and
four 10GbE SFP+ ports. In this document, one S3048-ON supports out-of-band (OOB) management traffic for
all examples.

Dell EMC PowerSwitch S3048-ON

A.2 Dell EMC PowerSwitch Z9100-ON

The Dell EMC PowerSwitch Z9100-ON is a 1-RU multilayer switch with thirty-two QSFP28 ports supporting
10/25/40/50/100GbE and two 10GbE SFP+ ports. A pair of Z9100-ON switches are used as leaf switches in
Scenario 1 in this guide.

Dell EMC PowerSwitch Z9100-ON

A.3 Dell EMC PowerSwitch S4148U-ON

The Dell EMC PowerSwitch S4148U-ON is a 1-Rack Unit (RU) switch with 48x SFP+ ports, 2x QSFP+ ports,
and 4x QSFP28 ports. In this document, two S4148U-ON supports storage traffic, and is the first of two leaf
switch options.
Stack ID

Dell EMC PowerSwitch S4148U-ON

A.4 Dell EMC PowerSwitch Z9264F-ON

The Dell EMC PowerSwitch Z9264F-ON is a 2RU fixed form-factor 100GbE multi-rate switch is optimized for
non-blocking 100GbE leaf/spine fabrics and high-density 25/50GbE in-rack server and storage connections.

114 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Provides up to 64 ports of 100GbE QSFP28 or up to 128 ports of 1/10/25/40/50GbE ports using breakout
cables. This switch may be used as a leaf or spine switch in a Leaf-spine topology.

Dell EMC PowerSwitch Z9264F-ON

A.5 Dell EMC Unity 500F storage array

The Unity 500F storage platform delivers all-flash storage with up to 8PB raw capacity. It has concurrent
support for NAS, iSCSI, and FC protocols. The Disk Processing Enclosure (DPE) has a 2-RU form factor,
redundant Storage Processors (SPs), and supports up to twenty-five 2.5" drives. Additional 2-RU Disk Array
Enclosures (DAEs) may be added providing twenty-five additional drives each. This array is used for the FC
connections in this document.

Dell EMC Unity 500F front view

Dell EMC Unity 500F rear view

A.6 Cisco Nexus 3232C

The Cisco Nexus 3232C is a 1-RU fixed form-factor 100GbE switch with thirty-two QSFP28 ports supporting
10/25/40/50/100GbE. A pair of Cisco Nexus 3232C switches are used as leaf switches in Scenario 2 in this
guide.

115 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

B Dell EMC Unity information
This section shows how an administrator can determine the WWPNs of CNAs and storage targets. The
WWPNs are used to connect FC storage targets to specific servers for file storage or OS boot.

B.1 Determine Unity 500F storage array FC WWPNs

The WWPNs of FC adapters in storage arrays are also used for FC configuration. WWPNs on Unity storage
arrays are determined as follows:

1. Connect to the Unisphere GUI in a web browser and log in.

2. Click the Settings icon near the top-right corner of the page.
3. In the left pane of the Settings window, select Access, then Fibre Channel.

The Fibre Channel Ports page is displayed as shown in Figure 100. A zoomed-in view of the area inside the
red box is shown in Figure 101.

Unisphere Fibre Channel Ports page

Zoomed-in view of SP A and SP B WWNs

Two WWNs are listed for each port. The World Wide Node Name (WWNN), outlined in black, identifies this
Unity storage array (the node). The WWPNs, outlined in blue, identify the individual ports. Record the
WWPNs as shown in Table 9.

Storage array FC adapter WWPNs

Service processor Port WWPN

SP A 0 50:06:01:66:47:E0:1B:19

SP A 1 50:06:01:67:47:E0:1B:19

SP B 0 50:06:01:6E:47:E0:1B:19

SP B 1 50:06:01:6F:47:E0:1B:19

116 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

B.2 CNA FCoE port WWPNs
In this example the MX740c server's FCoE adapter World Wide Port Names (WWPNs) are used for FC
connection configuration. Adapter WWPNs are determined as follows:

1. Connect to the first server's iDRAC in a web browser and log in.
2. Select System, then click Network Devices.
3. Click the CNA. In this example, it is NIC Mezzanine 1A. Under Ports and Partitioned Ports, the
FCoE partition for each port is displayed as shown in Figure 102:

FCoE partitions in iDRAC

4. The first FCoE partition is Port 1, Partition 2. Click the icon to view the MAC Addresses as
shown in Figure 103:

MAC address and FCoE WWPN for CNA port 1

5. Record the MAC Address and WWPN, outlined in red above.

Note: A convenient method is to copy and paste these values into a text file.

117 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 6. Repeat steps 4 and 5 for the FCoE partition on port 2.
7. Repeat in this section for the remaining MX740c servers.

The FCoE WWPNs and MAC addresses used in this deployment example are shown in Table 10.

Server CNA FCoE port WWPNs and MACs

Server Port WWPN MAC

MX740c-1 1 20:01:18:66:DA:71:50:AD 18:66:DA:71:50:AC

MX740c-1 2 20:01:18:66:DA:71:50:AF 18:66:DA:71:50:AE

MX740c-2 1 20:01:18:66:DA:77:D0:C3 18:66:DA:77:D0:C2

MX740c-2 2 20:01:18:66:DA:77:D0:C5 18:66:DA:77:D0:C4

B.3 Configure Unity FC storage

This section covers configuration of a Dell EMC Unity 500F storage array. Refer to the storage system
documentation for other FC storage devices.

B.3.1 Create a storage pool

1. Connect to the Unisphere GUI in a web browser and log in using the required credentials.
2. In the left pane under STORAGE, select Pools.
3. Click the (+) icon.
4. In the Create Pool dialog box, provide a name in the required field, then click Next.
5. Select appropriate storage tiers and RAID configuration for the pool, then click Next.
6. Under Select Amount of Storage, select the desired number of drives. The total number of drives
and the total capacity will be displayed next to Totals. Click Next
7. The Capability Profile Name section is optional. Click Next
8. Review selections on the Summary page and click Finish to create the pool. Once the Overall
status shows 100%, click Close.
9. The pool is displayed on the STORAGE > Pools page as shown in Figure 104.

118 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 Storage pool created

B.3.2 Add hosts

1. In the Unisphere left pane under ACCESS, select Hosts.
2. On the Hosts tab, click the (+) icon, then click Host.
3. Enter the Name of the server in the field provided, then click Next.
4. Select the Initiator IQN/WWN checkbox in the Discovered Initiators panel.
5. Click Next.
6. Click Finish in the Review the host configuration page.

Note: Additional hosts may be added as needed by clicking the icon from the Hosts tab.

B.3.3 Create LUNs and configure host access

1. In the Unisphere left pane under STORAGE, select Block
2. On the LUNs tab, click the (+) icon to open the Create LUNs dialog box.
3. On the Configure LUN(s) page, select the Number of LUNs.
4. Provide a name in the field provided, then select the Storage Pool.
5. From the Size section, make the required modifications, then click Next.
6. On the Access page, click the (+) icon and the select host(s) to be granted access to the LUN.
7. Click OK, then click Next.
8. On the Snapshot page, leave settings at their defaults and click Next.
9. On the Replication page, leave settings at their defaults and click Next.

119 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

 10. On the Summary page, review the details and click Finish to create the LUN.
11. On the Results page, click Close when Overall status shows 100% Completed.

The newly created LUN is now visible on the LUNs tab as shown in Figure 105. In this example, a LUN
named FC-80GB that is 80GB in size has been created.

LUN Created

Create additional LUNs and grant access (map) to hosts as needed.

Note: To modify host access at any time, check the box next to the LUN to select it. Click the pencil icon, and
select the Host Access tab.

120 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

C Additional information

C.1 Delete MCM group

To remove an MCM group using the OME-M console, perform the following steps:
1. Open the OME-M console.
2. In the MCM group pane, click the name of the lead chassis.
3. From the Configure menu, select Delete Group.
4. In the Delete Group dialog box, click Confirm.

At this point, the OME-M console removes the MCM group. To manage the chassis, use the individual IP
addresses assigned to each.

C.2 Reset chassis using RACADM

To reset the chassis to factory default settings, perform the following steps:
1. Connect to the MX9002m IP address using SSH. The default username is root, and the
default password is calvin.
2. In the RACADM shell, run the racadm racresetcfg command.
3. The factory reset process is initiated, and a status message displays.

Note: The process takes several minutes to complete.

Optionally, after the reset process is complete, use the LCD screen to reassign a static IP address.

C.3 Reset SmartFabric OS10 switch to factory defaults

To reset SmartFabric OS10 switches back to the factory default configuration, enter the following commands:

OS10# delete startup-configuration

Proceed to delete startup-configuration [yes/no(default)]:yes

OS10# reload

System configuration has been modified. Save? [yes/no]:no

Proceed to reboot the system? [confirm yes/no]:yes

The switch reboots with default configuration settings.

C.4 Reset Cisco Nexus 3232C to factory defaults

To reset the Cisco Nexus 3232C switches to the factory default configuration, enter the following commands:

3232C# write erase

Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y

After the next reboot the switch loads with default configuration settings.

121 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

C.5 Connect to IO Module console port via RACADM
To connect to an IOM console port, first connect to the OME-Modular IP address using SSH using the same
credentials used to login to the OME-M GUI.

Use the RACADM command from the MX9002m management module:

racadm connect [-b] -m <module>
-b is for Binary mode
-m is the Module option. The module option can be one of the following:
server-<n>: where n = 1 to 8
switch-<n>: where n = 1 to 6 or <a1 | a2 | b1 | b2 | c1 | c2>

For example:
Connect to I/O Module 1 serial console:
racadm connect -m switch-1

Connect to Server 1 serial console:

racadm connect -m server-1

To exit the IOM or server and switch back to racadm shell, use command ~.

122 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

D Validated components

D.1 Scenarios 1 and 2

The following tables include the hardware, software, and firmware used to configure and validate Scenario 1
and Scenario 2 in this document.

D.1.1 Dell EMC PowerSwitches

Dell EMC PowerSwitch switches and OS versions – Scenarios 1 and 2
Qty Item Version
2 Dell EMC PowerSwitch Z9100-ON leaf switches 10.4.0E(R3)
1 Dell EMC PowerSwitch S3048-ON OOB management switch 10.4.0E(R3P2)

D.1.2 Dell EMC PowerEdge MX7000 chassis and components

Dell EMC PowerEdge MX7000 chassis and components – Scenarios 1 and 2
Qty Item Version
2 Dell EMC PowerEdge MX7000 chassis -
4 Dell EMC PowerEdge MX740c sled (2 per chassis) -
4 Dell EMC PowerEdge M9002m modules (2 per chassis) 1.10.00
2 Dell EMC Networking MX9116n FSE (1 per chassis) 10.5.0.1
2 Dell EMC Networking MX7116n FEM (1 per chassis) -

MX740c sled details – Scenarios 1 and 2

Qty per
Item Firmware Version
sled
1 Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz -
12 16GB DDR4 DIMMs (192GB total) -
1 Boot Optimized Storage Solution (BOSS) Controller w/ 2x240GB 2.6.13.2008
SATA SSDs
1 PERC H730P MX 25.5.3.0005
3 600GB SAS HDD -
1 Intel(R) Ethernet 25G 2P XXV710 mezzanine card 18.5.17
- BIOS 1.0.1
- iDRAC with Lifecycle Controller 3.20.20.20

123 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

D.1.3 Cisco Nexus switches
Nexus switches and OS versions – Scenarios 1 and 2
Qty Item Version
2 Cisco Nexus 3232C 7.0(3)I4(1)

D.2 Scenarios 3 and 4

The tables in this section include the hardware, software, and firmware used to configure and validate the
examples in this document.

Switches and OS versions

Qty Item Software Version
1 Dell EMC PowerSwitch S3048-ON management switch 10.4.0E(R3P2)
2 Dell EMC Networking MX9116n FSE 10.5.0.1
2 Dell EMC Networking MX5108 10.5.0.1
2 Dell EMC PowerSwitch S4148U-ON 10.4.0E(R3)
2 Dell EMC Networking MX7116n FEM --

MX-series components
Qty Item Version
4 Dell EMC PowerEdge M9002m modules 1.10.00
4 Dell EMC PowerEdge MX740c compute sleds A01

MX740c compute sled details

Qty per
Item Firmware Version
sled
1 QLogic QL41262HMKR (25G) mezzanine CNA 14.07.07

2 Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz --

12 16GB DDR4 DIMMs (192GB total) --
1 Boot Optimized Storage Solution (BOSS) Controller w/ 2x240GB 2.6.13.3011
SATA SSDs
1 PERC H730P MX 25.5.5.0005
3 600GB SAS HDD --
- Dell OS Driver Pack 18.07.13
- BIOS 1.0.2
- iDRAC with Lifecycle Controller 3.20.20.20

124 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

E Technical resources
Dell EMC Networking Guides

Dell EMC PowerEdge MX IO Guide

Dell EMC PowerEdge MX Network Architecture Guide

Dell EMC PowerEdge MX SmartFabric Deployment Video

Dell EMC PowerEdge MX SmartFabric Deployment with Cisco ACI Video

MX Port-Group Configuration Errors Video

MX Port-Group Configuration Video

Dell EMC OpenManage Enterprise-Modular Edition User's Guide v1.00.01

OS10 Enterprise Edition User Guide for PowerEdge MX IO Modules Release 10.4.0E R3S

Dell EMC SmartFabric OS10 User Guide

Manuals and documents for Dell EMC PowerEdge MX7000

Manuals and documents for Dell EMC PowerSwitch MX9116n

Manuals and documents for Dell EMC PowerSwitch S3048-ON

Manuals and documents for Dell EMC PowerSwitch Z9100-ON

Dell EMC OME-M v1.00.01 for PowerEdge MX7000 Chassis User's Guide

Dell EMC Networking Layer 3 Leaf-Spine Deployment and Best Practices with OS10

FCoE-to-Fibre Channel Deployment with S4148U-ON in F_port Mode

Fibre Channel Deployment with S4148U-ON in F_port Mode

Manuals and documentation for Dell EMC Networking S4148U-ON

125 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide

F Support and feedback
Contacting Technical Support

Support Contact Information Web: http://www.dell.com/support

Telephone: USA: 1-800-945-3355

Feedback for this document

Dell EMC encourages readers to provide feedback on the quality and usefulness of this publication by
sending an email to [email protected]

126 Dell EMC PowerEdge MX SmartFabric Configuration and Troubleshooting Guide