Creating a Lossless infrastructure on the OS6900/OS10K in support of FCoE/Fiber-Channel Storage

 

Lossless Configuration on DataCenter

Creating a Lossless infrastructure on the OS6900/OS10K in support of FCoE/Fiber-Channel Storage

Fiber-Channel Storage systems are inherently Lossless. In support of Converged Networks (FCoE), a lossless infrastructue must be created over traditional Lossy ethernet fabrics.
TCP inherently supports a lossless mode via retransmissions and acknowledgements. Applications using UDP will have to implement their own recovery mechanisms.
FCoE as it runs over plain Ethernet will have to depend on Priority Flow Control, Enhanced Transmission Selection & DataCenter Bridging Exchange Protocol to create it.
If you want to learn more, go the IEEE webpage :

http://www.ieee802.org/1/pages/dcbridges.html

The OmniSwitch Series of Switches support DataCenter bridging protocols via the application of Profiles at a port Level.
In essence, The profiles are pre-defined templates which enable a particular priority for Lossless behavior ( at the ingress) ; Provide a bandwidth guarantee for a class of traffic at the egress
and enable auto-configuration of local/peers via LLDP/DCBx negotiations. Both the earlier CEE and current IEEE modes of DCBx are supported.

By Setting PFC and ETS Willing FLAGS accordingly , it is possible to steer the negotiations across the entire fabric to be consistent.

In most cases, the Servers are Willing and configure themselves to send FCOE traffic at a particular 802.1p priority based on LLDP negotiation with the Switch.

Lets take a predefined Template : DCP-8. DCP-8 is the default profile on all ports on bootup. It classifies traffic into 8 different traffic classes and schedules traffic based on Strict-Priority.
However, All traffic classes are non-Lossless. Hence they could be dropped when there is a congestion .

DC-EDGE-104-> show qos qsp dcb 8 tc

Legends: Linkdelay shown in KB.
* denotes user modified value

ETS ETS       ETS  ETS
Min Max ETS   Reco Reco  PFC  PFC
#   Name         TC Priorities BW  BW  Sched BW   Sched Mode LinkDelay
—+————+–+———-+—+—+—–+—-+—–+—-+———
8   dcp-8        0  0          0   100 SP    0    SP    nLL  0
8   dcp-8        1  1          0   100 SP    0    SP    nLL  0
8   dcp-8        2  2          0   100 SP    0    SP    nLL  0
8   dcp-8        3  3          0   100 SP    0    SP    nLL  0
8   dcp-8        4  4          0   100 SP    0    SP    nLL  0
8   dcp-8        5  5          0   100 SP    0    SP    nLL  0
8   dcp-8        6  6          0   100 SP    0    SP    nLL  0
8   dcp-8        7  7          0   100 SP    0    SP    nLL  0

In this example, we wish to use priority 3 for FCoE traffic. Hence we need to make priority 3 lossless.

Here we will create a Custom-Profile based on DCB-8 with PFC enabled for priority 3.

qos qsp dcb 83 import qsp dcb 8
qos qsp dcb 83 tc 3 pfc flow-type LL

DC-EDGE-104-> show qos qsp dcb 83 tc

Legends: Linkdelay shown in KB.
* denotes user modified value

ETS ETS       ETS  ETS
Min Max ETS   Reco Reco  PFC  PFC
#   Name         TC Priorities BW  BW  Sched BW   Sched Mode LinkDelay
—+————+–+———-+—+—+—–+—-+—–+—-+———
83  dcp-83       0  0          0   100 SP    0    SP    nLL  0
83  dcp-83       1  1          0   100 SP    0    SP    nLL  0
83  dcp-83       2  2          0   100 SP    0    SP    nLL  0
83  dcp-83       3  3          0   100 SP    0    SP    LL   52
83  dcp-83       4  4          0   100 SP    0    SP    nLL  0
83  dcp-83       5  5          0   100 SP    0    SP    nLL  0
83  dcp-83       6  6          0   100 SP    0    SP    nLL  0
83  dcp-83       7  7          0   100 SP    0    SP    nLL  0

Now that the Custom Profile is created, We need to apply it to all the Edge-facing ports on the Core.
It is also required to Configure the Core to be non-willing, So that it does not change its PFC/ETS behavior based on LLDP packets received.
The Edge should remain willing. This ensures that the configuration is applied at a central place at the Core and pushed downwards to the edges.

On DC-CORE-02/DC-CORE-01 ( Push Lossless Configs from Core Switch to Edge Switch Links)
————————-
qos qsi linkagg 22 qsp dcb “dcp-83”
qos qsi linkagg 22 dcb dcbx ets willing no
qos qsi linkagg 22 dcb dcbx pfc willing no

As PFC/ETS is required to be configured at a per-link level, we have to do the same at the edge-port connected to the server. In addition, we have to configure the LLDP Application TLV.
This is information which the Server acts upon to configure Lossless behavior and start the FCOE discovery process.

on DC-EDGE-104 port 1/1/18 ( Push Lossless Configs from Edge Switch to Server Link)
—————————
qos qsi port 1/1/18 qsp dcb “dcp-83”
qos qsi port 1/1/18 dcb dcbx pfc willing no
qos qsi port 1/1/18 dcb dcbx ets willing no
qos qsi port 1/1/18 stats admin-state enable

lldp nearest-bridge port 1/1/18 tlv application enable
lldp port 1/1/18 tlv application fcoe priority 3

Virtual Flow Control Show Commands to check if
DC-EDGE-104-> show qos qsi port 1/1/18 dcb dcbx status
DCBX
Local   Local Remote
DCBX Oper    Oper  Oper
Port       Oper Changed Ver   Ver    Error                     Action
———-+—-+——-+—–+——+————————-+—————————-
1/1/18     Ena  No      CEE   CEE    No                        –

DC-EDGE-104-> show qos qsi port 1/1/18 dcb pfc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm  Loc-Adm       Loc-Oper Loc-Oper         Remote Remote     802.3x
Port       Priorities MBC Will Cap Priorities MBC Will Cap Priorities MBC Will Cap Pause-Ready
———-+———-+—+—-+—+———-+—+—-+—+———-+—+—-+—+———–
1/1/18     3          No  No   8   3          No  No   8   34         No  Yes  2   No

DC-EDGE-104-> show lldp port 1/1/18 remote-system application-tlv
Chas/
Slot/      Remote
Port       ID            Selector               Protocol     Priority
———-+———–+————————-+———–+————
1/1/18     7             Ethertype                0x8906      3  (fcoe)
1/1/18     7             Ethertype                0x8914      3
1/1/18     7             Tcp/Sctp                 3260        4  (iscsi)

DC-EDGE-104-> show qos qsi port 1/1/18 dcb pfc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm  Loc-Adm       Loc-Oper Loc-Oper         Remote Remote     802.3x
Port       Priorities MBC Will Cap Priorities MBC Will Cap Priorities MBC Will Cap Pause-Ready
———-+———-+—+—-+—+———-+—+—-+—+———-+—+—-+—+———–
1/1/18     3          No  No   8   3          No  No   8   34         No  Yes  2   No

DC-EDGE-104-> show qos qsi port 1/1/18 dcb ets tc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm    Loc-Adm   Loc-Adm    Loc-Oper   Loc-Oper  Loc-Oper   Remote     Remote    Rem-Reco
Reco Reco                       Reco Reco
Port       TC Priorities BW  Sched BW   Sched Priorities BW  Sched BW   Sched Priorities BW  Sched BW  Sched
———-+–+———-+—+—–+—-+—–+———-+—+—–+—-+—–+———-+—+—–+—+—–
1/1/18     0  0          0   SP    0    SP    0          0   SP    0    SP    0124567    50  ETS   –   –
1/1/18     1  1          0   SP    0    SP    1          0   SP    0    SP    3          50  ETS   –   –
1/1/18     2  2          0   SP    0    SP    2          0   SP    0    SP    –          –   –     –   –
1/1/18     3  3          0   SP    0    SP    3          0   SP    0    SP    –          –   –     –   –
1/1/18     4  4          0   SP    0    SP    4          0   SP    0    SP    –          –   –     –   –
1/1/18     5  5          0   SP    0    SP    5          0   SP    0    SP    –          –   –     –   –
1/1/18     6  6          0   SP    0    SP    6          0   SP    0    SP    –          –   –     –   –
1/1/18     7  7          0   SP    0    SP    7          0   SP    0    SP    –          –   –     –   –

Advertisements

Configuring the OmniSwitch OS6900/OS10K for FIP-Snooping.

Multi-Hop FCOE Deployment

1.2 Configuring the OmniSwitch OS6900/OS10K for FIP-Snooping.

Please refer to the earlier post for steps to configure the NPIV Gateway. Once that is complete, the following should be configured in order to allow initiators connected to edge Switches to login to the Fibre-Channel Fabric.

a) Configure FCOE vlan and FIP-Snooping on Core and Edge Switches .
b) Configure port roles for Security ( Please refer to T11 FC-BB-5 documents)

c) Create a Lossless Fabric (PFC)& Bandwidth guarantees for Storage & TCP traffic( ETS)
d) Enable DataCenter Bridging Exchange Protocol and Application TLV’s to auto-configure the Converged Network adaptors on the Servers.

While configuration notes for the the first two items are shown  below, Creating a Lossless Infrastructure using DataCenter bridging protocols warrants a seperate post by itself.

Sample Configuration on DC-EDGE-104
====================================
fcoe fip-snooping admin-state enable
fcoe vlan 252 admin-state enable
fcoe vlan 252 name “FCoE VLAN”

fcoe port 1/1/18 role edge [Configure Port connected to Server as an Edge Port. This enables dynamic ACL’s which configures a level of security to the Initiator-Target communication ]
fcoe linkagg 22-23 role mixed [ Core Facing Ports can be either configured as Mixed/Trusted/FCF-Only/E-Node only based on the location in the network relative to the FCF or the Enodes]
vlan 252 members port 1/1/18 tagged [Tag both edge and network facing ports with the fcoe vlan]
vlan 252 members linkagg 22-23 tagged

Sample Configuration on Core-Switches DC-CORE-01 & DC-CORE-02
==============================================================

fcoe vlan 252 admin-state enable
fcoe vlan 252 name “FCoE VLAN”
vlan 252 members linkagg 21 tagged
vlan 252 members linkagg 23 tagged
spantree vlan 252 admin-state enable
spantree vlan 252 priority 4096  [ By manually configuring STP priority, the topolocy change becomes deterministic upon root-bridge failures ; ensure Backup root bridge has second highest priority]

fcoe fip-snooping admin-state enable
fcoe linkagg 21 role mixed
fcoe linkagg 23 role mixed

Here are some show commands which can help troubleshoot fcoe problems

DC-EDGE-104-> show fcoe fcf
FCF-MAC         VLAN    Config     Sessions  A-bit   MaxFrmVer  Priority
——————–+——-+———–+———+——–+———+———-
E8:E7:32:36:1E:F6    252      Dynamic     0         1         no        128

DC-EDGE-104-> show fcoe enode
Port        Enode MAC        VLAN    Sessions
——–+——————-+——-+———–
1/1/18   00:0E:1E:11:0E:69   252        1

DC-EDGE-104-> show fcoe ?
^
VSAN-MAP STATISTICS SESSIONS PORTS FILTERING-RESOURCES
FCF FC-MAP ENODE E-TUNNEL DISCOVERY-ADVERTISEMENT <cr>
(Fips Command Set)

DC-EDGE-104-> show fcoe statistics ?
^
VLAN R-NPIV PORT NPIV-PROXY LINKAGG
INTERFACE FCF ENODE E-TUNNEL
(Fips Command Set)

DC-EDGE-104-> show fcoe statistics vlan 252
Enode Statistics

VLAN     Sess      MDS     UDS     FLOGI     FDISC    LOGO       E KA       VN KA
——–+——–+——–+——–+——–+——–+——–+————+————
252            0     1209        8        5        0        2         9593          851
FCF Statistics

VLAN     Sess      MDA       UDA    FLOGI_ACC  FLOGI_RJT  FDISC_ACC  FDISC_RJT  LOGO_ACC  LOGO_RJT     CVL
——–+——–+——–+———-+———-+———-+———-+———-+———+———+————
252            0     9577          2          0          0          0          0         0         0            2

NPIV Enode-Discovery Statistics

Packets Received:
Vlan    Vlan Req    MDS       UDS
———+———+———+———
252              0      1209         0
Packets Sent:
Vlan   Vlan Res     MDA       UDA
———+———+———+———
252              0         0         0

NPIV Enode-Login Statistics

Packets Received:
Vlan    FLOGI     FDISC     LOGO      E_KA      VN_KA
——-+———+———+———+———+———
252            5         0         2      9593       851
Packets Sent:
Vlan  FLOGI_ACC FDISC_ACC FLOGO_ACC FLOGI_RJT FDISC_RJT FLOGO_RJT    CVL
——-+———+———+———+———+———+———+———
252            0         0         0         0         0         0         0

R-NPIV FCF-Discovery Statistics

Packets Received:
Vlan      MDA       UDA
———+———+———
252              0         0
Packets Sent:
Vlan      MDS       UDS
———+———+———
252              0         0

R-NPIV Node-Login Statistics

Packets Received:
Vlan   FLOGI_ACC FDISC_ACC FLOGI_RJT FDISC_RJT   CVL
———+———+———+———+———+———-
252              0         0         0         0         0
Packets Sent:
Vlan     FLOGI     FDISC     LOGO      VN-KA     E-KA
———+———+———+———+———+———
252              0         0         0         0         0

Configuring the OmniSwitch OS6900 with an XNI-U12E Fiber-Channel module expansion module as an NPIV Gateway

OS6900 NPIV Gateway

Outlined Below are the list of Configuration steps to make the OS6900 act as an NPIV Gateway. The reference topology is attached.

! VLAN:
fcoe vlan 252 admin-state enable [ Create an FCOE vlan]
fcoe vlan 252 name “FCoE VLAN”

! FCOE:
fcoe fip-snooping admin-state enable [ Enable the Fiber-Channel/FCOE protocol on the Switch]

fibre-channel port 1/2/9 mode NP [Connect Fiber-Channel Port 1/2/9 to a Fiber-Channel Switch and configure the mode as NP]
fibre-channel vsan 252 admin-state enable name VSAN252 [ Create a Fiber-Channel VSAN ; Note that this a logical construct to bind the FCOE vlan to the Storage Area Network]
fibre-channel vsan 252 members port 1/2/9 [ Bind the VSAN to the Fiber-Channel NP port]
fcoe vsan-map vsan 252 vlan 252 [ Map Fiber-Channel VSAN with Fcoe VLAN]

! Virtual Flow Control:
qos qsp dcb 16 import qsp dcb “dcp-8” 802.3x-pause [ Create a Custom DataCenter bridging profile which enables PAUSE]

qos qsi port 1/2/9 dcb dcbx pfc tlv disable
qos qsi port 1/2/9 dcb dcbx pfc willing no
qos qsi port 1/2/9 qsp dcb “dcp-16”  [ Associate the Profile to the Port. This enables the Port to participate in Fiber-Channel Flow Control by transmitting/receiving R_RDY ]
qos qsi port 1/2/9 dcb dcbx admin-state disable [ Disable DataCenter bridging negotiations on the Fiber-Channel ports as it is only applicable on Ethernet ports]
qos qsi port 1/2/9 dcb dcbx ets willing no
qos qsi port 1/2/9 stats admin-state enable

Here are some Useful validation Commands to check if the OS6900 is indeed acting as an NPIV Gateway :

DC-EDGE-103-> show fcoe fcf
FCF-MAC         VLAN    Config     Sessions  A-bit   MaxFrmVer  Priority
——————–+——-+———–+———+——–+———+———-
E8:E7:32:36:1E:F6    252      Npiv        4         1         no        0        >>>> Note that the Switch has started to act as a FC Forwarder and sends Advertisements on the FCOE VLAN.

DC-EDGE-103-> show module status
Operational
Chassis/Slot     Status      Admin-Status       MAC
————–+————-+————+——————
1/CMM-A        UP             POWER ON      e8:e7:32:36:1e:f5
1/SLOT-1       UP             POWER ON      e8:e7:32:36:1e:fc
1/SLOT-2       UP             POWER ON      e8:e7:32:94:68:14

DC-EDGE-103-> show fibre-channel port info
Legend: NP=Proxy N_Port, F=Fabric Port connected to N_port, TE=Tunnel E-Port

ports  mode  fcid        wwpn                   state
——-+—-+——–+————————-+————-

1/2/9   NP   61:00:00  10:00:E8:E7:32:94:68:1C  Up             >>>> The NP Port is UP and it has received an FCID from the Fiber-Channel Switch.

DC-EDGE-103-> show interfaces 1/2/9
Chassis/Slot/Port  1/2/9  :
Operational Status     : up,
Last Time Link Changed : Thu Apr 28 17:35:20 2016,
Number of Status Change: 1,
Type                   : Fiber Channel,
SFP/XFP                : SFP_FC_2_4_8G,
EPP                    : Disabled,
Link-Quality           : N/A,
WWPN                   : 10:00:e8:e7:32:94:68:1c,
BandWidth (Megabits)   :     4000,             Duplex           : Full,  >>>> Port Speed at 4G ; Note that the Port can also negotiate to 8G with capable FC switches ( for ex: Qlogic 5810)
Autonegotiation        :   1  [ 8000-F 4000-F 2000-F         ],
Long Frame Size(Bytes) : 2148,
Rx              :
Bytes Received  :            530897348, Unicast Frames :               401568,
Broadcast Frames:                    0,
UnderSize Frames:                    0, OverSize Frames:                    0,
Lost Frames     :                    0, Error Frames   :                    0,
CRC Error Frames:                    0, Link Failures  :                    1,
B2B Credit zero :                    0
Tx              :
Bytes Xmitted   :              6107412, Unicast Frames :                92898,
Broadcast Frames:                    0,
Lost Frames     :                    0, Error Frames   :                    0,
B2B Credit zero :                  141

Here is the Output on the FC Switch

FC_SW_DC1:admin> switchshow
switchName:     FC_SW_DC1
switchType:     32.0
switchState:    Online
switchMode:     McDATA Fabric
switchRole:     Subordinate
DomainIDOffset: 0x60
switchDomain:   1
switchId:       fffc21
switchWwn:      10:00:00:05:1e:35:fd:42
zoning:         ON (CFG4)
switchBeacon:   OFF
FC Router:      OFF
FC Router BB Fabric ID: 1

Index Port Address Media Speed State     Proto
==============================================
0   0   610000   id    N4   Online      FC  F-Port  1 N Port + 5 NPIV public

FC_SW_DC1:admin> portshow 0 ( As you can see some of the initiators have already logged in, and I will show how this is done in subsequent posts)
portName:
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portFlags: 0x25b03       PRESENT ACTIVE F_PORT G_PORT U_PORT NPIV LOGICAL_ONLINE RRDY_MODE LOGIN NOELP LED ACCEPT FLOGI
portType:  10.0
POD Port: Port is licensed
portState: 1    Online
Protocol: FC
portPhys:  6    In_Sync         portScn:   32   F_Port
port generation number:    114
state transition count:    17

portId:    610000
portIfId:    43020001
portWwn:   20:00:00:05:1e:35:fd:42
portWwn of device(s) connected:
20:00:00:1b:21:d8:45:13
21:00:00:24:ff:37:dd:b8
21:00:00:0e:1e:11:5b:61
21:00:00:0e:1e:11:0e:61
10:00:e8:e7:32:94:68:1c
Distance:  normal
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts:        166        Link_failure: 2          Frjt:         0
Unknown:           150323     Loss_of_sync: 49         Fbsy:         0
Lli:               166        Loss_of_sig:  52
Proc_rqrd:         26638996   Protocol_err: 0
Timed_out:         48         Invalid_word: 0
Rx_flushed:        0          Invalid_crc:  0
Tx_unavail:        0          Delim_err:    0
Free_buffer:       0          Address_err:  0
Overrun:           0          Lr_in:        6
Suspended:         0          Lr_out:       6
Parity_err:        0          Ols_in:       3
2_parity_err:      0          Ols_out:      6
CMI_bus_err:       0

Port part of other ADs: No

 

UseCase 1 : Deploying An FCoE Storage Solution with the OmniSwitch OS6900 Data Center Switches

The OmniSwitch series of DataCenter Switches support the deployment of FCoE storage solutions through the implementation of following features :

FCoE

a) NPIV Gateway Support
b) FIP-Snooping on Top of Rack and Core Switches
c) Reverse NPIV.
d) E Port Tunneling features

Data_Center Bridging protocols in support of FCoE

a) A Lossless Fabric through support for Priority Flow Control (PFC)
b) BandWidth Sharing among Various Classes of traffic ( based on 802.1p/DSCP) via Enhanced Transmission Selection [ ETS]
c) Auto-Configuration of Lossless priority and bandwidth sharing via LLDP Application TLV support and DataCenter Bridging Exchange Protocols.

In the following posts, I will present the Configurations :

1.1 Configuring the OmniSwitch OS6900 with an XNI-U12E Fiber-Channel module expansion module as an NPIV Gateway
1.2 Configuring the OmniSwitch OS6900/OS10K for FIP-Snooping.
1.3 Configuring the OmniSwitch OS6900 with an XNI-U12E Fiber-Channel module expansion module for Reverse-NPIV.

2.1 Configuring DataCenter bridging Protocols (DCBx,PFC,ETS) and LLDP in support of the above features.

Stay Tuned …

Equipment in the Lab

Apart from the Alcatel-Lucent Enterprise OmniSwitch series of Switches [ OS6900,OS6860 & OS10K], the data-center houses the following equipment.

  • Fiber-Channel Switches ( Nexus 5K, Brocade Silkwom 4100, Qlogic 5800 )
  • NetApp E-2660 Fiber-Channel and Iscsi Storage ( managed by Santricity )
  • NetApp FA Series Storage
  • IoMega (EMC) NFS Server
  • Dell/HP Servers running Esxi 5.5 with Qlogic/Intel/Emulex Converged Network adapators and HBA’s
  • Dell/HP Servers running Ubuntu 14.04.3 LTS with Virtualization Support
  • Alcatel-Lucent Enterprise OmniAccess Series of SOHO routers.

Servers in the Data Center

I use Linux/OpenSource for most of my testing needs. I installed Ubuntu 14.04.3 LTS with Virtualization support because I was comfortable with its package management options.

Here are the servers that I installed on my system. Contrary to prevalent opinion, I found documentation for installation and setting up the services fairly easy to follow and straightforward.

Authentication, authorization, and accounting (AAA) – I use Both Tacacs & FreeRadius for AAA. OpenSource options for both are available.

FreeRadius – http://www.ubuntugeek.com/install-freeradius-on-ubuntu-15-04-server-and-manage-using-daloradius-freeradius-web-management-application.html

Tacacs – http://www.routingloops.co.uk/cisco/tacacs-on-ubuntu-14-04-lts/

Domain Name System (DNS) /Bind – https://help.ubuntu.com/lts/serverguide/dns.html

SFLOW –  ntopng http://idroot.net/tutorials/how-to-install-ntopng-on-ubuntu-14-04/

Syslog Server (rsyslogd) – https://community.spiceworks.com/how_to/65683-configure-ubuntu-server-12-04-lts-as-a-syslog-server

Network Time Protocol Server (NTP) – http://ubuntuforums.org/showthread.php?t=862620

NFS Server – https://help.ubuntu.com/community/SettingUpNFSHowTo & https://www.digitalocean.com/community/tutorials/how-to-set-up-an-nfs-mount-on-ubuntu-14-04

DHCP Server – The OmniSwitch series of switches have DHCP-Server support.

In addition, I have other applications running
Python with Paramiko for scripting support
Github/Golang
OpenSsh
TFTP
<To be updated as I remember>

Data Center Topology

MyTinyDataCenterThe Testbed is designed to simulate commonly deployed Data Center topologies.

The testbed is based on a Leaf & Spine design with OS6900’s and OS6860’s acting as Leaf Switches.

The Leaf switches are named DC-EDGE-101 through DC-EDGE-1xx. The Leaf Switches act as Default gateway for those subnets/vlans present on the switches. The subnets are isolated to the particular Leaf switch.

However, Some L2 domains such as FCOE vlans and a couple of other vlans to test L2 Multicast & Broadcast will stretch across the entire network.

In general, it is good networking practice to isolate L2 broadcast domains to a single switch, and use newer Encapsulation technologies such as VxLAN/SPB to support expansion of tenant/Customer domains. This also encourages us to think and implement networks via the Service model (Think Vxlan VNID or SPB isid rather than VLAN/Subnet).

Servers’s, ISCSI Storage devices and Fiber-Channel Switches are connected to the Leaf Switches. Fiber-Channel Storage is connected to the FC Switch. Remember, it is not a good idea to connect Servers directly to the Core Switch.

Each of the Leaf switches are connected via LACP Link-Aggregation or fixed port uplinks  to Spine Switches DC-CORE-01(vc OS10K) and DC-CORE-02 (vc OS6900x72 + x40).

These connections are L3 point-to-point and can either be OSPF/BGP/ISIS. In my testbed, all three adjacencies are present. It is only a matter of changing the route-redistribution scheme on each of the leaf switches to route the subnets on the Leaf Switches across the core via a particular protocol. By default route-distribution is done via OSPF. Refer Dr Google for Data Center technologies built with BGP ( Microsoft/Yahoo etc)

Since all three routing protocols support ECMP, traffic flow is across all the Spine Switches, as well as being fully redundant in case of a failure. Currently, The OS10K (DC-CORE-01) virtual-Chassis is configured as the rendezvous point for all PIM sparse groups. PIM-Bidir is used to create distribution trees for Vxlan.

In addition to routed traffic, there is also L2 traffic (FCOE, L2 Broadcast test traffic and L2 Multicast). For those L2 vlans which need to be propagated across multiple leaf switches, The OS6900 VC (DC-CORE-02) is configured to be the spanning-tree root bridge.
The OS10K (DC-CORE-01) is configured as the backup root bridge.

Work in Progress( Dual-Stacking)– Currently the DataCenter is predominantly IPV4. An IPV6 network is alsocreated across the same physical topology. The IPV6 addressing scheme closely follows the IPV4 addressing scheme.

A note about Redundancy

There is multiple levels of redundancy built in. Link-Aggregation with hashing ensures that traffic takes one of many ports. ECMP ensures that traffic takes one of many equal-cost paths with sub-second convergence.
The connections reflect test requirements. Real-datacenters might not have both L2 and L3 redundancy because the costs might scale up quite quickly.

 

Resources :

http://bradhedlund.com/2012/01/25/construct-a-leaf-spine-design-with-40g-or-10g-an-observation-in-scaling-the-fabric/

https://www.nanog.org/meetings/nanog55/presentations/Monday/Lapukhov.pdf

https://en.wikipedia.org/wiki/Multitenancy