A Layer 3 Underlay architecture for Vxlan

L3 Routing Underlay

Everyone is aware of the benefits of restricting L2 broadcast domains to the Top of the Rack switches. ( Route when you can, Switch when you must)
In a Leaf/Spine Network, Routing can be extended until the Leaf Switches(TOR) with Point to Point Routing to each of the Spines. Traffic Hashing is done via ECMP on a per-flow basis.
There are some interesting demonstrations of deploying BGP in the data center as an IGP. of course, OSPF and ISIS can be used as well.


In the test-setup that I use, VLAN’s are restricted to the top of the rack switch. The TOR acts as the default gateway for servers connected to it. Hence there is no VRRP or additional spanning tree configs.
These Local vlans are -redistributed to the Spine(Core) via a L3 routing adjacacencies.

Most Overlay Technologies (SPB,Trill, VxLAN) have to solve the problem of the Flooding Domain. How to create a E-LAN or a tree such that BUM traffic could be flooded across efficiently and only to those end-points which need them.

Since Vxlan does not have a Control Plane (Unlike SPB/Trill which uses ISIS ) ; PIM Bidir is used since it is most efficient for the (*.G) joins which is needed by gateways to reflect dynamic VM presence.
There is a another mode of running Vxlan called Headend Mode which does not need PIM deployment.

In future posts, I will demonstrate how the OmniSwitch OS6900 series of switches act as a Vxlan Gateway (Hardware Vxlan Tunnel Endpoints).

These Tunnel End points/VxLAN gateway are configured at the edges ( Leaf Switches). The Core does not need to support any additional new features except Layer 3 routing and PIM Bidir ( Optional) since they do not do any tunnel terminations or encapsulations.

The Use Cases I will demonstrate are:

a) Supporting Multi-tenancy with Vxlan Gateway Configuration on the AOS Switches
b) Interop with VMWare Esx 5.5 ( Over a Converged Network )
c) Interop with IP based Storage (ISCSI/NFS) Solution
d) Interop with Software Vxlan Tunnel Endpoints [OpenVSwitch(KVM/Qemu)]
e) Applying Quality of Service guarantees to a Virtualized Environment [including VM Snooping and differential Treatment based on Tenant/VNID at Core ]
e) User Network Profiles to apply Network Access Control to devices based on matching Characteristics.

Stay Tuned.

Bringing it all Together : Creating a FCoE Storage Solution

Now that you have learned how to Configure the Alcatel-Lucent OmniSwitch Series of DataCenter Switches for FCOE Storage Support, here is a quick note on the remaining pieces.
1) Configuring the Converged network Adaptors on Esxi

Depending upon the features that one requires on the Esxi hosts, the appropriate device drivers have to be downloaded & installed. It is straightforward to find the driver software at either Vmware (for esxi) or the Network Adaptor vendor’s website ( for Windows/Linux etc)
Installation via Command line would be through the esxcli vib install command. A reboot would be necessary in most cases.

It is also worth investigating whether a Remote management for the CNA’s ( either through a Vcenter plugin or a standAlone application) will make life easy.

2) Setting up NetApp E2660 Fiber-Channel Storage via Santricity Array management Software

Setting up Fiber-Channel Storage on the array is a matter of a few steps. Note that the configurations that I perform may not be Accurate nor COMPLETE for deployment. They are only listed here to show the complete list of steps to provide a proof of Concept for an FCoE Storage Solution.
2.a) Configure Storage Array –
Create Volume Groups
Create Volumes within the newly Created Volume Group
Create Hosts by creating a Host Port Identifier #
Create a Host group if required.
2.b) Map Volumes to Host/Host Groups for use in I/O Operations & associate with a Logical Unit Number


3) Setting up Fiber-Channel Switch (Brocade/Qlogic) Zoning

In order for Esxi hosts to see the DataStores, another additional but important task ( which can be overlooked) is putting the targets and initiators in the same zone so that they see each other.

Here is the link that I found useful for performing Zoning Config on the Brocade Silkworm Fiber-Channel Switch


Performing Zoning on the Qlogic 5810 is also very straight-forward if done via the Web interface.

if everything is set up correctly, from Vcenter, you should be able to see the LUN’s & Storage devices from the Storage Adaptors tab under the FCOE adaptors.

FC_SW_DC1:admin> zoneshow
Default Zone: OFF
Safe Zone: OFF
Defined configuration:
cfg: CFG4 all_esx
zone: all_esx 20:00:00:1b:21:d8:45:13; 21:00:00:0e:1e:11:0e:60;
21:00:00:0e:1e:11:51:c0; 21:00:00:24:ff:37:dd:b8;
21:00:00:0e:1e:12:e9:d1; 21:00:00:0e:1e:11:5b:61;
20:03:00:05:1e:35:fd:42; 20:07:00:05:1e:35:fd:42;
20:04:00:80:e5:2d:06:66; 20:44:00:80:e5:2d:06:66;
20:35:00:80:e5:2d:06:66; 50:0a:09:81:88:2a:a7:cc;
10:00:00:00:c9:e4:62:a0; 21:00:00:0e:1e:11:0e:61;
10:00:00:00:c9:bb:c2:43; 20:00:00:00:c9:bb:c2:43;
10:00:00:00:c9:bb:c2:3f; 10:00:00:00:c9:e4:62:a1

Effective configuration:
cfg: CFG4
zone: all_esx 20:00:00:1b:21:d8:45:13

FC_SW_DC1:admin> switchshow
switchName: FC_SW_DC1
switchType: 32.0
switchState: Online
switchMode: McDATA Fabric
switchRole: Subordinate
DomainIDOffset: 0x60
switchDomain: 1
switchId: fffc21
switchWwn: 10:00:00:05:1e:35:fd:42
zoning: ON (CFG4)
switchBeacon: OFF
FC Router: OFF
FC Router BB Fabric ID: 1

Index Port Address Media Speed State Proto
0 0 610000 id N4 Online FC F-Port 1 N Port + 4 NPIV public
1 1 610100 — N4 No_Module FC
2 2 610200 — N4 No_Module FC
3 3 610300 id N2 Online FC G-Port
4 4 610400 — N4 No_Module FC
5 5 610500 — N4 No_Module FC
6 6 610600 — N4 No_Module FC
7 7 610700 — N4 No_Module FC
8 8 610800 — N4 No_Module FC
9 9 610900 — N4 No_Module FC
10 10 610a00 id N2 Online FC E-Port 10:00:00:05:1e:35:fb:2a “FC_SW_DC2” (upstream)
11 11 610b00 id N4 Online FC F-Port 50:0a:09:81:88:2a:a7:cc
12 12 610c00 — N4 No_Module FC
13 13 610d00 — N4 No_Module FC
14 14 610e00 — N4 No_Module FC
15 15 610f00 — N4 No_Module FC
16 16 611000 id N4 Online FC F-Port 20:35:00:80:e5:2d:06:66
17 17 611100 — N4 No_Module FC
18 18 611200 — N4 No_Module FC
19 19 611300 — N4 No_Module FC
20 20 611400 — N4 No_Module FC
21 21 611500 — N4 No_Module FC
22 22 611600 — N4 No_Module FC
23 23 611700 — N4 No_Module FC
24 24 611800 — N4 No_Module FC
25 25 611900 — N4 No_Module FC
26 26 611a00 id N4 No_Light FC
27 27 611b00 — N4 No_Module FC
28 28 611c00 — N4 No_Module FC
29 29 611d00 — N4 No_Module FC
30 30 611e00 — N4 No_Module FC
31 31 611f00 — N4 No_Module FC
FC_SW_DC1:admin> portshow 0
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portType: 10.0
POD Port: Port is licensed
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 120
state transition count: 19

portId: 610000
portIfId: 43020001
portWwn: 20:00:00:05:1e:35:fd:42
portWwn of device(s) connected:
Distance: normal
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts: 190 Link_failure: 2 Frjt: 0
Unknown: 150397 Loss_of_sync: 57 Fbsy: 0
Lli: 190 Loss_of_sig: 60
Proc_rqrd: 26644714 Protocol_err: 0
Timed_out: 48 Invalid_word: 0
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 6
Suspended: 0 Lr_out: 6
Parity_err: 0 Ols_in: 3
2_parity_err: 0 Ols_out: 6
CMI_bus_err: 0

Port part of other ADs: No
FC_SW_DC1:admin> portshow 16
portHealth: HEALTHY

Authentication: None
portDisableReason: None
portCFlags: 0x1
portType: 10.0
POD Port: Port is licensed
portState: 1 Online
Protocol: FC
portPhys: 6 In_Sync portScn: 32 F_Port
port generation number: 0
state transition count: 1

portId: 611000
portIfId: 43020010
portWwn: 20:10:00:05:1e:35:fd:42
portWwn of device(s) connected:
Distance: normal
portSpeed: N4Gbps

LE domain: 0
FC Fastwrite: OFF
Interrupts: 36 Link_failure: 0 Frjt: 0
Unknown: 0 Loss_of_sync: 3 Fbsy: 0
Lli: 21 Loss_of_sig: 4
Proc_rqrd: 28925 Protocol_err: 0
Timed_out: 136 Invalid_word: 73
Rx_flushed: 0 Invalid_crc: 0
Tx_unavail: 0 Delim_err: 0
Free_buffer: 0 Address_err: 0
Overrun: 0 Lr_in: 1
Suspended: 0 Lr_out: 0
Parity_err: 0 Ols_in: 0
2_parity_err: 0 Ols_out: 1
CMI_bus_err: 0

Port part of other ADs: No
4) The next step would be to add new DataStores. The Options are either Disk/LUN or NFS. Choose the LUN that you have originally created on the NetApp Storage Device. The DataStore thus created should be visible from all the Esxi hosts which have access to the Storage device. This is essential for VM migration.





5) Bringing it all Together
Via vcenter, you can create a new VM on any of the hosts which has access to the newly created DataStore.



Creating a VM with FC Storage