Creating a Lossless infrastructure on the OS6900/OS10K in support of FCoE/Fiber-Channel Storage

 

Lossless Configuration on DataCenter

Creating a Lossless infrastructure on the OS6900/OS10K in support of FCoE/Fiber-Channel Storage

Fiber-Channel Storage systems are inherently Lossless. In support of Converged Networks (FCoE), a lossless infrastructue must be created over traditional Lossy ethernet fabrics.
TCP inherently supports a lossless mode via retransmissions and acknowledgements. Applications using UDP will have to implement their own recovery mechanisms.
FCoE as it runs over plain Ethernet will have to depend on Priority Flow Control, Enhanced Transmission Selection & DataCenter Bridging Exchange Protocol to create it.
If you want to learn more, go the IEEE webpage :

http://www.ieee802.org/1/pages/dcbridges.html

The OmniSwitch Series of Switches support DataCenter bridging protocols via the application of Profiles at a port Level.
In essence, The profiles are pre-defined templates which enable a particular priority for Lossless behavior ( at the ingress) ; Provide a bandwidth guarantee for a class of traffic at the egress
and enable auto-configuration of local/peers via LLDP/DCBx negotiations. Both the earlier CEE and current IEEE modes of DCBx are supported.

By Setting PFC and ETS Willing FLAGS accordingly , it is possible to steer the negotiations across the entire fabric to be consistent.

In most cases, the Servers are Willing and configure themselves to send FCOE traffic at a particular 802.1p priority based on LLDP negotiation with the Switch.

Lets take a predefined Template : DCP-8. DCP-8 is the default profile on all ports on bootup. It classifies traffic into 8 different traffic classes and schedules traffic based on Strict-Priority.
However, All traffic classes are non-Lossless. Hence they could be dropped when there is a congestion .

DC-EDGE-104-> show qos qsp dcb 8 tc

Legends: Linkdelay shown in KB.
* denotes user modified value

ETS ETS       ETS  ETS
Min Max ETS   Reco Reco  PFC  PFC
#   Name         TC Priorities BW  BW  Sched BW   Sched Mode LinkDelay
—+————+–+———-+—+—+—–+—-+—–+—-+———
8   dcp-8        0  0          0   100 SP    0    SP    nLL  0
8   dcp-8        1  1          0   100 SP    0    SP    nLL  0
8   dcp-8        2  2          0   100 SP    0    SP    nLL  0
8   dcp-8        3  3          0   100 SP    0    SP    nLL  0
8   dcp-8        4  4          0   100 SP    0    SP    nLL  0
8   dcp-8        5  5          0   100 SP    0    SP    nLL  0
8   dcp-8        6  6          0   100 SP    0    SP    nLL  0
8   dcp-8        7  7          0   100 SP    0    SP    nLL  0

In this example, we wish to use priority 3 for FCoE traffic. Hence we need to make priority 3 lossless.

Here we will create a Custom-Profile based on DCB-8 with PFC enabled for priority 3.

qos qsp dcb 83 import qsp dcb 8
qos qsp dcb 83 tc 3 pfc flow-type LL

DC-EDGE-104-> show qos qsp dcb 83 tc

Legends: Linkdelay shown in KB.
* denotes user modified value

ETS ETS       ETS  ETS
Min Max ETS   Reco Reco  PFC  PFC
#   Name         TC Priorities BW  BW  Sched BW   Sched Mode LinkDelay
—+————+–+———-+—+—+—–+—-+—–+—-+———
83  dcp-83       0  0          0   100 SP    0    SP    nLL  0
83  dcp-83       1  1          0   100 SP    0    SP    nLL  0
83  dcp-83       2  2          0   100 SP    0    SP    nLL  0
83  dcp-83       3  3          0   100 SP    0    SP    LL   52
83  dcp-83       4  4          0   100 SP    0    SP    nLL  0
83  dcp-83       5  5          0   100 SP    0    SP    nLL  0
83  dcp-83       6  6          0   100 SP    0    SP    nLL  0
83  dcp-83       7  7          0   100 SP    0    SP    nLL  0

Now that the Custom Profile is created, We need to apply it to all the Edge-facing ports on the Core.
It is also required to Configure the Core to be non-willing, So that it does not change its PFC/ETS behavior based on LLDP packets received.
The Edge should remain willing. This ensures that the configuration is applied at a central place at the Core and pushed downwards to the edges.

On DC-CORE-02/DC-CORE-01 ( Push Lossless Configs from Core Switch to Edge Switch Links)
————————-
qos qsi linkagg 22 qsp dcb “dcp-83”
qos qsi linkagg 22 dcb dcbx ets willing no
qos qsi linkagg 22 dcb dcbx pfc willing no

As PFC/ETS is required to be configured at a per-link level, we have to do the same at the edge-port connected to the server. In addition, we have to configure the LLDP Application TLV.
This is information which the Server acts upon to configure Lossless behavior and start the FCOE discovery process.

on DC-EDGE-104 port 1/1/18 ( Push Lossless Configs from Edge Switch to Server Link)
—————————
qos qsi port 1/1/18 qsp dcb “dcp-83”
qos qsi port 1/1/18 dcb dcbx pfc willing no
qos qsi port 1/1/18 dcb dcbx ets willing no
qos qsi port 1/1/18 stats admin-state enable

lldp nearest-bridge port 1/1/18 tlv application enable
lldp port 1/1/18 tlv application fcoe priority 3

Virtual Flow Control Show Commands to check if
DC-EDGE-104-> show qos qsi port 1/1/18 dcb dcbx status
DCBX
Local   Local Remote
DCBX Oper    Oper  Oper
Port       Oper Changed Ver   Ver    Error                     Action
———-+—-+——-+—–+——+————————-+—————————-
1/1/18     Ena  No      CEE   CEE    No                        –

DC-EDGE-104-> show qos qsi port 1/1/18 dcb pfc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm  Loc-Adm       Loc-Oper Loc-Oper         Remote Remote     802.3x
Port       Priorities MBC Will Cap Priorities MBC Will Cap Priorities MBC Will Cap Pause-Ready
———-+———-+—+—-+—+———-+—+—-+—+———-+—+—-+—+———–
1/1/18     3          No  No   8   3          No  No   8   34         No  Yes  2   No

DC-EDGE-104-> show lldp port 1/1/18 remote-system application-tlv
Chas/
Slot/      Remote
Port       ID            Selector               Protocol     Priority
———-+———–+————————-+———–+————
1/1/18     7             Ethertype                0x8906      3  (fcoe)
1/1/18     7             Ethertype                0x8914      3
1/1/18     7             Tcp/Sctp                 3260        4  (iscsi)

DC-EDGE-104-> show qos qsi port 1/1/18 dcb pfc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm  Loc-Adm       Loc-Oper Loc-Oper         Remote Remote     802.3x
Port       Priorities MBC Will Cap Priorities MBC Will Cap Priorities MBC Will Cap Pause-Ready
———-+———-+—+—-+—+———-+—+—-+—+———-+—+—-+—+———–
1/1/18     3          No  No   8   3          No  No   8   34         No  Yes  2   No

DC-EDGE-104-> show qos qsi port 1/1/18 dcb ets tc

Legends: * indicates port oper status is different than the configured status
Remote: Shows Remote Operational for IEEE port.
Shows Remote Configured for CEE port.

Loc-Adm    Loc-Adm   Loc-Adm    Loc-Oper   Loc-Oper  Loc-Oper   Remote     Remote    Rem-Reco
Reco Reco                       Reco Reco
Port       TC Priorities BW  Sched BW   Sched Priorities BW  Sched BW   Sched Priorities BW  Sched BW  Sched
———-+–+———-+—+—–+—-+—–+———-+—+—–+—-+—–+———-+—+—–+—+—–
1/1/18     0  0          0   SP    0    SP    0          0   SP    0    SP    0124567    50  ETS   –   –
1/1/18     1  1          0   SP    0    SP    1          0   SP    0    SP    3          50  ETS   –   –
1/1/18     2  2          0   SP    0    SP    2          0   SP    0    SP    –          –   –     –   –
1/1/18     3  3          0   SP    0    SP    3          0   SP    0    SP    –          –   –     –   –
1/1/18     4  4          0   SP    0    SP    4          0   SP    0    SP    –          –   –     –   –
1/1/18     5  5          0   SP    0    SP    5          0   SP    0    SP    –          –   –     –   –
1/1/18     6  6          0   SP    0    SP    6          0   SP    0    SP    –          –   –     –   –
1/1/18     7  7          0   SP    0    SP    7          0   SP    0    SP    –          –   –     –   –

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s