This topic covers Cisco Fabric Extender in detail that ships today as the Nexus 2000 series hardware.
The Modular Switch
The concept is easy to understand referencing existing knowledge. Everybody is familiar with the distributed switch architecture commonly called a modular switch:
Consider the typical components:
- Supervisor module/s are responsible for the control and management plane functions.
- Linecards or I/O modules, offers physical port termination taking care of the forwarding plane.
- Connections between the supervisors and linecards to transport frames e.g., fabric cards, or backplane circuitry.
- Encapsulating mechanism to identify frames that travel between the different components.
- Control protocol used to manage the linecards e.g., MTS on the catalyst 6500.
Most linecards nowadays have dedicated ASICs to make local hardware forwarding decisions, e.g., Catalyst 6500 DFCs (Distributed Forwarding Cards).
Cisco took the concept of removing the linecards from the modular switch and boxing them with standalone enclosures. These linecards could then be installed in different locations connected back to the supervisors modules using standard Ethernet. These remote linecards are called Fabric Extenders (a.k.a. FEXs).
Three really big benefits are gained by doing this.
- The reduction of the number of management devices in a given network segment since these remote linecards are still managed by the supervisor modules.
- The STP footprint is reduced since STP is unaware of the co-location in different cabinets.
- Another benefit is the cabling reduction to a distribution switches. I’ll cover this in a later post. Really awesome for migrations.
Lets take a deeper look at how this is done.
A fabric extender, the term marketed by Cisco, is basically a port extender as it is referenced in the developing
IEEE 802.1Qbh (Bridge Port Extension). The 802.1Qbh standard is specific to control protocol used between the controlling bridge and the port extender, as it is referred in the draft.
Supporting standards, also currently being developed like
IEEE 802.1Qbg (Edge Virtual Bridging) and
IEEE 802.1Qbc (Provider Bridging), are required to make this work.
The same elements of a modular is still present, although it is called different names.
The Fabric Extender Architecture
The components involved:
- Controlling Bridge (Parent Switch) to provide the control and management plane functions. This could be one or two Nexus 5000 or Nexus 7000 switches.
- Port Extender which provides the physical port termination. This would be the Nexus 2000 series.
- Connecting the FEX to the controlling bridge is done using SFPs over Ethernet fiber.
- Encapsulation mechanism to transport frames from the FEX to the controlling bridge.
- Control protocols to manage/monitor the FEXs
Cisco calls the encapsulation mechanism used on between the FEX and controlling bridge VN-Tag (previously VN-link). Controlling bridge is IEEE terminology, whereas parent switch is Cisco terminology. The IEEE 802.1Qbh working group was initiated by Cisco in a hope to standardize their VN-Tag technology. VN-Tag provides the capability to differentiate traffic between different host interfaces traversing the fabric uplinks. The VN-Tag header consists of the following fields:
More details on the Cisco suggested VN-Tag proposal can found
here.
Spanning-tree is not extended beyond the parent switch. Obviously loops still need to be avoided, which would require alternate loop avoidance methods. Additionally the FEX need to be managed somehow from the parent switch.
Cisco deploys the following control protocols to address these concerns:
- SDP (Satellite Discovery Protocol) is used to discover a FEX when the fabric ports of the FEX is connected to the Nexus 5000/7000.
- SMP (Satellite Management Protocol) is used to control the FEX and provide loop avoidance.
- MTS (Message and Transmission Service) facilitates the inter-process communications such as event notification, synchronization, and message persistence between system services, system components and the controlling bridge.
The Fabric Extender Forwarding
A FEX or a Nexus 2000 operate as a remote linecard, but does not support local switching, all forwarding is performed on the parent switch. This is in contrast to most modular switches like the DFCs on Catalyst 6500. One of the reasons this was done was re-usability. By offloading the forwarding and intelligent decisions, the idea Cisco had in mind is that by upgrading the parent switch, the FEX being deployed in larger numbers can remain.
Where the DFC on a Catalyst 6500 lives on the line card, the equivalent processing lives on the parent switch, be it the Nexus 7000/5000. Thus upgrading the parent switch upgrades that FEX capability since all it does is encapsulate traffic for identification.
In large deployments where the cost of hundreds of FEXs out ways the cost of the Nexus 5000s used, this makes perfect sense. In very small deployments, this reason becomes arguable.
The Fabric Extender Management
It was briefly mentioned before that a parent switch and all its FEXs are treated as a single management device. This is accomplished by a small satellite image running on the FEX. This image is a smaller compatible version of the parent NX-OS image pushed from the parent switch. The parent switch is responsible for this and happens with no user involvement.
Same applies to when the parent switch is upgraded, every attached FEX is upgraded during this time too.
The Fabric Extender Operation
Lets take a deep look at the backend operations. There are various interfaces involved:
- HIF (Host Interface): Are the physical user/host interfaces on the FEX. These interfaces receive normal Ethernet traffic before it is encapsulated with the VN-Tag header. Each HIF interface is assigned a unique VN-Tag ID that is used with the encapsulation.
- NIF (Network Interface): Physical uplink interfaces on the FEX. These interfaces can only connect back to the parent switch and carries only VN-Tagged traffic.
- LIF (Logical Interface): Is the logical interface representation of the HIF and its configuration on the parent switch. Forwarding decisions are based on the LIF.
- VIF (Virtual Interface): Is a logical interface on the FEX. The parent switch assigns/pushes the config of a LIF to the VIF of an associated FEX which is mapped to a physical HIF. This is why replacing a FEX becomes trivial in that the broken FEX is unplugged and the replacement is plugged in.
The diagram below should make this a bit more clear:
The Fabric Extender Configuration
The concept of a line card is replicated in configuration. When a linecard is inserted in a modular switch, it is inserted in one of the available slots. The ports on this linecard is prepended by the slot number the linecard was inserted in.
I.e., Port 1 and 2 on a linecard in slot 7 port will presented as 7/1, 7/2, etc.
When a linecard is not physically inserted any number indicating the slot can be used provided it is unique. The available slot numbers with the FEX configuration are 100-199.
Thus to enable a FEX two things are required:
- Change the physical port type where the FEX uplink is attached.
- Configure the “linecard slot” called the FEX number.
Below a FEX is attached to Ethernet 1/1 on a Nexus 5010:
3 | switchport mode fex-fabric |
The discovery protocol (SDP) would be initiated, discover the attached FEX, update the software on the FEX if needed and be reloaded automatically if the software was updated by the parent switch. Once the FEX comes online it would report the local HIFs back to the parent switch. The ports 1/1-1/24 are the HIFs represented in the LIF notation
Eth {fex/mod/port}.
02 | FEX: 109 Description: FEX0109 state: Online |
03 | FEX version: 5.0(3)N2(1) [Switch version: 5.0(3)N2(1)] |
04 | FEX Interim version: 5.0(3)N2(1) |
05 | Switch Interim version: 5.0(3)N2(1) |
06 | Extender Model: N2K-C2224TP-1GE, Extender Serial: xxxxxxxx |
08 | Card Id: 132, Mac Addr: xx:xx:xx:xx:xx:xx, Num Macs: 64 |
09 | Module Sw Gen: 21 [Switch Sw Gen: 21] |
11 | pinning-mode: static Max-links: 1 |
12 | Fabric port for control traffic: Eth1/9 |
13 | Fabric interface state: |
14 | Eth1/9 - Interface Up. State: Active |
15 | Fex Port State Fabric Port |
16 | Eth109/1/1 Down Eth1/9 |
17 | Eth109/1/2 Down Eth1/9 |
18 | Eth109/1/3 Down Eth1/9 |
19 | Eth109/1/4 Down Eth1/9 |
20 | Eth109/1/5 Down Eth1/9 |
21 | Eth109/1/6 Down Eth1/9 |
22 | Eth109/1/7 Down Eth1/9 |
23 | Eth109/1/8 Down Eth1/9 |
24 | Eth109/1/9 Down Eth1/9 |
25 | Eth109/1/10 Down Eth1/9 |
26 | Eth109/1/11 Down Eth1/9 |
27 | Eth109/1/12 Down Eth1/9 |
28 | Eth109/1/13 Down Eth1/9 |
29 | Eth109/1/14 Down Eth1/9 |
30 | Eth109/1/15 Down Eth1/9 |
31 | Eth109/1/16 Down Eth1/9 |
32 | Eth109/1/17 Down Eth1/9 |
33 | Eth109/1/18 Down Eth1/9 |
34 | Eth109/1/19 Down Eth1/9 |
35 | Eth109/1/20 Down Eth1/9 |
36 | Eth109/1/21 Down Eth1/9 |
37 | Eth109/1/22 Down Eth1/9 |
38 | Eth109/1/23 Down Eth1/9 |
39 | Eth109/1/24 Down Eth1/9 |
The Fabric Extender Caveats
Does the interface config reside on a FEX?
No, it resides on the parent switch. This means the configuration tied to the FEX number can be moved easily to a different port/FEX by changing the port-to-FEX association.
Can FEX be redundantly connected?
Yes, by using a port-channel to the same parent switch, or by using VPC (Virtual Port-Channeling) to two different parent switched.
How can the HIF interfaces of a FEX be easily identified on the parent switch?
Any interface starting with a number between 100-199 would be a HIF
Can you log on and configure a FEX separately?
Same a normal linecard. You can log onto a FEX by using the command “attach fex {100-199}” command. This could be used to issue show commands, but no configuration can be done directly on the FEX.
No comments:
Post a Comment