Network Enhancers - "Delivering Beyond Boundaries" Headline Animator

Tuesday, November 29, 2011

Cisco NX-OS and Virtual Device Contexts (VDCs)

Cisco built the next-generation data  center-class operating  system designed for maximum scalability and  application availability. The NX-OS  data center-class operating system  was built with modularity, resiliency, and  serviceability at its  foundation. NX-OS is based on the industry-proven Cisco  Storage Area  Network Operating System (SAN-OS) Software and helps ensure  continuous  availability to set the standard for mission-critical data center   environments. The self-healing and highly modular design of Cisco NX-OS  enables  for operational excellence increasing the service levels and  enabling  exceptional operational flexibility. Several advantages of  Cisco NX-OS include the  following:
  • Unified data center operating system
  • Robust and rich feature set with a variety of Cisco innovations
  • Flexibility and scalability
  • Modularity
  • Virtualization
  • Resiliency
  • IPv4 and IPv6 IP routing and multicast features
  • Comprehensive security, availability, serviceability, and  management features
 One key benefit of NX-OS is the use of VDCs.  Cisco Nexus  7000 Series switches can be segmented into virtual devices  based on customer  requirements. VDCs offer several benefits such as  fault isolation,  administration plane, separation of data traffic, and  enhanced security. This  logical separation provides the following  benefits:
  • Administrative and management separation
  • Change and failure domain isolation from other VDCs
  • Address, VLAN, VRF, and vPC isolation
Each VDC appears as a unique device and  allows for separate  Roles-Based Access Control Management (RBAC) per  VDC. This enables VDCs to be  administered by different administrators  while still maintaining a rich,  granular RBAC capability. With this  functionality, each administrator can  define virtual routing and  forwarding instance (VRF) names and VLAN IDs  independent of those used  in other VDCs safely with the knowledge that VDCs  maintain their own  unique software processes, configuration, and data-plane  forwarding  tables.

Each VDC also maintains an individual  high-availability (HA)  policy that defines the action that the system  will take when a failure occurs  within a VDC. Depending on the hardware  configuration of the system, there are  various actions that can be  performed. In a single supervisor system, the VDC  can be shut down,  restarted, or the supervisor can be reloaded. In a redundant  supervisor  configuration, the VDC can be shut down, restarted, or a supervisor   switchover can be initiated.
There are components that are shared between  VDC(s), which include the following:
A single instance of the kernel which supports all of the  processes and VDCs
  • Supervisor modules
  • Fabric modules
  • Power supplies
  • Fan trays
  • System fan trays
  • CMP
  • CoPP
  • Hardware SPAN resources
This figure shows the logical segmentation  with VDCs on the Nexus 7000. A common use case is horizontal  consolidation to reduce the quantity of physical switches at the data  center aggregation layer. In this figure, there are two physical Nexus  7000 chassis; the logical VDC layout is also shown.

VDC Configuration Examples

This section shows the required steps to  creating a VDC; once the VDC is created, you will assign resources to  the VDC. VDC(s) are always created from the default admin VDC context,  VDC context 1.

Note: The maximum number of  VDCs that can be configured per Nexus 7000 chassis is four; the default  VDC (VDC 1) and three additional VDC(s).

This example shows how to configure the VDC  core on Egypt:

egypt(config)# vdc core

   Note:  Creating VDC, one moment please ...

   egypt# show vdc
 vdc_id    vdc_name     state     mac
 1         egypt        active    00:1b:54:c2:38:c1
 2         core         active    00:1b:54:c2:38:c2

egypt# show vdc core detail

 vdc id: 2
 vdc name: core
 vdc state: active
 vdc mac address: 00:1b:54:c2:38:c2
 vdc ha policy: RESTART
 vdc dual-sup ha policy: SWITCHOVER
 vdc boot Order: 2
 vdc create time: Mon Feb 22 13:11:59 2010
 vdc reload count: 1
 vdc restart count: 0

Once the VDC is created, you now have to  assign physical interfaces to the VDC. Depending on the Ethernet modules  installed in the switch, interface allocation is supported as follows: 
  • In the 32-port 10-Gigabit Ethernet Module (N7K-M132XP-12),  interfaces can be allocated on a per port-group basis; there are eight  port-groups. For example, port-group 1 are interfaces e1, e3, e5, e7;  port-group 2 are interfaces e2, e4, e6, e8.
  • The 48-port 10/100/1000 I/O Module (N7K-M148GT-11) can be  allocated on a per-port basis.
  • The 48-port 1000BaseX I/O Module (N7K-M148GS-11) can be allocated  on a per-port basis.

In a future module, N7K-D132XP-15, the  interfaces will be allocated per 2 ports per VDC.

 Note: It is not possible to  virtualize a physical interface and associate the resulting logical  interfaces to different VDCs. A supported configuration is to virtualize  a physical interface and associate the resulting logical interfaces  with different VRFs or VLANs. By default, all physical ports belong to  the default VDC.

 This example demonstrates how to allocate  interfaces to a VDC:
egypt(config)# vdc core
 eqypt(config-vdc)# allocate interface Ethernet1/17
 egypt(config-vdc)# allocate interface Ethernet1/18

To verify the interfaces allocation, enter  the show vdc membership command as demonstrated in this  example:

egypt(config-vdc)# show vdc membership

       Ethernet1/26      Ethernet1/28      Ethernet1/30
       Ethernet1/32      Ethernet2/2       Ethernet2/4
       Ethernet2/6       Ethernet2/8       Ethernet2/26
       Ethernet2/28      Ethernet2/30      Ethernet2/32
       Ethernet3/4       Ethernet3/5       Ethernet3/6
       Ethernet3/7       Ethernet3/8       Ethernet3/9
       Ethernet3/11      Ethernet3/12      Ethernet3/13
       Ethernet3/14      Ethernet3/15      Ethernet3/16
       Ethernet3/17      Ethernet3/18      Ethernet3/19
       Ethernet3/20      Ethernet3/21      Ethernet3/22
       Ethernet3/23      Ethernet3/24      Ethernet3/25
       Ethernet3/26      Ethernet3/27      Ethernet3/28
       Ethernet3/29      Ethernet3/30      Ethernet3/31
       Ethernet3/32      Ethernet3/33      Ethernet3/34
       Ethernet3/35      Ethernet3/36      Ethernet3/39
       Ethernet3/40      Ethernet3/41      Ethernet3/42
       Ethernet3/43      Ethernet3/44      Ethernet3/45
       Ethernet3/46      Ethernet3/47      Ethernet3/48

vdc_id: 2 vdc_name: core interfaces:

       Ethernet1/17      Ethernet1/18      Ethernet1/19
       Ethernet1/20      Ethernet1/21      Ethernet1/22
       Ethernet1/23      Ethernet1/24      Ethernet1/25
       Ethernet1/27      Ethernet1/29      Ethernet1/31
       Ethernet2/17      Ethernet2/18      Ethernet2/19
       Ethernet2/20      Ethernet2/21      Ethernet2/22
       Ethernet2/23      Ethernet2/24      Ethernet2/25
       Ethernet2/27      Ethernet2/29      Ethernet2/31
       Ethernet3/1       Ethernet3/2       Ethernet3/3

In addition to interfaces, other physical  resources can be allocated to an individual VDC, including IPv4 route  memory, IPv6 route memory, port-channels, and SPAN sessions. Configuring  these values prevents a single VDC from monopolizing system resources.  This example demonstrates how to accomplish this:

egypt(config)# vdc core
 egypt(config-vdc)# limit-resource port-channel minimum 32 maximum  equal-to-min
 egypt(config-vdc)# limit-resource u4route-mem minimum 32 maximum  equal-to-min
 egypt(config-vdc)# limit-resource u6route-mem minimum 32 maximum  equal-to-min
 egypt(config-vdc)# limit-resource vlan minimum 32 maximum equal-to-min
 egypt(config-vdc)# limit-resource vrf minimum 32 maximum equal-to-min

Defining the VDC HA policy is also done  within the VDC configuration sub-mode. Use the ha-policy command to  define the HA policy for a VDC as demonstrated in this example:

egypt(config)# vdc core
 eqypt(config-vdc)# ha-policy dual-sup bringdown

The HA policy will depend based upon the  use-case or VDC role. For example, if you have dual-supervisor modules  in the Nexus 7000 chassis or if the VDC role is development/test, the  VDC HA policy may be to just shut down the VDC. If the VDC role is for  the core and aggregation use case, then the HA policy would be  switchover.

How To Log Tracebacks in Cisco IOS

In the event that Cisco IOS is malfunctioning without logging any tracebacks that may help in troublshooting the root cause, use the following undocumented Cisco internal engineering commands:
  • Router#conf t
  • Router(config)# service log backtrace
  • Router(config)# end

This hidden command will log tracebacks for every IOS action executed.

Be sure to disable this feature afterwards as it is CPU intensive and may flood your logs with Cisco IOS routine hexadecimal addresses and tracebacks messages - which may look unnecessarily alarming.
To disable it:

  • Router#conf t
  • Router(config)# no service log backtrace
  • Router(config)# end

Sunday, November 27, 2011

Bringing Cisco’s Network Virtualization (nV) Technology to Mobile Networks

The Cisco ASR 9000 edge routing system has gone mobile.
Cisco announced that it is bringing its nV (Network Virtualization) technology to mobile networks and unveiled three powerful new platforms for the ASR 9000 family:
  • ASR 901 cell site router, a high-capacity, low-power router for 2G, 3G and 4G mobile cell sites;
  • ASR 903 unified Ethernet access router, a compact Ethernet access device for business, residential, and mobile applications; and
  • ASR 9901 small edge router, a smaller version of the ASR 9000 for low-capacity deployments.
Benefits include:
  • Simplified Network Operations: The ASR 9000 system uses Cisco nV (network virtualization) technology to lower operating costs by up to 69% and capital expense costs by up to 67% (when compared with competing edge platforms).
  • Simplified IPv6 Migrations: Cisco delivers on its strategy of building IPv6 next-generation networks to simplify the design, deployment and management of services.
  • Simplified Service Management: Plug-and-play capabilities, singular point-of-service management using Cisco Prime, cost-effective configurations and ease of deployment reduce the need for costly on-site setup, support and maintenance, while providing hardware and software savings.
nV technology capabilities (unveiled June 7) now extend all the way to the access layer to help operators further optimize operations and maximize the cost benefits of virtualized infrastructures. nV technology also provides topology-, place- and capability-agnostic resource consolidation and virtualization for simplified operations, increased network capacity and accelerated IPv6 services. Operators can deploy nV technology with a simple software upgrade.

These innovative solutions deliver on the promise of the Cisco MOVE (Monetization, Optimization, Video Experience) strategic framework, which helps operators better manage, enhance and profit from the rapidly growing volume of Internet traffic and the proliferation of connected devices.

Friday, November 25, 2011

Junosphere: the first impressions

Courtesy - Ivan Pepelnjak

Junosphere, a great network-in-the-Clouds solution from Juniper. You might be familiar with Olive, the “non-existent” way of running Junos on an x86 machine (including a VM); Junosphere is the supported version of the same concept, including a real forwarding plane (it’s my understanding Olive lacks that, which makes certain protocols behave in unexpected ways).

Compared to other similar offerings (including our remote labs and Cisco’s IOS-in-a-Cloud), Junosphere has several significant advantages:

You can create your own topologies that include as many routers as you need, allowing you to recreate complex routing/migration scenarios. Although the topology file format is a bit arcane at the moment, I had no problems creating my own topologies (but you already know I’m more than a bit crazy). For saner people, there’s a tool that can take your OSPF or IS-IS database and turn it into a Junosphere topology.

The VJX1000 router (Junos-in-a-VM) supports Gigabit Ethernet interfaces and Junosphere allows you to connect them together with simple virtual bridges. Think of those bridges as cables/hubs; unlike Dynamips, Junosphere bridges have no explicit VLAN support or access/trunk links. Serial or POS interfaces are also not available.


Just keep adding topologies

You can access your virtual routers directly using Juniper’s SSL VPN solution. After starting the SSL VPN connection, you can use SSH to connect to the devices or SCP to copy files to/from them.
Obviously, once you have SSH up and running, you can test all sorts of Junos automation/SDK tricks (starting with NETCONF).

You can connect physical devices to Junosphere. A Junosphere connector (a VM running in VMware environment – be it VMware Player, Workstation or ESX) can establish a link between any Junosphere bridge and an interface (vNIC) in your workstation/hypervisor host. You can use it to connect Junosphere LAN to your physical interface or to anything else VMware player can use (including a tap interface of a Linux box ... you do know why that’s interesting, don’t you?).

Unfortunately, somehow the SSL VPN Java applet didn’t work on my Linux machine (Fedora 14 with Firefox 3.6.23) where I run all the other simulation stuff; I had to use Internet Explorer on my Windows laptop to connect to the labs.

You can load and save your topologies and configurations. This was one of the best features (from my perspective). The default configuration of the VJX routers includes an event trigger that transfers current configuration to a FTP server every time you execute a commit. Regardless of what you do, a copy of the configuration is always in a safe place and can be saved through the web-based UI and later copied (as a .tgz file) to your workstation.

You can choose the Junos release you want to run. At the moment, the set of releases you can choose from is fixed, but it does include a stable-and-supported release, an experimental release (11.4) and a few others.

You can run other VMs in the same sandbox, including Centos servers, Junos Space and a few test tools.

Do I like Junosphere? Absolutely. Are there any drawbacks? Sure, like every other system Junosphere has a few glitches, from UI that could use some improvements to minor configuration nuisances that can play havoc with the configuration saving feature ... but the major roadblock is the current pricing and go-to-market strategy.

The current list price for Junosphere is $5/router/day (Amazon’s small EC2 instance costs $2.04 per day and is charged by the minute), and you can only purchase it through regular Juniper’s sales channels (including partners). That makes perfect sense if you’re working on a customer demo, proof-of-concept or a migration scenario for a large enterprise network ... and you have direct contact with Juniper or got Junosphere access as a deal closing sweetener. But do you really think a Juniper partner would be interested in getting a $250 purchase order for a 10-day access to a 5-router Junosphere environment? How about a simple use-your-credit-card approach Cisco is using with its e-learning labs?

The per-day charging model is another pain point. With proper preparation, planning and scheduling, the current model could work for me or someone who has to get fluent with Junos really fast to support the next project. Obviously I would be throwing away more than two thirds of the allotted time because I’m too old to work on the routers for more than 8-10 hours a day, but paying $50/day (10-router topology) for something that helps you earn real money shouldn’t be a showstopper.

However, I really like the ability to run a lab for an hour or so to test the next idea that hatched in the back of my brains while I was working on something else. Paying for the whole day just to be able to test a few things might not be too expensive in absolute terms, but definitely feels like a total waste of money.

Juniper’s marketing is doing a great job trying to persuade networking engineers to embrace Junos – from Day One books to Junos as a second language and FastTrack programs. It’s too bad they’re not making the final step and getting everyone interested in kicking some Junos tires (or working really hard on mastering another platform) a simple on-demand access to live Junos environment.

Thursday, November 24, 2011

Essar Group selects Juniper's secure mobility service

Network equipment manufacturer Juniper Networks has provided a secure, multi-platform remote network access service to the India's Essar Group. Deployment of the Juniper Networks Junos Pulse Mobile Security Suite and Juniper Networks SA Series SSL VPN Appliances initially in India enables Essar to provide remote and mobile employees, customers and partners with anytime, anywhere secure SSL VPN access to corporate network resources and applications from a range of mobile devices. The Juniper Networks Junos Pulse Mobile Security Suite which integrates mobile security, secure connectivity and mobile device management supports the adoption of a 'bring your own device' policy for Essar employees. Junos Pulse Mobile Security Suite protects smartphones, tablets and other mobile devices running most mobile OS from viruses, malware, loss or theft, physical compromise, and other exploits and threats. The SA Series appliances deliver secure, mobile remote access and connectivity with performance for organisations requiring secure remote VPN access and authorisation. The Essar Group is using a network of over 1,000 mobile device retail outlets to support the roll-out of Junos Pulse Mobile Security Suite to smartphones owned by employees and other target users.

Wednesday, November 23, 2011

How To Forcibly Log a User out of a Juniper Router

user@router> show system users
5:32PM up 197 days, 4:11, 2 users, load averages: 0.41, 0.16, 0.06
shafiqul d0 - 5:09PM 23 -cli (cli)
mustafa p0 5:25PM - -cli (cli)

user@router> request system logout terminal shafiqul
user@router> request system logout terminal d0

user@router> show system users
5:33PM up 197 days, 4:11, 1 user, load averages: 0.27, 0.14, 0.06
mustafa p0 5:25PM - -cli (cli)

Tuesday, November 22, 2011

How To Enable The Use of Third-Party GBICs and SFPs on Cisco Switches

Enable the command below. Please note that the following Cisco warning message will be auto-displayed when this command is enabled.

Switch(config)#service unsupported-transceiver

Warning: When Cisco determines that a fault or defect can be traced to the use of third-party transceivers installed by a customer or reseller, then, at Cisco's discretion, Cisco may withhold support under warranty or
a Cisco support program. In the course of providing support for a Cisco networking product Cisco may require that the end user install Cisco transceivers if Cisco determines that removing third-party parts will assist Cisco in diagnosing the cause of a support issue.

Monday, November 21, 2011

Cross-Stack EtherChannel on a Catalyst 3750 Switch Configuration Example

This document provides a sample configuration for the configuration of cross-stack EtherChannel on a Cisco Catalyst 3750 Switch that runs Cisco IOS® system software Release 12.2(25)SEC. EtherChannel can be called Fast EtherChannel or Gigabit EtherChannel. This depends on the speed of the interfaces or ports that are used to form the cross-stack EtherChannel.

Background Theory

 In this document, these interfaces are bundled for the cross-stack EtherChannel:

  • Two Gigabit Ethernet interfaces of one of the Catalyst 3750 Switches
  • One Gigabit Ethernet interface of another Catalyst 3750 Switch of the same stack
  • Three Gigabit Ethernet interfaces on a Catalyst 3750 Switch of a different stack

The Cisco StackWise interconnect technology is designed with two counter-rotating paths of 16 Gb each. In order to efficiently load balance traffic, packets are allocated between these two logical counter-rotating paths, which creates the 32-Gb interconnection. There are dual paths from any port to any other port within the Catalyst 3750 stack. Therefore, maximum uptime is ensured because there is always an alternate path available if a failure occurs in either path. The Catalyst 3750 supports:

  • Cross-stack EtherChannel
  • Cross-stack UplinkFast (with subsecond failover)
  • Cross-stack equal cost routes across different switches in the stack

Link Aggregation Control Protocol (LACP) and Port Aggregation Protocol (PAgP)

EtherChannels have automatic configuration with either Port Aggregation Protocol (PAgP) or Link Aggregation Control Protocol (LACP). PAgP is a Cisco-proprietary protocol that you can only run on Cisco switches and on those switches that licensed vendors license to support PAgP. IEEE 802.3ad defines LACP. LACP allows Cisco switches to manage Ethernet channels between switches that conform to the 802.3ad protocol.

PAgP cannot be enabled on cross-stack EtherChannels while LACP is supported on cross-stack EtherChannels from Cisco IOS Software Release 12.2(25)SEC and later. Switch interfaces exchange LACP packets only with partner interfaces with the active or passive mode configuration. You can configure up to 16 ports to form a channel. Eight of the ports are in active mode, and the other eight are in standby mode. When any one of the active ports fails, a standby port becomes active.

Interfaces with the on mode configuration do not exchange PAgP or LACP packets.

These EtherChannel modes are supported on cross-stack EtherChannel:

  • active—Places an interface into an active negotiation state, in which the interface starts negotiations with other interfaces by sending LACP packets.
  • passive—Places an interface into a passive negotiation state, in which the interface responds to LACP packets that the interface receives, but does not start LACP packet negotiation. This setting minimizes the transmission of LACP packets.
  • on—Forces the interface into an EtherChannel without PAgP or LACP. With the on mode, a usable EtherChannel exists only when an interface group in the on mode has a connection to another interface group in the on mode.

EtherChannel and Switch Stacks

If a stack member that has ports participating in an EtherChannel fails or leaves the stack, the stack master removes the failed stack member switch ports from the EtherChannel. The remaining ports of the EtherChannel, if any, continue to provide connectivity.

When a switch is added to an existing stack, the new switch receives the running configuration from the stack master and updates itself with the EtherChannel-related stack configuration. The stack member also receives the operational information (the list of ports that are up and are members of a channel).

When two stacks merge that have EtherChannels configured between them, self-looped ports result. Spanning tree detects this condition and acts accordingly. Any PAgP or LACP configuration on a winning switch stack is not affected, but the PAgP or LACP configuration on the losing switch stack is lost after the stack reboots.

With PAgP, if the stack master fails or leaves the stack, a new stack master is elected. A spanning-tree reconvergence is not triggered unless there is a change in the EtherChannel bandwidth. The new stack master synchronizes the configuration of the stack members to that of the stack master. The PAgP configuration is not affected after a stack master change unless the EtherChannel has ports residing on the old stack master.

With LACP, the system-id uses the stack MAC address from the stack master, and if the stack master changes, the LACP system-id can change. If the LACP system-id changes, the entire EtherChannel will flap, and there will be an STP reconvergence. Use the stack-mac persistent timer command to control whether or not the stack MAC address changes during a master failover.


In the attached network diagram, there are two Catalyst 3750 Switch stacks, Stack A and Stack B. Stack A has three switch members, and Stack B has only one switch member. The EtherChannel is formed with two ports on Switch 1 and one port on Switch 3 of Stack A. These ports connect to the three ports in Stack B.

The network setup is used to configure the ports as trunk ports.


This document uses these configurations:

  • [#config1 Configure Cross-Stack EtherChannel Without PAgP or LACP]
  • [#config2 Configure Cross-Stack EtherChannel with LACP]

Configure Cross-Stack EtherChannel Without PAgP or LACP

This configuration example provides the cross-stack EtherChannel configuration if you turn off PAgP or LACP:
Catalyst 3750 Switch Stack A
3750switchstackA(config)#interface range gigabitethernet 1/0/4 - 5
3750switchstackA(config-if-range)#channel-group 1 mode on
 <font>!--- This command creates the port channel 1 interface. Because the mode 
 !--- is configured ON, both the PAgP and LACP are disabled on these ports. 
 !--- Issue the channel-group command first, before you enter any other commands on these 
 !--- interfaces. Any commands that you issue on these interfaces after you issue the 
 !--- channel-group command are added to the port channel interface automatically.
 !--- If you configure the port with all the commands and you issue the channel-group 
 !--- command last, the port channel interface is created but does not have any 
 !--- configurations. You must then add the other commands to the port channel interface 
 !--- manually.</font>

3750switchstackA(config-if-range)#switchport trunk encapsulation dot1q
3750switchstackA(config-if-range)#switchport mode trunk

3750switchstackA(config)#interface gigabitethernet 3/0/3
3750switchstackA(config-if)#channel-group 1 mode on
3750switchstackA(config-if)#switchport trunk encapsulation dot1q
3750switchstackA(config-if)#switchport mode trunk 
Catalyst 3750 Switch Stack B
3750switchstackB(config)#interface range gigabitethernet 1/0/2 - 4
3750switchst(config-if-range)#channel-group 1 mode on
3750switchst(config-if-range)#switchport trunk encapsulation dot1q
3750switchst(config-if-range)#switchport mode trunk 

Configure Cross-Stack EtherChannel with LACP

This example shows the configuration of the EtherChannel when you enable LACP. The minimum version of IOS that supports LACP in cross-stack Etherchannel is Cisco IOS Software Release12.2(25)SEC. This example uses the active-active mode LACP configuration:

Catalyst 3750 Switch Stack A
3750switchstackA(config)#interface range gigabitethernet 1/0/4 - 5
3750switchstackA(config-if-range)#channel-group 1 mode active
 <font>!--- This creates port channel 1 and configures it with LACP.</font>

3750switchstackA(config-if-range)#switchport trunk encapsulation dot1q
3750switchstackA(config-if-range)#switchport mode trunk

3750switchstackA(config)#interface gigabitethernet 3/0/3
3750switchstackA(config-if)#channel-group 1 mode active
3750switchstackA(config-if)#switchport trunk encapsulation dot1q
3750switchstackA(config-if)#switchport mode trunk 
Catalyst 3750 Switch Stack B
3750switchstackB(config)#interface range gigabitethernet 1/0/2 - 4
3750switchst(config-if-range)#channel-group 1 mode active
3750switchst(config-if-range)#switchport trunk encapsulation dot1q
3750switchst(config-if-range)#switchport mode trunk 

Sunday, November 20, 2011

Network Fundamentals - For Freshers

Juniper has some great tutorials and some incentives to certify on their products.To get an agnostic understanding of networking basics then click Network Fundamentals

Saturday, November 19, 2011

How to Power Off and On a Cisco GSR 12000 Linecard

Issue the following hidden Cisco IOS EXEC command:

To power off a linecard;
  • Router# test mbus power <slot #> off
To power on a linecard;
  • Router# test mbus power <slot #> on

Alternatively, use the following documented IOS EXEC commands to administratively shutdown line and fabric cards. This should also be used prior to the removal of switch fabric cards, waiting at least one minute afterwards before actual reomoval. To shutdown a linecard or fabric card, issue;
  • Router#conf t
  • Router(config)#hw-module slot <slot#> shutdown
  • Router(config)#exit
To bring up an adminstratively shutdown linecard or fabric card:
  • Router#conf t
  • Router(config)#no hw-module slot <slot#> shutdown
  • Router(config)#exit


Wednesday, November 16, 2011

How OSPF SPF Adaptive Timers are implemented in IOS and JUNOS

It became a fact that both of Cisco Systems and Juniper Networks have proved their strong market penetration and most of the operators and providers deploying the various platforms of both of them. Based on this, it became an essential for the networking engineers specially those who are working on operator’s environment to know how each vendor’s platforms are architectured, and how their OS are structured as well as how to configure it. However this will not be adequate for the design engineers who has to assure their multi-vendor network are perfectly merged and converged without any interoperability issues, so, they have to dig more and understand how each of the leading vendors are implementing the technologies and matching the RFCs.

Today we will start explaining how each of Cisco Systems and Juniper networks are implementing the OSPF SPF adaptive timers or what called SPF throttling (Cisco) or SPF hold-down (Juniper)
Before we dig into that, let’s talk a little bit about what OSPF SPF Adaptive Timers are designed to do for us, and then we’ll take a look at how each vendor is implementing the concept.

If we can recall from our OSPF background, OSPF SPF algorithm has design to run upon arrivals of LSAs. So, if each LSA triggers a full or incremental SPF run, and if they are arriving fast, SPF can begin eating up the majority of your CPU.

The challenge in large-scale networks is to quickly react to network changes while at the same time not allowing SPF calculations to dominate the route processors. This is the goal of SPF delay, also called SPF hold-down or SPF throttling.

Rather than kick off an SPF calculation every time a new LSA/LSP arrives, SPF delay forces the router to wait a bit between SPF runs. If a large number of LSA/LSPs are being flooded, a delay between SPF runs means that more LSA/LSPs are added to the link state database during the hold-down period. Efficiency is then increased because when the hold-down period expires and SPF is run, more network changes are included in a single calculation.

But this efficiency you are getting from SPF delay, it has its costs which it increase your network convergence time. So, the challenge is to set the delay interval long enough when abnormal things happen while keeping it short when the network is stable so you got a quick convergence. This leads to the concept of adaptive SPF timers.

Both Cisco and Juniper are offering adaptive SPF timers, but with different approaches. In the coming sections, we are going to explain the mechanism used by each vendor.

Adaptive SPF Timers in JUNOS

Juniper Networks uses a linear fast/slow algorithm for adaptive SPF timers. So, it introduced the SPF delay timer which is the minimum delay in the time between the detection of a topology change and when the SPF algorithm actually runs. This period is 200ms by default. The period is configurable with the spf-delay command to between 50 and 8000ms.

Secondly, they introduce a second parameter which is rapid-runs. If three (the default) SPF runs are triggered in quick succession, indicating instability in the network, the router will enter the “slow mode” and a third parameter called the hold-down timer will start. Any subsequent SPF calculation is not run until the hold-down timer expires. The routers remain in this “slow mode” until the hold-down period have passed since the last SPF run—indicating that the network has converged—and then switches back to “fast mode”, and the system reverts to the configured values for the delay and rapid-runs statements.

The default values for SPF calculations in JUNOS can be seen below:

Default SPF timers values in JUNOS
r2@r2> show ospf overview | match SPF
Full SPF runs: 280SPF delay: 0.200000sec, SPF holddown: 5 sec, SPF rapid runs: 3

Changing SPF Timers in JUNOS

The configuration stanza for JunOS shows how these settings may be changed.

1spf-options {
2 delay milliseconds;
3 holddown milliseconds;
4 rapid-runs number;

These default values can be changed with the following command:

[edit protocols ospf]
r1@r1> set spf-options delay milliseconds holddown milliseconds rapid-runs number

Now we are going to play with the timers and run the debugs, and examine the behavior. We will set the delay to 1 sec and the hold-down timer to 20 sec while keeping the rapid-runs as default.

spf-options {
delay 1000;
holddown 20000;

The log entry below shows, on lines 2,6 and 10, that the SPF run occurs every 1 second after the LSA Update. Once the SPF run has completed 3 iterations it moves into a slower mode of operation.

12:01:50.465905 OSPF full SPF refresh scheduled for topology default
12:01:50.466445 OSPF SPF scheduled for topology default in 1s
12:01:51.467761 Starting full SPF refresh for topology default
12:02:04.540150 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:02:04.541073 OSPF full SPF refresh scheduled for topology default
12:02:04.541581 OSPF SPF scheduled for topology default in 1s
12:02:05.543546 Starting full SPF refresh for topology default
12:02:12.886187 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:02:12.892787 OSPF full SPF refresh scheduled for topology default
12:02:12.893285 OSPF SPF scheduled for topology default in 1s
12:02:13.894226 Starting full SPF refresh for topology default

The next log entry shows that SPF started after 20 sec from the SPF run (at t=12:02:13). The default number of SPF calculations that can occur in succession is 3. The range that you can configure is from 1 through 5. Each SPF algorithm is run after the configured SPF delay. When the maximum number of SPF calculations occurs, the hold-down timer begins. We previously configured this to be 20 seconds. Any subsequent SPF calculation is not run until the hold-down timer expires. This is why the received LSA update on line 4 does not immediately trigger an SPF run.

12:02:20.739927 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:02:20.747717 OSPF full SPF refresh scheduled for topology default
12:02:20.756118 OSPF SPF scheduled for topology default in 13.140569s
12:02:26.990677 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:02:33.896073 Starting full SPF refresh for topology default

Next, the log shows the router once again enters the fast mode…

12:02:59.734614 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:02:59.753923 OSPF full SPF refresh scheduled for topology default
12:02:59.754409 OSPF SPF scheduled for topology default in 1s
12:03:00.755847 Starting full SPF refresh for topology default
12:03:07.494415 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:03:07.501625 OSPF full SPF refresh scheduled for topology default
12:03:07.502166 OSPF SPF scheduled for topology default in 1s
12:03:08.503663 Starting full SPF refresh for topology default
12:03:57.215931 OSPF rcvd LSUpdate -> (fxp1.100 IFL 95 area
12:03:57.223481 OSPF full SPF refresh scheduled for topology default
12:03:57.223998 OSPF SPF scheduled for topology default 1s
12:03:58.225848 Starting full SPF refresh for topology default

We can also observe from the previous log that although 3 more SPF runs have taken place, the router does not move into slow mode again. This is because there has been 50sec between the first and the last SPF run in the set of 3. If the 3 SPF runs happen within 3 x “delay value“, or in our case 3 seconds, the router will start to throttle the number of SPF runs, and start the holddown timer countdown. If the SPF runs are outwith 3 x the configured delay value, the rapid-run counter is reset to 0 and no back-off algorithms are run.

Now, shown in the next log snippet, the router will enter the slow mode and the holddown timer will start, because three SPF runs have occurred in succession.

12:04:03.364745 OSPF rcvd LSUpdate -&gt; (fxp1.100 IFL 95 area
12:04:03.378123 OSPF full SPF refresh scheduled for topology default
12:04:03.378655 OSPF SPF scheduled for topology default in 1s
12:04:04.379888 Starting full SPF refresh for topology default
12:04:15.329694 OSPF rcvd LSUpdate -&gt; (fxp1.100 IFL 95 area
12:04:15.349992 OSPF full SPF refresh scheduled for topology default
12:04:15.350510 OSPF SPF scheduled for topology default in 1s
12:04:16.352016 Starting full SPF refresh for topology default

And finally, the following log shows that SPF again started after 20 sec from the last SPF run (at t=12:04:16)

The figure below is charting the above debug which can help you in more understanding the JunOS behaviour with the SPF timers

Adaptive SPF Timers in IOS

Cisco Systems introduced an exponential backoff algorithm for the adaptive SPF timers by using three different configurable timers.
This exponential functionality limits the number of SPF computations during times of network instability by doubling the delay associated with the SPF run, up to a maximum hold delay, for the period of instability. When the period of instability ends, the delay is reset to the original value. Three timers are associated SPF exponential backoff: Start Time, Initial-Hold Time, and Max-Hold Time.
IOS internally has an internal timer called the waiting-interval which the SPF computation will be delayed till it expires. When a topology change is received for the first time, the waiting-interval will be set to the start timer which is similar to the spf-delay in JUNOS, and the SPF computation is delayed for the value set by start timer. When the SPF computation completes, a waiting-interval starts with the value of the initial-hold timer and the router will enter the “slow mode”. If there is a topology change during waiting-interval, the SPF computation will run at the expiration of the initial-hold timer. At the completion of the SPF computation the waiting-interval is set to the twice the value of initial-hold timer and then run again. So for example, if the start timer is 100ms and the initial-hold timer is 1000ms, the router delays the first SPF run by 100ms, the second by 1000ms, the third by 2000ms, the fourth by 4000ms, and so on.
The waiting-interval grows exponentially as 2^t*initial-hold until it reaches the max_hold-time value. After this, any topology change during the current waiting-interval would result in the next SPF computation will run at the expiration of the max hold time and next waiting-interval being equal to the constant max-hold timer. This ensures that exponential growth is limited. If the SPF has not run for twice the time specified by the max-hold timer, the router switches back to “fast” mode in which the start delay timer is used and the waiting-interval is reset back to the initial value.
The default values for SPF calculations in IOS can be seen below:

Default SPF timers values in IOS
R2#sh ip ospf | i SPF�
Initial SPF schedule delay 5000 msecs
Minimum hold time between two consecutive SPFs 10000 msecs
Maximum wait time between two consecutive SPFs 10000 msecs

Changing SPF Timers in IOS

These default values can be changed with the following command:
R1(config)# router ospf 100
R1(config-router)# timers throttle spf spf-start spf-hold spf-max-wait

As we did above with JunOS, will play with the SPF throttle timers and run the debugs, and examine the behavior. We will set the Start delay timer to 1 sec and the initial-hold timer to 5 sec and the max-hold timer to 50 sec.
The log entry below shows, on lines 2, that the SPF run at t= 21:30 which is 1 second after the LSA Update, and the next wait_interval set to the initial-hold time which is 5 sec as shown in line 5.

12:21:29: OSPF: Detect change in LSA type 1, LSID, from area 0
12:21:30: OSPF: Begin SPF at 54881.208ms, process time 13316ms
12:21:30: spf_time 15:14:43.672, wait_interval 1000ms
12:21:30: OSPF: End SPF at 54889.600ms, Total elapsed time 84ms
12:21:30: Schedule time 15:14:44.672, Next wait_interval 5000ms

The next log entry shows that the waiting_interval is getting doubled after each SPF run. Starting with a waiting_interval equal to 5 sec which is the initial-hold timer as shown on line 3, the next waiting_interval on line 8 is set to 10 sec then to 20 sec on line 14 and 40 sec on line 22.
While the router is in the slow mode no SPF will run until the wait_interval elapses no matter how many topology changes have been detected. This is why the received LSA update on lines 13 and 14 and also on lines 20,21,22 and 23 does not immediately trigger an SPF run.

12:21:32: OSPF: Detect change in LSA type 1, LSID, from area 0
12:21:35: OSPF: Begin SPF at 54889.600ms, process time 13424ms
12:21:35: spf_time 15:14:44.672, wait_interval 5000ms
12:21:35: OSPF: End SPF at 54889.672ms, Total elapsed time 72ms
12:21:35: Schedule time 15:14:49.672, Next wait_interval 10000ms
12:21:41: OSPF: Detect change in LSA type 1, LSID, from area 0
12:21:45: OSPF: Begin SPF at 54899.672ms, process time 13516ms
12:21:45: spf_time 15:14:49.672, wait_interval 10000ms
12:21:45: OSPF: End SPF at 54899.720ms, Total elapsed time 48ms
12:21:45: Schedule time 15:14:59.720, Next wait_interval 20000ms
12:21:58: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:03: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:05: OSPF: Begin SPF at 54919.720ms, process time 13580ms
12:21:05: spf_time 15:14:59.720, wait_interval 20000ms
12:22:05: OSPF: End SPF at 54919.776ms, Total elapsed time 56ms
12:22:05: Schedule time 15:15:19.776, Next wait_interval 40000ms
12:22:22: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:27: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:32: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:39: OSPF: Detect change in LSA type 1, LSID, from area 0
12:22:45: OSPF: Begin SPF at 54959.776ms, process time 13684ms
12:22:45: spf_time 15:15:19.776, wait_interval 40000ms
12:22:46: OSPF: End SPF at 54959.884ms, Total elapsed time 108ms
12:22:46: Schedule time 15:15:59.884, Next wait_interval 50000ms

The next log entry shows that the waiting_interval is reached the max-hold time (50 sec) and upcoming waiting_interval being equal to the constant max-hold timer as on lines 3 and 8 .

12:23:17: OSPF: Detect change in LSA type 1, LSID, from area 0
12:23:36: OSPF: Begin SPF at 55009.884ms, process time 13808ms
12:23:36: spf_time 15:15:59.884, wait_interval 50000ms
12:23:36: OSPF: End SPF at 55009.928ms, Total elapsed time 44ms
12:23:36: Schedule time 15:16:49.928, Next wait_interval 50000ms
12:24:26: OSPF: Detect change in LSA type 1, LSID, from area 0
12:24:26: OSPF: Begin SPF at 55059.928ms, process time 13872ms
12:24:26: spf_time 15:16:49.928, wait_interval 50000ms
12:24:26: OSPF: End SPF at 55059.968ms, Total elapsed time 40ms
12:24:26: Schedule time 15:17:39.968, Next wait_interval 50000ms

We can also observe from the previous log that although the LSA on line 6 arrived 60 sec after last SPF run has taken place which is more than the waiting_interval , the router does not move into fast mode again. This is because that the condition is that to divert back to the fast mode the SPF should not run for twice the time specified by the max-hold timer.
Now, shown in the next log snippet, the router will enter the slow mode and the holddown timer will start, because the SPF has not run for 100 sec which is twice the time specified by the maximum delay period.

12:26:22: OSPF: Detect change in LSA type 1, LSID, from area 0
12:26:23: OSPF: Begin SPF at 55177.420ms, process time 13932ms
12:26:23: spf_time 15:19:36.420, wait_interval 1000ms
12:26:23: OSPF: End SPF at 55177.488ms, Total elapsed time 68ms
12:26:23: Schedule time 15:19:37.488, Next wait_interval 5000ms
12:26:29: OSPF: Detect change in LSA type 1, LSID, from area 0
12:26:28: OSPF: Begin SPF at 55177.488ms, process time 14016ms
12:26:28: spf_time 15:19:37.488, wait_interval 5000ms
12:26:28: OSPF: End SPF at 55184.660ms, Total elapsed time 108ms
12:26:28: Schedule time 15:19:42.660, Next wait_interval 10000ms

For more clarity, I have reflected the debugs on the following figure, so you can use both the debugs and the figure to examine the behavior

Tuesday, November 15, 2011

Riverbed 101

Bob Gilbert gives and introduction to Riverbed's 5 product lines.

Best Practices for cleaning the Juniper router

1) request system zeroize 

Erase all data, including configuration and log files.  "show system commit" data was not deleted.  On the other hand, all the old configuration files(juniper.conf.[1-3].gz) were deleted and router will came up with the very default configuration.
To deleted "show system commit" information the following technique should be used.

root> start shell sh
# echo "" > /var/db/commits
# exit
3) rm -rf /var/home/*

Sunday, November 13, 2011

TelePresence turns 5! What's next for Enterprise Video

Didier Moretti, VP/GM of Cisco's Media Experience and Analytics business, discusses what's new in enterprise video with the recent Cisco TelePresence announcement on October 26.

All About Cisco Catalyst 6500 Supervisor 2T

Introducing the Cisco Catalyst 6500 Supervisor 2T

Supervisor 2T Deep Dive

Advancing the Catalyst 6500 with the Supervisor 2T

CCDE Certification Written and Practical Exams Revised to Version 2.0

The last day to test using the Cisco CCDE Practical Exam v1.0 is October 21, 2011. Candidates who are intending to take practical exams after October 22, 2011 should prepare using the CCDE Practical Exam Topics v2.0 as well as the CCDE Practical v2.0 Exam Preparation Checklist and the CCDE Written Exam Topics v2.0. The CCDE Practical Exam v2.0 is scheduled for its first administration at the end of March 2012.

The revised CCDE written and practical exams will cover advanced network infrastructure design principles and fundamentals, including analysis of design requirements, development of network design and related implementation plan, and validation and optimization of network design. Focus will be placed on the Layer 3 control plane, with additional emphasis on the Layer 2 control plane, network virtualization, and other design considerations such as Quality of Service (QoS), network management, security, and access technologies (such as wireless, optical, and SAN fabric deployment).

The revised CCDE written and practical exams are scheduled for release in all worldwide testing centers at the end of March 2012 and will replace Version 1.0 exams at that time.

Friday, November 11, 2011

Reasons for the Junos MX Series MPC Crash

View Bulletin PSN-2011-08-327

Title: MX Series MPC crash in Ktree::createFourWayNode after BGP UPDATE

Products Affected:  This issue can affect any MX Series router with port concentrators based on the Trio chipset -- such as the MPC or embedded into the MX80 -- with active protocol-based route prefix additions/deletions occurring.

Platforms Affected

JUNOS 11.x
JUNOS 10.x
SIRT Security Advisory
SIRT Security Notice

Revision Number             1

Issue Date:         2011-08-08

 PSN Issue :

MPCs (Modular Port Concentrators) installed in an MX Series router may crash upon receipt of very specific and unlikely route prefix install/delete actions, such as a BGP routing update. The set of route prefix updates is non-deterministic and exceedingly unlikely to occur. Junos versions affected include 10.0, 10.1, 10.2, 10.3, 10.4 prior to 10.4R6, and 11.1 prior to 11.1R4. The trigger for the MPC crash was determined to be a valid BGP UPDATE received from a registered network service provider, although this one UPDATE was determined to not be solely responsible for the crashes. A complex sequence of preconditions is required to trigger this crash. Both IPv4 and IPv6 routing prefix updates can trigger this MPC crash.

There is no indication that this issue was triggered maliciously. Given the complexity of conditions required to trigger this issue, the probability of exploiting this defect is extremely low.

The assertions (crash) all occurred in the code used to store routing information, called Ktree, on the MPC. Due to the order and mix of adds and deletes to the tree, certain combinations of address adds and deletes can corrupt the data structures within the MPC, which in turn can cause this line card crash. The MPC recovers and returns to service quickly, and without operator intervention.

This issue only affects MX Series routers with port concentrators based on the Trio chipset, such as the MPC or embedded into the MX80. No other product or platform is vulnerable to this issue.


The Ktree code has been updated and enhanced to ensure that combinations and permutations of routing updates will not corrupt the state of the line card. Extensive testing has been performed to validate an exceedingly large combination and permutation of route prefix additions and deletions.

All Junos OS software releases built on or after 2011-08-03 have fixed this specific issue. Releases containing the fix specifically include: 10.0S18, 10.4R6, 11.1R4, 11.2R1, and all subsequent releases (i.e. all releases built after 11.2R1).
This issue is being tracked as PR 610864. While this PR may not be viewable by customers, it can be used as a reference when discussing the issue with JTAC.

KB16765 - "In which releases are vulnerabilities fixed?" describes which release vulnerabilities are fixed as per our End of Engineering and End of Life support policies.


No known workaround exists for this issue.

Thursday, November 10, 2011

Juniper adds OpenFlow to its routers, switches

Juniper Networks this week said it is making the source code of its OpenFlow application accessible to developers of applications for its Junos networking operating system software.

OpenFlow is an interface that enables software-defined networks (SDN), in which multivendor switches and routers are programmable through software. OpenFlow provides a layer of abstraction from the physical network to the control element, allowing that network to be configured or manipulated through software, which then opens it up to further customization, proponents says.

It’s been pitched as a key element to enabling flexibility required in large data centers and in cloud computing environments. Other observers, though, are hard pressed to find applicability for OpenFlow and SDNs in the enterprise.

Juniper rival Cisco is currently considering how to respond to OpenFlow and its momentum. Cisco says it is adding OpenFlow to its Nexus switches; but it’s believed that OpenFlow and SDNs may further commoditize network hardware, which would squeeze Cisco’s and other vendors’ profits on equipment sales and diminish the proprietary value of software.

Juniper is supporting OpenFlow version 1.0 in its SDK. Adding OpenFlow to the Junos SDK is intended make Juniper’s routers and switches programmable through software. The Junos SDK allows developers to build custom applications on top of Junos to expand or create additional functionality on Juniper switches and routers.

The OpenFlow application works with Junos to change the control plane of those switches and routers to create more dynamic network programmability, Juniper says. This will simplify control of network devices and allow more rapid development of applications and capabilities across the network infrastructure, the company says.

Juniper demonstrated the OpenFlow protocol running on its MX 3D edge routers at the Carrier Ethernet World Congress and presented at last week’s Open Networking Summit at Stanford University. Juniper is a founding member of the Open Networking Foundation, an organization of users, vendors and service providers evangelizing OpenFlow and SDNs, and plans to participate in this week’s Applied OpenFlow Symposium in San Jose.

Wednesday, November 9, 2011

Brocade takes on Cisco in the campus

Brocade this week has unveiled switches for the enterprise campus designed to help users to affordably scale their networks.

The Ethernet switches are intended to attract customers looking for increased price/performance at lower cost. They are aimed directly at Cisco's dominance in enterprise switching, with Brocade claiming that they offer five times the bandwidth at one-third the cost of comparable Cisco products.

The first of these new products is the Brocade ICX 6610, an Ethernet access switch that features a stacking bandwidth of 320Gbps -- five times that of Cisco and other competitors, Brocade says -- and 8x10 Gigabit Ethernet uplink ports.

The 6610 is available in 24- or 48-port RJ45 configurations, and 24-port SFP. There's a PoE+ option available for the switches, to power VoIP phones or IP video cameras. The uplinks can be either 8x Gigabit Ethernet SFP, or 8x10G Ethernet SFP+, with an additional license.

The dedicated stacking ports are 4x40G Ethernet QSFP.

The 6610s support full Layer 3 capabilities -- IPv4, IPv6 and multicast -- sFlow for network traffic accounting, 12,000 ACLs, 16,000 routes, 32,000 MAC table entries and 8,000 multicast groups. The line also features hitless stacking failover, redundant stacking links and redundant, removable, load-sharing power supplies and fans.

For the aggregation and core areas of the network, Brocade also rolled out new blades and performance and scalability enhancements for its FastIron SX series of chassis-based switches. That line also includes the new high-density 8x10G Ethernet blades, which enables the FastIron SX to scale up to 128 ports of 10G Ethernet, up to four times the port density of the previous generation.
Other enhancements to the line include hitless failover to provide high levels of availability, Multi-Chassis Trunking (MCT) for active-active resiliency that's designed to overcome the traditional active-passive redundancy of Spanning Tree, plus MACsec encryption and Energy Efficient Ethernet-ready hardware for investment protection.

MCT lets two FSX chassis operate as a single fully redundant active/active logical switch with no Spanning Tree overhead, Brocade says. It is a Brocade-developed technology and is not based on the IEEE's Shortest Path Bridging or the IETF's TRILL specifications.

Hitless failover and in-service software upgrades mean there's no service interruption with outages or new software image loading. The FSX also features hot module and line card replacement.
Brocade also claims the FastIron SX chassis switches are 40% less than the Cisco Catalyst 6500 and Nexus 7000 switches per 10G Ethernet port.

The Brocade ICX 6610 will be available this month starting at $5,595. The new FastIron SX Series modules will also be available this month, with a starting price of $4,495. Brocade says the switches are also eligible for the company's network subscription purchasing service.

MCT will be available in early 2012 for the FastIron SX Series. Brocade has also reduced the price of its FCX Series stackable switches by an average of 20%.

My Blog List

Networking Domain Jobs