Storage 101 - Part 6

Storage 101 - Part 1

Storage 101 - Part 2

Storage 101 - Part 3

Storage 101 - Part 4

Storage 101 - Part 5

This article concludes this series by discussing point to point and ring topologies for Fibre Channel.

Introduction

In the previous article in this series, I discussed the Fibre Channel switched fabric topology. As I mentioned in that article, switched fabric is by far the most common Fibre Channel topology used in Storage Area networks. Even so, there are two additional Fibre Channel topologies that I wanted to show you.

The Point to Point Topology

Point to point is by far the simplest Fibre Channel topology. It is so simple in fact, that its simplicity renders it unsuitable for use in SAN environments.

Point to point topology can best be thought of as a direct connection between two Fiber Channel nodes. In this topologies the N_Port from one node is connected to the N_Port of another. The cable that is used to make the connection performs a cross over so that the traffic being transmitted from the first node gets sent to the receive port on the second node. Likewise, the second node’s transmit port sends traffic to the first node’s receive port. The process is very similar to connecting two Ethernet devices together without a switch by using a crossover cable.

As you can see, a point to point topology is extremely simple in that there are no switches used. The down side to using point to point connectivity is that this type of topology severely limits your options because the design can’t be scaled to service more complex storage requirements without switching to a different topology.

Arbitrated Loop Topology

The other type of topology that is sometimes used with Fibre Channel is known as an arbitrated loop. This type of topology is also sometimes referred to simply as Loop or as FC-AL.

The Arbitrated Loop topology has been historically used as a low cost alternative to the switched fabric topology that I discussed in the previous article. Switched fabric topologies can be expensive to implement because of their reliance on Fibre Channel switches. In contrast, the arbitrated loop topology does not use switches.

It is worth noting that today Fibre Channel switches are less expensive than they once were, which makes the use of switched fabric more practical than it was a few years ago. The reason why I mention this is because in a SAN environment you really should be using a switched fabric. A switched fabric provides the highest level of flexibility and the highest resiliency when a component failure occurs. Even so, an arbitrated loop can be a valid option in smaller organizations with limited budgets, so I wanted to at least talk about it.

Just as the fabric topology can be implemented (cabled) in several different ways, so too can the ring topology. Although the phrase ring topology implies that devices will be cabled together in a ring, this concept does not always hold true.

The first way in which a ring topology can be cabled is in a ring. In doing so, the Fibre Channel devices are arranged in a circle (at least from a cabling standpoint anyway) and each device in the circle has a physical connection to the device to its left and to the device to its right.

This type of design has one major disadvantage (aside from the limitations that are shared by all forms of the ring topology). That disadvantage is that the cabling can become a single point of failure for the ring. If a cable is damaged or unplugged then the entire ring ceases to function. This occurs because there is no direct point to point connection between devices. If one device wants to communicate with another device then the transmission must be passed from device to device until it reaches its intended destination.

Another way in which the ring topology can be implemented is through the use of a centralized Fibre Channel hub. From a cabling prospective, this topology is not a ring at all, but rather a star topology. Even so, the topology is still defined as a ring topology because it makes use of NL_Ports (node loop ports) rather than the N_Ports that are used with a switched star topology.

So why would an organization use a Fibre Channel hub as a part of a ring topology? It’s because using a hub prevents the ring’s cabling from becoming a single point of failure. If a cable is broken or unplugged it will cause the associated device to become inaccessible, but the hub ensures that the affected device is bypassed and that the rest of the ring can continue to function. This would not be the case without the Fibre Channel hub. If the same failure were to occur on a Fibre Channel ring that was not based around a hub then the entire ring would cease to function.

The other advantage to using a Fibre Channel hub is that the hub can increase the ring’s scalability. I will talk more about scalability in a moment, but for right now I’m sure that some of you are curious as to the cost of a Fibre Channel hub. The price varies among vendors and is based on the number of ports on the hub. However, prices for Fibre Channel hubs start at less than a hundred dollars, and higher end hubs typically cost less than a thousand dollars. By way of comparison, some low end Fibre Channel switches cost less than five hundred dollars, but most cost several thousand.

Public Loops

Occasionally Fibre Channel loops use a design known as a public loop. A public loop is a hub based Fibre Channel loop that is also tied into a switched fabric. In this type of topology, the devices within the ring connect to the hub using NL_Ports. However, the hub itself is also equipped with a single FL_Port that connects the loop to a single port on a Fibre Channel switch. Needless to say, this is a low performance design since the switch port’s bandwidth must be shared by the entire ring.

Scalability

Arbitrated loops are limited to a total of 127 ports. When a hub is not used, each device in the loop requires two ports, because it must link to the device to its left and the device to its right. When a hub is used then each device only requires a single port, so the loop could theoretically accommodate up to 127 devices (although hardware limitations often limit the actual number of devices that can be used).

On the lower limit, an arbitrated loop can have as few as two devices. Although a loop consisting of two devices is cabled similarly to a point to point topology, it is a true ring topology because unlike a point to point topology, NL_Ports are being used.

One last thing that you need to know about the ring topology is that it is a serial architecture and the ring’s bandwidth is shared among all of the devices on the ring. In other words, only once device can transmit at a time. This is a stark contrast to a switched fabric in which multiple communications can occur simultaneously.

Conclusion

As you can see, Fibre Channel uses some unique hardware, but the technology does share some similarities with Ethernet, at least as far as networking topologies are concerned.

 

Storage 101 - Part 5

 

Storage 101 - Part 1

Storage 101 - Part 2

Storage 101 - Part 3

Storage 101 - Part 4

 

This article continues the discussion of Fibre Channel SANs by discussing Fibre Channel topologies and ports.

In the previous article in this series, I spent some time talking about Fibre Channel switches and how they work together to form a switched fabric. Now I want to turn my attention to the individual switch ports.

One of the big ways that Fibre Channel differs from Ethernet is that not all switch ports are created equally. In an Ethernet environment, all of the Ethernet ports are more or less identical. Sure, some Ethernet ports support a higher throughput than others and you might occasionally encounter an uplink port that is designed to route traffic between switches, but in this day and age most Ethernet ports are auto sensing. This means that Ethernet ports are more or less universal, plug and play network ports.

The same cannot be said for Fibre Channel. There are a wide variety of Fibre Channel switch port types. The most common type of Fibre Channel switch port that you will encounter is known as an N_port, which is also known as a Node Port. An N_port is simply a basic switch port that can exist on a host or on a storage device.

Node ports exist on hosts and on storage devices, but in a SAN environment traffic from a host almost always passes through a switch before it gets to a storage device. Fibre Channel switches do not use N_ports. If you need to connect an N_port device to a switch then the connection is made through an F_port (known as a Fabric Port).

Another common port type that you need to know about is an E_port. An E_port is an Expansion Port. Expansion ports are used to connect Fibre Channel switches together, much in the same way that Ethernet switches can be connected to one another. In Fibre Channel an E_port link between two switches is known as an ISL or Inter switch link.

One last port type that I want to quickly mention before I move on is a D_port. A D_port is a diagnostic port. As the name implies, this port is used solely for switch troubleshooting.

It is worth noting that these are the most basic types of ports that are used in Fibre Channel SANs. There are about half a dozen more standard port types that you might occasionally encounter and some of the vendors also define their own proprietary port types. For example, some Brocade Fibre Channel switches offer a U_port, or Universal Port.

Right about now I’m sure that many of you must be wondering why there are so many additional port types and why I haven’t addressed each one individually. In a Fibre Channel SAN the types of ports that are used depend largely on the Fibre Channel topology that is being used.

Fibre Channel topologies loosely mimic the topologies that are used by other types of networks. The most common topology is the switched fabric topology. The switched fabric topology uses the same basic layout as most Ethernet networks. Ethernet networks use a lot of different names for this topology. It is often referred to a hub and spoke topology or a star topology.

Regardless of the name, the basic idea behind this topology is that each device is connected to a central switch. In the case of Ethernet, each network node is connected to an Ethernet switch. In the case of Fibre Channel, network nodes and storage devices are all connected to a switch. The switch port types that I discussed earlier are all found in switched fabric topologies.

It is important to understand that describing a switched fabric topology as a star or as a hub and spoke topology is a bit of an over simplification. When you describe a network topology as a star or as a hub and spoke, the assumption is that there is a central switch and that each device on the network ties into that switch.

Although this type of design is a perfectly viable option for Fibre Channel networks, most Fibre Channel networks use multiple switches that are joined together through E_ports. Often times a single switch lacks a sufficient number of ports to accommodate all of the network devices, so multiple switches must be joined together in order to provide enough ports for the devices.

Another reason why the star or hub and spoke model is a bit of an over simplification is because in a SAN environment it is important to provide full redundancy. That way, a switch failure cannot bring down the entire SAN. In some ways a redundant switched fabric could still be thought of as a star or a hub and spoke topology, it’s just that the redundancy requirement creates multiple “parallel stars”.

To give you a more concrete idea of what I am talking about, check out the diagrams below. Figure A shows a basic switched fabric that completely adheres to the star or hub and spoke topology. In this figure you can see that hosts and storage devices are all linked to a central Fibre Channel switch.

Figure A: This is the most basic example of a switched fabric.

In contrast, the Fibre Channel network shown in Figure B uses the same basic topology, but with redundancy. The biggest difference between the two diagrams is that in Figure B, each host and each storage device is equipped with two Host Bus Adapters. There is also a second switch present. Each host and storage device maintains two separate Fibre Channel connections – one connection to each switch. This prevents there from being any single points of failure on the SAN. If a switch were to fail then storage traffic is simply routed through the remaining switch. Likewise, this design is also immune to the failure of a host bus adapter or a Fibre Channel cable, because the network is fully redundant.

Figure B: This is a switched fabric with redundancy.

As you look at the diagram above, you will notice that there is no connection between the two Fibre Channel switches. If this were a real Fibre Channel network then the switches would in all likelihood be equipped with expansion ports (E_ports). Even so, using them is unnecessary in this situation. Remember, our goal is to provide redundancy, not just to increase the number of available F_ports.

In a larger SAN there would typically be many more switches than what is shown in the diagram. That’s because you would typically need to use expansion ports to provide greater capacity, but also continue to provide redundancy. To see how this works check out Figure C.

Figure C: This is a redundant multi-switch fabric.

In the figure above I have omitted all but one node and one storage device for the sake of clarity. Even so, the diagram uses the same basic design as the diagram shown in Figure B. Each node and each storage device uses multiple host bus adapters to connect to two redundant switches.

The thing that makes this diagram different is that we have made use of the switch’s expansion ports and essentially formed two parallel, redundant networks. Each side of the network uses a series of switches that are linked together through their expansion ports to provide the needed capacity, but the two networks are not joined to one another at the switch level.

Conclusion

Although switched fabric is the most common Fibre Channel topology, it is not the only topology that can be used in a SAN. In the next article in this series, I will show you the point to point topology and the ring topology. As I do, I will discuss the unique port requirements for Fibre Channel rings.

 

Enterprise Storage Heats up Again !

Once you spend any time inside the storage marketplace, you'll come to appreciate there are many segments and subsegments.
The need to store information is ubiquitous -- the approaches are not.

Sit down and attempt to segment the storage marketplace, and you'll quickly end up with a fairly complicated model.
One familiar category is what is imprecisely called "enterprise storage" or sometimes "tier 1" -- the storage that supports the enterprise's most critical applications.

These are no-compromise environments that demand the best in predictability: performance, availability, recoverability and so on.

A subsegment of this market appears to be heating up very quickly: the entry-level enterprise-class storage array.

EMC has its very successful VMAX 10K. Hitachi has recently offered the HUS VM. And HP has invested heavily in enhancing 3PAR in this segment. IBM, NetApp, et. al. really don't play in this game, despite what their aspirational marketing might say.

So -- one has to ask -- why is this particular subsegment of a rather familiar storage landscape becoming popular all at once?

And -- of course -- what is EMC doing about it?

What Makes Enterprise Storage Different

Since no one governs the use of the term, you'll see it applied to all sorts of storage products. I’m a bit more demanding: just because someone uses a storage product in an enterprise doesn't make itenterprise storage in my book.

Think consolidation of many demanding enterprise workloads, with an iron-clad requirement for predictability. Superior performance, even under duress. The ability to withstand multiple component failures. Sophisticated replication at scale. Advanced management capabilities. And so on.

Historically, these mission-critical applications ran on dedicated physical servers, but that's changed as well -- the preference is quickly becoming virtual-first, which introduces even more demanding requirements into the environment.

From an architectural perspective, it's hard to claim to be an enterprise-class storage array if you only have two storage controllers. Why? If the first one fails over to its twin, you're looking at a 50% performance degradation. While that might be acceptable in many environments; it's unacceptable in the most demanding. So let's presume multi-controller designs as at least one of the defining characteristics.

So why are smaller (and more cost-effective!) versions of these multi-controller arrays becoming so darn popular?

The Challenges Are The Same Everywhere

If you're a bank, or a telco, or any other large-scale operation, there are a handful of applications that keep you in business. Bad IT days are to be avoided at all reasonable costs. If you're operating in, say, one of the western economies, you're probably operating a decent scale -- and can justify full-blown implementations of these storage architectures.

But let's say you're doing business in China, or Brazil, or the Middle East. The business requirements are still the same, but you probably have a lot less capacity to deal with. And, indeed, there's surprisingly strong demand for these entry-level enterprise-class arrays outside the western economies.
But there's more at play here ...

A Longer Term Perspective Emerges

I've seen storage buying habits evolve now for almost 20 years, and -- not surprisingly -- a lot of storage gets sold in response to a specific requirement.

Here's what we need for SAP. Here's what we need for email. Sure, there's some opportunistic capacity sharing going on, but it's not the main thought.
Doing storage this way means that your landscape will grow willy-nilly over the years, and at some point there's a forcing function to consider a more consolidated approach, which motivates the customer to think in term of a smaller number of bigger arrays.

Of course, going to that sort of consolidated storage environment isn't the easiest IT project in the world ...
I think there are now enough intelligent enterprise IT buyers out there who have seen this movie before, and want to avoid it if possible.

As a result, they're willing to pay a small premium for an enterprise-class storage architecture that starts at reasonable size, but can expand to accommodate dramatic growth in application requirements, number of applications, performance, protection, availability, etc.
They want to start out on the right foot, so to speak.

Now, we here at EMC don't want to be caught napping when the market moves, so we targeted this specific space last year with the VMAX 10K. It’s been surprisingly successful. But, since then, both Hitachi and HP have gotten into the game.

So we need to bring more game.

Introducing The New VMAX 10K
Not one to rest on our laurels, there's now a new member of the VMAX family to consider -- a significantly enhanced version of the 10K.

More performance. More functionality. Aggressive price. And it's a VMAX.

I think it's going to do extremely well in this segment.

VMAX 10K At A Glance
Having been around this business for so long, I continue to be amazed at just how far these technologies have evolved.
Case in point: the "baby" VMAX 10K array is fairly impressive in its own right:

• one to four storage engines (that's a total of eightcontrollers)
• up to 512 GB of storage cache
• up to 1,560 drives and 64 ports, max 1.5 PB usable capacity
• can start as small as a single storage engine and 24 drives

Plus, of course, all the rich software that's integral to the VMAX: Enginuity, SRDF, FAST VP, et. al.
So, with that as our baseline -- what's new?

2X Performance Bump

Nice, but where does this particular claim come from?
A bunch of little things, and a few big ones -- the use of 2.8 GHz Xeon Westmere Intel processors, which now deliver 12 cores @ 2.8 GHz vs. the previous 8 cores @ 2.4 Ghz per storage engine. The 10K also now uses the same internal interconnect as its big brother, the VMAX 40K.

Translated, this means roughly 2x the back-end IOPS, and a useful 30% bump on front-end bandwidth.

Of course, the *exact* performance improvement a specific customer may see is dependent on all sorts of factors, but just about everyone should see something noticeable.
More performance also means more workloads that can be consolidated, larger effective capacities can be driven harder, more advanced functionality (e.g. tiering, replication) can be used with less impact, and so on.
More performance is always a good thing.

Federated Tiered Storage Improvements

Over the last few years, many enterprise storage arrays have learned a neat trick: they can connect to existing block storage arrays and "front end" them.

This can be especially useful if you've got the typical 'stranded asset' problem -- the old storage now is somewhat more useful: it can benefit from the caching being done by newer array, it can be used with all the cool software features of the new enterprise array, and so on.
Both EMC and Hitachi have done this for a while. As far as I know, HP doesn't do this with 3PAR for architectural reasons.

Not surprisingly, EMC does this particular trick a bit better than Hitachi:

- EMC implements an error-checking protocol on data transfers between the VMAX and the older storage
- EMC allows the external capacity to play at anytier as part of FAST VP
This last bit is more useful than it might sound. While there's plenty of need to recycle older, slower storage as a capacity tier, we're guessing that people will eventually want to consolidate newer, faster arrays (maybe flash-based?) as part of a larger, consolidated tiering environment.

And, while we're on FTS (federated tiered storage), don't underestimate the power of bringing newer features to older storage: things like VPLEX and Recoverpoint just work of course, but also snaps, clones, remote replicas, storage management tools, etc. etc. I think it's an under-utilized capability in most shops.

Data At Rest Encryption (D@RE)

Another small bit of magic that the VMAX team has come up with is their implementation of data encryption. It's about as seamless and transparent as you can imagine.

Because it uses encryption engine on the controller (vs. on the drive) it can encrypt any storage type, including externally attached storage arrays. There is *no* measurable performance impact. And it's now integrated with the RSA Key Manager for a somewhat easier deployment.

If someone comes to you and says "hey, we should be encrypting all of our drives", it's sort of a turn-it-on-and-forget-about-it situation for most customers. If you think you're going to need the encryption feature, it should be specified when you order the product -- it can't be easily added later.

Data Compression For Old Data

In this release of Enginuity, we've got the first implementation of yet another efficiency technology to put into play -- in-place compression of stale data.

The customer sets a threshold (e.g. not accessed in the last 60 days), and the VMAX will compress it. Start accessing the data again, and it expands. While this doesn't meet the ultimate goal of real-time on-the-fly dynamic space reduction, it's still very useful in most production environments.

Quick thought -- how much of your storage capacity hasn't been accessed in the last 60 days?

Host I/O Limits

Most storage arrays try to please everyone -- turn around each and every I/O request just as fast as you can.

While that's an admirable goal, it's not ideal for everyone.
For example, we've got a growing cadre of service provider customers who deliver tiered storage services using VMAX.
This new feature helps them articulate *exactly* what customers are getting when they sign up for a particular class of storage service, e.g. 2000 IOPS.

Customers who sign up for a low storage performance level shouldn't get a "free lunch", as there would be less incentive to move up to a higher service class.

More pragmatically, customers and service providers continually want to push these arrays to their max, and these controls help them better utilize aggregated resource limits: CPU, memory, bandwidth. If you can limit what one group of applications can get (no need to overprovision) you can get much more useful work out of a given asset.


Not exactly a typical requirement, but it does exist.
This release of Enginuity implements host I/O controls (defined either by storage group and/or port group) to limit how much bandwidth and/or IOPs are allowed. This function is integrated into Unisphere for VMAX, which makes it straightforward to set up and monitor.
There's a longer list of other features in the new version of Enginuity as well -- available on all the VMAXen -- but these are the ones that stood out for me.

Packaging

The VMAX 10K now supports popular third-party racks. Again, a little thing that means a lot to certain customers.
Customers can either use the EMC-supplied enclosures, or go with a decent number of popular racking options for the VMAX 10K components. There's even a nifty Cylon-style VMAX bezel available to preserve the nice looks :)

While we're on physical packaging, the VMAX 10K now supports what's dubbed "system bay dispersion", allowing two enclosures to be separated to aid in weight distribution (these are heavy boxes) or just give you a bit more flexibility as to where everything goes.

There's now support for intermixed 2.5" drives as well as the more familiar 3.5".
There are more than a few IT organizations who've put a lot of thought and effort into their data center physical infrastructure, and they've standardized on racking for all the right reasons. Service providers, in particular, care about this greatly.

The Partner Opportunity
Historically, the VMAX hasn't been what you'd describe as a "partner friendly" product for a variety of reasons.

Well, with the VMAX 10K, that's apparently changed. I was quite pleased to see just how many VMAX 10Ks have been sold as part of partner-led engagements.

For partners, you can see why it's an attractive proposition: the VMAX is a very differentiated offering, it targets customers who are looking for a high-value solution, and there's typically a nice services component that goes with it.
And, of course, the product has a helluva good reputation :)

What All Of This Might Mean
This particular segment of the market (entry-level enterprise-class storage arrays) have recently become popular -- which sort of surprises me.

If I could point to a root cause, I'd suggest it's a maturation of perspective: enterprise storage is much more than just a bunch of disks sitting behind a pair of controllers.

Like any growth area, it draws new competitive entrants. We at EMC have to compete vigorously in each and every segment of the broader storage marketplace. Nothing is easy in the storage market -- there are plenty of hungry companies out there.

If anything, the new VMAX 10K sends a clear message: we're bringing our best game to our customers, each and every day.

 

Storage 101 - Part 4

Storage 101 - Part 1

Storage 101 - Part 2
Storage 101 - Part 3


This article continues the discussion of Storage Area Networking by discussing Fibre Channel switches.

Introduction

In my previous article in this series, I talked about some of the fabric topologies that are commonly used in Storage Area Networks. In case you missed that particular article, a fabric is essentially either a single switch or a collection of switches that are joined together to form the Storage Area Network. The way in which the switches are connected forms the basis of the topologies that I discussed in the previous article.

Fibre Channel Switches

Technically a SAN can be based on either iSCSI or Fibre Channel, but Fibre Channel SANs are far more common than iSCSI SANs. Fibre Channel SANs make use of Fibre Channel switches.

Before I get too far into a discussion on Fibre Channel switches, I need to explain that although a fabric is defined as one or more switches used to form a storage area network, the fabric and the SAN are not always synonymous with one another. The fabric is the basis of the SAN, but a SAN can consist of multiple fabrics. Typically multi fabric SANs are only used in large, enterprise class organizations.

There are a couple of reasons why an organization might opt to use a multi fabric SAN. One reason has to do with storage traffic isolation. There might be situations in which an organization needs to isolate certain storage devices from everything else either for business reasons or because of a regulatory requirement.

Another reason why some organizations might choose to use a multi-fabric SAN is because doing so allows an organization to overcome limitations inherent in Fibre Channel. Like Ethernet, Fibre Channel limits the total number of switches that can be used on a network. Fibre Channel allows for up to 239 switches to be used within a single fabric.

Given this limitation it is easy to assume that you can get away with building a single fabric SAN so long as you use fewer than 239 switches. For the most part this idea holds true, but there some additional limitations that must be taken into consideration with regard to fabric design.

Just as the Fibre Channel specification limits the number of switches that can be used within a fabric, there are also limitations to the total number of switch ports that can be supported. Therefore, if your Fibre Channel fabric is built from large switches with lots of ports then the actual number of switches that you can use will likely be far fewer than the theoretical limit of 239.

Fibre Channel Switching Basics

One of the first things that you need to know about Fibre Channel switches is that not all switches are created equally. Fibre Channel is a networking standard and every Fibre Channel switch is designed to adhere to that standard. However, many of the larger switch manufacturers incorporate proprietary features into their switches. These proprietary features are not directly supported by the Fibre Channel specification.

That being the case, the functionality that can be achieved within a SAN varies widely depending upon the switches that are used within the SAN. It is perfectly acceptable to piece a SAN together using Fibre Channel switches from multiple vendors. Doing so is fairly common in fact, simply because of how quickly some vendors offer and then discontinue various models of switches. For example, an organization might purchase a Fibre Channel switch and later decide to expand their SAN by adding an additional switch. By the time that the second switch is needed the vendor who supplied the previously existing switch might have stopped making that model of switch. Hence the organization could end up using a different model of switch from the same vendor or they might choose to use a different vendor’s switch.

When a fabric contains switches from multiple vendors (such as HP and IBM) the fabric is said to be heterogeneous. Such situations are also sometimes referred to as an open fabric. When a SAN consists of one or more open fabrics, the switch’s proprietary features must usually be disabled. This allows one vendor’s Fibre Channel switch to work with another vendor’s switch since each switch adheres to a common set of Fibre Channel standards.

The alternative of course is to construct a homogeneous fabric. A homogenous fabric is one in which all of the switches are provided by the same vendor. The advantage of constructing a homogenous fabric is that the switches can operate in native mode, which allows the organization to take full advantage of all of the switch’s proprietary features.

The main disadvantage to using a homogenous fabric is something that I like to call vendor lock. Vendor lock is a situation in which an organization uses products that are provided by a single vendor. When this goes on for long enough, the organization becomes completely dependent upon the vendor. Vendor dependency can lead to inflated pricing, poor customer service, and ongoing sales pressure.

Regardless of which vendor’s switches you choose to use, Fibre Channel switches generally fall into two different categories – Core and Edge.

Core switches are also sometimes called Director switches. They are used primarily in situations in which redundancy is essential. Typically a core switch is built into a rack mounted chassis that uses a modular design. The reason why the switch has a modular design is for redundancy. A core switch is generally designed to prevent individual components within the switch from becoming single points
of failure.

The other type of switch is called an edge switch. Edge switches tend to have fewer configuration options and less redundancy than core switches. However, some edge switches do have at least some degree of redundancy built in.

It is important to understand that the concepts of core switches and edge switches are not a part of the Fibre Channel specification. Instead, vendors market various models of switches as either core switches or edge switches based on how they intend for a particular switch model to be used. The terms core and edge give customers an easy way to get a basic idea of what they can expect from the switch.

SAN Ports

I plan to talk in detail about switch ports in Part 5 of this series, but for right now I wanted to introduce you to the concept of Inter Switch Linking. Fibre Channel switches can be linked to one another through the use of an Inter Switch Link (ISL). ISLs allow storage traffic to flow from one switch to another.

As you will recall, I spent some time earlier talking about how some vendors build proprietary features into their switches that will only work if you use switches from that vendor. Some of these features come into play with regard to ISLs.

ISLs are a Fibre Channel standard, but some vendors use ISLs in a non-standard way. For example, most switch vendors support a form of ISL aggregation in which multiple ISL ports can be used together to emulate a single very high bandwidth ISL. Cisco refers to this as EISL, whereas Brocade refers to it as trunking. The point is that if you want to use ISL aggregation you will have to be vendor consistent with your Fibre Channel switches.

Conclusion

In this article I have tried to familiarize you with some of the basics of Fibre Channel switches. In Part 5, I plan to talk about Fibre Channel switch ports.

 

Storage 101 - Part 3

Storage 101 - Part 1

Storage 101 - Part 2

This article continues the discussion of storage area networks by talking about the storage fabric and about the three most commonly used fabric topologies.

Introduction

In the second part of this article series, I talked all about hosts and host hardware. In this article, I want to turn my attention to the storage fabric.


As previously explained, SANs consist of three main layers – the host layer (which I talked about in the previous article), the fabric layer, and the storage layer. The fabric layer consists of networking hardware that establishes connectivity between the host and the storage target. The fabric layer can consist of things like SAN hubs, SAN switches, fiber optic cable, and more.

Fabric Topologies

Before I get too far into my discussion of the fabric layer, I need to explain that SANs are really nothing more than networks that are dedicated to the sole purpose of facilitating communications between hosts and storage targets. That being the case, it should come as no surprise that there are a number of different topologies that you can implement. In some ways SAN fabric topologies mimic the topologies that can be used on regular, non-SAN networks. There are three main fabric topologies that you need to know about. These include point to point, arbitrated loop, and switched fabric.

Point to Point

Point to point is the simplest and least expensive SAN fabric topology. However, it is also the least practical. A point to point topology is essentially a direct connection between a host and a storage target. The simplicity and cost savings come into play in the fact that no additional SAN hardware is needed (such as switches and routers). Of course the price for this simplicity is that the fabric can only include two devices – the host and the storage target. The fabric cannot be expanded without switching to a different topology. Because of this, some would argue that point to point isn’t even a true SAN topology.

Arbitrated Loop

The simplest and least expensive “true SAN” topology is an arbitrated loop. An arbitrated loop makes use of a Fibre Channel hub. Hubs are kind of like switches in that they contain ports and devices can communicate with each other through these ports. The similarities end there however.

Fibre channel hubs lack the intelligence of a switch, and they do not segment communications like a switch does. This leads to a couple of important limitations. For starters, all of the devices that are attached to a hub exist within a common collision domain. What this means is that only one device can transmit data at a time. Otherwise, if two devices attempted simultaneous communications the transmissions would collide with each other and be destroyed.

Because of the way that Fibre Channel hubs work, each hub provides for a certain amount of bandwidth and that bandwidth must be shared by all of the devices that are connected to the hub. This means that the more devices you connect to a Fibre Channel hub, the more each device must compete with other devices for bandwidth.

Because of bandwidth limitations and device capacity, the arbitrated loop topology is suitable only for small or medium sized businesses. The limit to the number of devices that can be connected to an arbitrated loop is 127. Even though this probably sounds like a lot of devices, it is important to remember that this is a theoretical limit, not a practical limit.

In the real world, Fibre Channel hubs are becoming more difficult to find, but you can still buy them. Most of the Fibre Channel hubs that I have seen recently only offer eight ports. That isn’t to say that you can only build a hub with eight devices however. You can use a technique called hub cascading to join multiple hubs together into a single arbitration loop.

As the arbitration loop grows in size, there are a few things to keep in mind. First, the 127 device limit that I mentioned previously is the limit for the entire arbitration loop, not just for a single hub. You can’t exceed the device limit just by connecting an excessive number of hubs together.
Another thing to consider is that the hub itself counts as a device. Therefore, an eight port hub with a device plugged into each port would actually count as nine devices.

Probably the most important thing to remember with regard to hub cascading is that hardware manufacturers have their own rules about hub cascading. For example, many of the Fibre Channel hubs on the market can be cascaded twice, which means that the maximum number of hubs that you could use in your arbitration loop would be three. If you assume that each hub contains eight ports then the entire arbitration loop would max out at 24 devices (although the actual device count would be 27 because each of the three hubs counts as a device).

Keep in mind that this represents a best case scenario (assuming that the manufacturer does impose a three hub limit). The reason why I say this is because in some cases you might have to use a hub port to connect the next hub in the cascade. Some hubs offer dedicated cascade ports separate from the device ports, but others do not.

Earlier I mentioned that using an arbitration loop was the cheapest and easiest way to build a SAN. The reason for this is that Fibre Channel hubs do not typically require any configuration. Devices can simply be plugged in, and the hub does the rest. Keep in mind however, that arbitration loops tend to be slow (compared to switched fabrics) and that they lack the flexibility for which SANs have come to be known.

Switched Fabric

The third topology is known as a switched fabric. Switched fabric is probably the most widely used Fibre Channel topology today. It is by far the most flexible of the three topologies, but it is also the most expensive to implement. When it comes to SANs however, you usually get what you pay for.

As the name implies, the switched fabric topology makes use of a Fibre Channel switch. Fibre Channel switches are not subject to the same limitations as hubs. Whereas an arbitration loop has a theoretical limit of 127 devices, a switched fabric can theoretically scale to accommodate millions of devices. Furthermore, because of the way that a switched fabric works, any device within the fabric is able to communicate with any other device.

As you can see, Fibre Channel switches are very powerful, but they also have the potential to become a single point of failure. A switch failure can bring down the entire SAN. As such, switched fabrics are usually designed in a way that uses redundant switches. This allows the SAN to continue to function in the event of a switch failure. I will discuss switched fabrics in much more detail in the next article in this series.

Conclusion

In this article, I have introduced you to the three main topologies that are used for SAN communications. In Part 4 of this article series, I plan to talk more in depth about the switched fabric topology and about Fibre Channel switches in general.

 

Seven Shifts in the Storage

If you're in the IT industry, you're intimately familiar with technology shifts: you've seen plenty of them, and there are undoubtedly many more to come.

That's what keeps things interesting ...

Of all the foundational infrastructure technologies; none appears to be changing faster than storage. From a macroeconomic perspective, the shift to digital business models means information: orders of magnitude more of it gathered, analyzed and acted on.
More storage, please.

And, on a more pragmatic note, when IT budgets are cramped, one of the first categories that always comes under intense scrutiny is -- you guessed it -- storage.

One of the questions I get asked is -- what are the big shifts currently going on in the storage world?
I thought my answer was worth sharing -- see if you agree.

Shifts Abound
Knowing that something is changing, and doing something about it, turn out to be two very different activities.

For EMC as a storage vendor, we are very mindful of each and every one of these shifts, and are investing heavily along these lines.
To not do so -- and do so aggressively -- means failing to keep pace with technology evolution; a fate we would not want to suffer like so many of our colleagues.
For our customers and partners, the challenge is a bit different: not all of this can be put into practice due to multiple real-world constraints.
That's where the notion of a strategy comes in: here's an idealized end-state; and here are some progressive steps we can take to always make sure we're moving in the right direction.
If anything, I hope you find some of this useful fodder for your own strategy moving forward.


#1) Storage Media

The first key shift is in storage media: tape has been largely replaced by disk, and traditional uses for disks are quickly being replaced by flash storage. With regards to tape, inline deduplication technology (e.g. EMC Data Domain) has lowered the cost of using disk for backup and archive to the point where disk-based data protection solutions are often now both faster and cheaper than their tape predecessors.

For primary storage, the promise of flash instead of disk is simple: vastly more performance at lower cost.
As a result, flash is quickly finding its way into the entire storage hierarchy: in the server (EMC VFcache) in the storage network (Project Thunder) and in the array (EMC FAST) -- all orchestrated by software that ensures the right data is in the right place at the right time.

The use of multi-core processors and inline deduplication is now making possible all-flash storage arrays (XtremIO) that can compete effectively with hybrid flash/disk combinations.

And, of course, the core SLC and MLC flash technologies (as they are on a semiconductor technology curve) will continue to get ever cheaper, denser and more cost effective.

#2) Technology Base

The second key shift is around the technology base. Previously, storage arrays used proprietary or customized hardware to achieve superior performance and availability.
This has quickly given way to the use of industry standard components running advanced software, shifting the value proposition away from hardware and towards modern storage software stacks.

Look inside any modern storage array, it's essentially a specialized server built from familiar components that represent the very best in price/performance -- running an amazingly sophisticated software stack.

Hardware still plays a role -- of course -- but it is less pronounced over time. Take this idea to its logical extreme, and it's not hard to imagine rich, virtualized software stacks running on very standardized hardware.
All EMC storage products reflect this philosophy.


#3) Architecture

The third key shift is architectural: in a world of big data, scale-out storage architectures are demanded.
With scale-out, multiple modules are clustered into progressively larger configurations that behave as a single unit from a performance, capacity and management perspective.

Can we always make bigger, traditional scale-up storage arrays? Of course. Will they be enough to keep up with even bigger data growth and performance requirements? Not so well.

This design theme can be seen in EMC products such as VMAX (scale-out block storage), Isilon (scale-out NAS storage) and Atmos (scale-out object storage).



#4) Convergence

The fourth key shift is convergence: storage, network and servers are growing together, largely in response for even greater efficiencies.

One current example of this theme is the popular VCE Vblock, a tightly-integrated enterprise cloud platform built on converged infrastructure from Cisco, VMware and EMC.
Future technology directions point to even more variations of converged storage / compute / memory modules that auto-configure and seamlessly redistribute workloads.
The implication is clear: over time, we'll be thinking less about storage as a standalone component, and more as part of a converged environment.

#5) Distance

The fifth key shift is the need to overcome distance: storage in data centers increasingly demand geographical separation for protection, performance or resource balancing concerns. There's just too much information pouring in (and being globally consumed) to simply assume it all can be sent to (and distributed from) a small number of centralized locations.

Enter the new world of federated architectures: infrastructure, data services, etc.

Newer EMC technologies (VPLEX, Atmos), are able to geographically distribute data without being disruptive to applications or management operations: a capability that is becoming increasingly important.


#6) Management And Orchestration

The sixth key shift is management and orchestration: storage resources are increasingly presented as a dynamic and variable service to other layers of the IT stack: hypervisor, middleware, application, etc.

This leads to a new management and operations model that demands new tools and methodologies (EMC Unisphere and ProSphere, EMC DataBridge, VMware vCenter Operations Manager, etc.).

While there will always be value in managing storage as a self-contained domain; the path forward lies in producing and orchestrating storage services that are consumed by others, which leads us to ...

#7) Consumption Model

The IT model has started to shift to an "as a service model". IT owns the relationship, and builds or brokers a variable service catalog to meet requirements.

The technology discussion quickly morphs into a service delivery discussion, and the storage domain is not immune from this shift.
This shift has two components: a re-thinking about how storage teams are organized and measured (service delivery); and an increasing attractiveness of brokered (e.g. external) storage services.

New models mean new roles and new skills -- and the modern storage IT professional looks very different than the ones from just a few years ago.






Challenges -- Or Opportunities?
I meet with more than my fair share of customers and partners who look at all of this storage stuff like it's one big, honkin' headache: rapid technology shifts, an information deluge, limited budget and a shortage of skills and talent.
Where's the magical silver bullet?

I haven't found one yet. But there is a hard choice to be made ...
Do you -- the IT leader -- want to invest in the capabilities to get really good at this storage stuff? Service catalogs, management and orchestration, federation -- all that?
Or, in the final consideration, do you choose to invest your resources elsewhere, and just bring in someone to do it all for you, and deliver a service that's efficient and easy to consume?
That's not a philosophical discussion, it's becoming a hard-nosed business decision. There's only so much resource available for the IT team; and choices have to be made around where to invest, and where to externalize.
But this much is clear -- you can only sit in the middle of the road for so long ...

 

Storage 101 - Part 2

Storage 101 - Part 1

This article continues the series by discussing some of the hardware that is used in storage area networks. Specifically, this article will focus primarily on Fibre Channel host bus adapters.

Introduction

In the first part of this article series, I explained that a SAN is different from NAS and from other types of networked storage. In this article, I want to discuss some of the hardware components that make up a typical SAN.

The Basic Architecture

SAN components generally fall into three categories – hosts, fabric, and storage. These three generic component types make up the SAN architecture. I will be discussing each of these component categories individually.

Hosts

Hosts aren’t technically a SAN component since they are actually the servers that connect to the SAN rather than the SAN itself. Even so, the hosts do contain hardware that makes this connectivity possible.

The hardware that is present in a host varies depending upon how the host connects to the SAN. If the host connects to the SAN using the iSCSI protocol then the host will establish this connectivity through a standard network adapter. This might be a gigabit network adapter, a ten gigabit network adapter, or even a series of teamed (bonded) NICs.

Fibre Channel

Fibre Channel connectivity is a bit more interesting. Hosts connect to a Fibre Channel SAN by using a host bus adapter. A host bus adapter is similar to a network card in that it gets installed into a slot within the server and provides connectivity to the SAN. The primary difference between host bus adapters and Ethernet adapters is that host bus adapters are designed to work with fiber optic cables rather than Ethernet cables. Another difference is that unlike Ethernet, Fibre Channel does not use the TCP/IP protocol or carry Ethernet packets.

Another aspect of the host bus adapter that might seem peculiar to those who have only worked with Ethernet cards is that host bus adapters are dependent upon another component called the Gigabit Interface Converter. The Gigabit Interface Converter contains the lasers that are used for fiber optic communications and it also provides the physical connectivity to the fiber optic cable. Some host bus adapters have a gigabit interface converter built into the card, but other adapters require the gigabit interface converter to be plugged into a slot in the card.

As previously mentioned, Fibre Channel is based on the use of fiber optics. Fiber optics use lasers to send data in the form of pulses of light. These pulses of laser light travel across cables made of woven glass.

Every Fibre Channel device contains two ports – a send port and a receive port. A host bus adapter’s send port is connected to another device’s receive port. That other device is usually a switch, but I’m getting ahead of myself. Likewise, the host bus adapter’s receive port is connected to another device’s send port. What this means is that data is sent and received through two separate ports, which allows for full duplex communications. In other words, Fibre Channel host bus adapters are able to send and receive data at the same time.

So far I have given you a general overview of how host bus adapters work, but I haven’t really talked about why the gigabit interface converter is sometimes separate from the host bus adapter.
Host bus adapters with integrated gigabit interface connectors are beginning to become the norm, but there is a good reason why some adapters still require a separate gigabit interface connector. It’s because Fibre Channel is not a one size fits all medium.

To show you what I mean, think for a moment about USB. When first released, USB was designed to be universal – hence the name Universal Serial Bus. However, today there is no such thing as a standard USB cable. USB cables come in several different form factors of varying sizes and shapes. The need to incorporate USB connectivity into smaller and smaller electronic devices drove the development of small form factor USB interfaces such as micro USB.

This same basic principle also applies to host bus adapters. There are two main types of connectors that host bus adapters use. Having gigabit interface connectors that are separate from the host bus adapter itself makes for a modular design that allows the host bus adapter to be more versatile.
The first type of connector that you need to know about is called an SC connector. SC connectors are the older type of connector, but are still in use today. They are designed to be used with Fibre
Channel connections that operate at a speed of 1 gigabit per second or slower.

The other type of Fibre Channel connector is called an LC connector. This is the connector type that is most commonly used today. LC connectors are smaller than SC connectors and are used with Fibre Channel hardware that operates at much higher speeds than those devices that use SC connectors.

Devices that use LC connectors can operate at speeds of 2, 4, or 8 gigabits per second. In some cases a host bus adapter’s overall bandwidth is divided among multiple sets of ports. For example, some eight gigabit host bus adapters offer twin four gigabit ports. A port refers to a set of send and receive connectors, so a card with twin ports would contain two send connectors and two receive connectors.

You might occasionally see references to ten gigabit Fibre Channel, but at the present time 10 gigabit Fibre Channel is not based on the same technology as that of lower Fibre Channel speeds. 10 gigabit Fibre Channel is based on Fibre Channel over Ethernet (FCoE), which sends Fibre Channel traffic over 10 gigabit Ethernet. The fact that the Fibre Channel traffic has to be encapsulated into Ethernet packets means that 10 gigabit FCoE does not deliver the same level of performance that true 10 gigabit Fibre Channel would.

Just as Fibre Channel hardware can use different form factor connections and can operate at different speeds, there are also ratings for distance. Most of the host bus adapters on the market use short wave lasers. Short wave lasers operate in the 780 nm to 850 nm range and can reliably send and receive data at distances of up to 500 meters. This is typically more than adequate since SAN components are normally located in close physical proximity to one another.

Sometimes however, a SAN implementation might require data to be sent over a much longer distance. This is where long wave lasers come into play. Long wave lasers generally operate at a wavelength of 1300 nm. The distance that they can achieve varies depending on the device. Some long wave Fibre Channel hardware has a distance limitation of 10 KM (6.21 miles). However, some long wave Fibre Channel hardware can send and receive data at distances in excess of 100 KM (62.10 miles).

Conclusion

Now that I have discussed host hardware, I want to turn my attention in Part 3 to the fabric layer of storage area networks. The fabric layer consists of things like hubs, switches, and routers that have been specially designed for use with Fibre Channel.

 

Storage 101 - Part 1

 

Introduction

It seems that everywhere you turn these days, you hear someone talking about Storage Area Network (SAN) storage. You might have heard someone say that SANs are complicated or expensive, but you might have wondered how a SAN differs from a traditional network. In this article series, I will discuss the basics of Storage Area Networking. My plan is to start out by talking about what a SAN is (and is not). From there, I want to talk about some of the hardware that is used in a SAN, as well as some common SAN architectures. Reading this article series won’t make you a SAN expert, but it should give you a much better understanding of Storage Area Networking.

What is a Storage Area Network?

I will never forget the first time that I ever heard someone mention a SAN. Many years ago a friend called me on the phone excited because he had just implemented a SAN. When I asked him what a SAN was he told me that his storage was tied directly to the network. I remember wondering what the big deal was since networked storage had been around for years.

 

At that point in my life (which was many years ago) I had never heard of a SAN, so perhaps I misunderstood my friend’s explanation. It’s also possible however, that my friend didn’t fully understand what a SAN was either. In either case, my friend’s definition of a SAN was somewhat accurate, but completely inadequate.

 

A SAN is a collection of networked storage devices as my friend had said, but a SAN is completely different from Network Attached Storage (NAS) which is also a form of networked storage.

 

There are three main ways in which a SAN differs from NAS. First, a SAN uses different hardware than NAS does. Second, SANs utilize different protocols than NAS devices do. Third, SANs read and write data in a different way than NAS does.

 

To show you what I mean, consider the nature of a NAS device. There are a lot of different types of NAS devices on the market, and some are more sophisticated than others, but generally speaking, a NAS device is an appliance that connects to the network via one or more Ethernet cables. A NAS appliance contains one or more disks, and is usually configurable through a Web interface. This interface usually allows the device’s storage to be partitioned or to be used as a RAID array.

 

Once the NAS device is put into production, the device is treated much like a typical file server. Users connect to the NAS device through an Ethernet connection using the TCP/IP protocol.

 

Depending on the type of NAS, it might also support SMB or NetBIOS over TCP/IP. In any case, you can think of a NAS device as a self-contained file server.

 

SAN storage works completely differently from NAS storage. When a NAS is in use, a user with the proper permissions can connect directly to a NAS volume (through a file share) and read and write files.

 

SANs can be configured to provide similar functionality, but there is a lot going on behind the scenes. For starters, users cannot generally connect directly to SAN storage because user workstations communicate with other computers on the network using TCP/IP. Although there are exceptions, SAN storage is usually accessed through Fibre Channel.

 

Even though this difference in protocols might seem trivial, it actually hints at the very essence of a SAN. Networks that depend on TCP/IP and SMB are primarily designed to access file system data. In other words, these types of networks are ideally suited for reading and writing files that are stored on file servers, Web servers, etc.

 

In contrast, Fibre Channel doesn’t work at the file level, but rather at the storage block level. As such, you wouldn’t use Fibre Channel to read a file that is stored on a file share. Instead, Fibre Channel reads and writes individual storage blocks.

 

There are a couple of reasons why this seemingly trivial distinction is important. First, Fibre Channel offers much higher performance than a traditional TCP/IP network does. Although network throughput does play a role in the overall speed of the connection, the main reason why Fibre Channel is so much faster than TCP/IP is because Fibre Channel is a more efficient protocol with less overhead. Having less overhead allows Fibre Channel to move data more quickly.

 

The other reason why Fibre Channel’s block level storage interaction is significant is because Fibre Channel communicates directly (and natively) with the storage device. This means that in a SAN environment, it is possible to treat a remote storage device as if it were a local storage resource.

 

To give you a better idea of what I am talking about, consider what happens when you connect a Windows machine to a NAS device. The NAS storage gets mapped to a network drive. In the case of a SAN however, it is possible to get Windows to treat a SAN volume as local storage (as opposed to a network drive), even if the physical storage device is located remotely.

 

This is an important distinction because the Windows operating system treats local and networked storage differently. For example, there are Windows applications that can be installed to local storage, but not to a network drive. These types of applications can be installed to SAN storage however, because the Windows operating system does not distinguish between true local storage and SAN storage (at least not as far as the application is concerned).

 

Keep in mind that I am not saying that SAN storage is always treated as local storage or that it cannot be used for anything else. Often times the end users actually see SAN storage as a mapped network drive.

 

So how can this be? It all has to do with the fact that the users workstations do not normally connect directly to SAN storage. It is usually servers (or virtual workstations) that make use of SAN storage. Imagine for example that a file server is configured to use SAN storage instead of true local storage. The file server is connected to the SAN in a way that allows the storage to be treated as local. However, when end users attach to the file server they might be accessing files that are stored on the SAN, but they are not directly connecting to the SAN. Instead, the users are connecting to the file server via TCP/IP. The file server is the only machine that accesses the SAN storage directly.

 

This architecture really isn’t much different than it would be if the file server were using direct attached storage. Even if the file server’s storage were truly local, the users wouldn’t access the storage directly. The users communicate with the file server’s operating system and it is the operating system that hands off disk requests to the storage subsystem. Exactly the same thing happens if the file server is connected to a SAN. The only difference is that the storage is not local to the file server.

Conclusion

In this article, I have explained that there are some major differences between Storage Area Networks (SAN) and Network Attached Storage (NAS). In Part 2 of this series, I will begin discussing the hardware that is used in a SAN.

Storage 101 - Part 2
 

FCIP, IFCP, iSCSI in IP Storage

IP storage: A review of iSCSI, FCIP, iFCP

By Jane Shurtleff

With the advent of new IP storage products and transport protocol standards�iSCSI, FCIP, and iFCP (due out in mid-2002)�end users now have more choices for accessing data over IP networks. With the emergence of these products and standards, the Storage Networking Industry Association's (SNIA) IP Storage Forum is rising to the challenge of educating end users on the differences among the three data transport protocols.

The SNIA IP Storage Forum is made up of more than 50 system, storage, networking, and application vendors. At the Storage Networking World conference last month, the IP Storage Forum demonstrated a number of storage applications running on iSCSI, FCIP, and iFCP. They also presented a tutorial on IP storage networking ("Clearing the Confusion: A Primer on Internet Protocol Storage") on which this article is based. The IP Storage Forum tutorial, as well as a variety of white papers on each of the IP storage networking technologies, can be found on the SNIA Website, www.snia.org.

Benefits of IP storage The benefits of IP storage networking have been well recognized within the network-attached storage (NAS) arena for moving files over IP-based LANs. IP storage leverages the large installed base of Ethernet-TCP/IP networks and enables storage to be accessed over LAN, MAN, or WAN environments, without needing to alter storage applications. It also lets IT managers use the existing Ethernet/IP knowledge base and management tools.

However, for block-level data that is stored as either direct-attached storage (DAS) or on a Fibre Channel storage area network (SAN), taking advantage of these benefits requires new transport protocols for moving that data over IP networks. The development of IP storage networking transport mechanisms for block-level storage enables IT managers to create and manage heterogeneous environments where DAS and Fibre Channel SANs can be integrated over a common IP network backbone. These environments will allow better utilization of storage resources and support existing storage applications such as backup and disaster recovery. New developments in IP storage networking (e.g., storage virtualization, which enables managers to create virtual storage pools among geographically dispersed DAS, NAS, and SAN data resources) have also fostered new applications to better manage these environments.

iSCSI, FCIP, and iFCP The three IP storage networking transports are significantly different, but they all provide a common function: transporting block-level storage over an IP network. All three transports enable end users to Leverage existing storage devices (SCSI and Fibre Channel) and networking infrastructures (Gigabit Ethernet); Maximize storage resources to be available to more applications; Extend the geographical limitations of DAS and SAN access; Use existing storage applications (backup, disaster recovery, and mirroring) without modification; and Manage IP-based storage networks with existing tools and IT expertise.

The Internet Small Computer Systems Interface (iSCSI) protocol defines the rules and processes to transmit and receive block storage applications over TCP/IP networks by encapsulating SCSI commands into TCP and transporting them over the network via IP.

Fibre Channel over TCP/IP (FCIP) provides a mechanism to "tunnel" Fibre Channel over IP-based networks. This enables the interconnection of Fibre Channel SANs, with TCP/IP used as the underlying wide-area transport to provide congestion control and in-order delivery of data.

The Internet Fibre Channel Protocol (iFCP) supports Fibre Channel Layer 4 FCP over TCP/IP. It is a gateway-to-gateway protocol where TCP/IP switching and routing components complement and enhance, or replace, the Fibre Channel fabric.

Figure 1 illustrates the protocols supported at each end device and their underlying fabric services. The end device is either a host or a storage device, and the fabric services include routing, device discovery, management, authentication, and inter-switch communication.

When considering deployment of any of these IP storage networking mechanisms, you first need to consider your current storage environment and what you want to achieve. Here is a closer look at each of the three transports and how they are deployed.

Figure 1: End devices include hosts or target storage devices, and fabric services include routing, device discovery, management, authentication, and inter-switch communication. iSCSI The primary market driver for the development of the iSCSI protocol is to enable broader access of the large installed base of DAS over IP network infrastructures. By allowing greater access to DAS devices over IP networks, these storage resources can be maximized by any number of users or utilized by a variety of applications such as remote backup, disaster recovery, and storage virtualization. A secondary driver of iSCSI is to allow other SAN architectures such as Fibre Channel to be accessed from a wide variety of hosts across IP networks. iSCSI enables block-level storage to be accessed from Fibre Channel SANs using IP storage routers or switches, furthering its applicability as an IP-based storage transport protocol.

Between the standards efforts coming to completion and the SNIA IP Storage Forum's multi-vendor interoperability testing and demonstrations, iSCSI-compliant products will enable users to rapidly deploy IP SAN environments and immediately take advantage of the "plug-and-play" benefits of iSCSI. Many iSCSI products are already available, based on early versions of the specification.

How iSCSI works iSCSI defines the rules and processes to transmit and receive block storage applications over TCP/IP networks. At the physical layer, iSCSI supports a Gigabit Ethernet interface so that systems supporting iSCSI interfaces can be directly connected to standard Gigabit Ethernet switches and/or IP routers. The iSCSI protocol sits above the physical and data-link layers and interfaces to the operating system's standard SCSI Access Method command set. iSCSI enables SCSI-3 commands to be encapsulated in TCP/IP packets and delivered reliably over IP networks.

iSCSI can be supported over any physical media that supports TCP/IP as a transport, but today's iSCSI implementations are on Gigabit Ethernet. The iSCSI protocol runs on the host initiator and the receiving target device. iSCSI can run in software over a standard Gigabit Ethernet network interface card (NIC) or can be optimized in hardware for better performance on an iSCSI host bus adapter (HBA).

iSCSI also enables the access of block-level storage that resides on Fibre Channel SANs over an IP network via iSCSI-to-Fibre Channel gateways such as storage routers and switches.

Considerations for iSCSI deployment Initial iSCSI deployments are targeted at small to medium-sized businesses and departments or branch offices of larger enterprises that have not deployed Fibre Channel SANs. iSCSI is an affordable way to create IP SANs from a number of local or remote DAS devices. If there is Fibre Channel present, it is typically in a data center, which can be accessed by the iSCSI SANs (and vice versa) via iSCSI-to-Fibre Channel storage routers and switches.

Figure 2: iSCSI enables SCSI-3 commands to be encapsulated in TCP/IP packets and delivered reliably over IP networks. iSCSI SANs can be deployed within LAN, MAN, or WAN environments, as shown in Figure 2. The important cost saving factor to realize in any iSCSI SAN deployment is that the network infrastructure supporting iSCSI SANs is standard Gigabit Ethernet switches and/or IP routers. You can use your existing network and IT support resources with an iSCSI deployment, reducing TCO.

FCIP The emerging FCIP protocol standard takes advantage of the installed base of Fibre Channel SANs, as shown in Figure 3, and the need to interconnect these SANs to support mission-critical environments. SANs provide the high performance and reliability required to support business continuance and disaster tolerance environments, including remote backup/archiving, high availability, remote mirroring, and centralized management.

Click here to enlarge image Figure 3: FCIP enables multiple local Fibre Channel SANs to be interconnected, or remote SANs to be managed, over an IP network backbone. For most of these applications, Fibre Channel SANs can be interconnected to meet the needs for remote storage access. However, by combining IP networking with SAN technology, you can extend the interconnectivity of SANs across much longer distances. FCIP provides the transport for traffic going between specific Fibre Channel SANs over LANs, MANs, and WANs. Like iSCSI, the FCIP protocol is also being developed within the Internet Engineering Task Force (IETF) IP Storage Working Group and is expected to be completed by mid-year.

How FCIP works FCIP solutions encapsulate Fibre Channel packets and transport them via TCP/IP, which enables applications that were developed to run over Fibre Channel SANs to be supported under FCIP. It also enables organizations to leverage their current IP infrastructure and management resources to interconnect and extend Fibre Channel SANs.

FCIP is a tunneling protocol that uses TCP/IP as the transport while keeping Fibre Channel services intact. FCIP relies on IP-based network services and on TCP/IP for congestion control and management. It also relies on both TCP/IP and Fibre Channel for data-error and data-loss recovery.

In FCIP, gateways are used to interconnect Fibre Channel SANs to the IP network and to set up connections between SANs, or between Fibre Channel devices and SANs. Like iSCSI, there are a number of "pre-standard" FCIP products on the market.

Considerations for FCIP deployment FCIP enables multiple local or remote Fibre Channel SANs to be interconnected over an IP network backbone. Since FCIP keeps Fibre Channel services intact, it enables you to maintain a high-performance SAN base, while transparently increasing the interconnectivity and data sharing between SANs on an IP network.

FCIP gateways enable you to connect to a standard Gigabit Ethernet/IP infrastructure, so you are able to cost-effectively set up and manage an IP-based SAN-to-SAN network backbone. FCIP SANs can be deployed over LANs, MANs, or WANs.

iFCP Like FCIP, the primary market drivers for iFCP are the large installed base of Fibre Channel devices, combined with the momentum toward IP storage networking. The emerging iFCP standard leverages the high performance and interoperability of the Fibre Channel protocol, while taking advantage of IP networks.

Figure 4: iFCP allows Fibre Channel SANs to be interconnected via TCP/IP networks of any distance, using standard Gigabit Ethernet switches and routers. With iFCP, the lower-layer Fibre Channel transport is replaced with TCP/IP and Gigabit Ethernet. iFCP enables the rapid deployment of IP-based SANs linking to Fibre Channel devices or Fibre Channel SANs (see Figure 4). It allows you to implement enterprise-class solutions based on existing applications, which already communicate with the FCP layer. iFCP enables highly scalable implementations using existing Fibre Channel storage products and also allows multiple Fibre Channel SANs to be interconnected via TCP/IP networks of any distance, using standard Gigabit Ethernet switches and routers.

Enterprise-class solutions within a data center such as centralized backup, remote mirroring, storage management, and storage virtualization are supported within an iFCP environment due to the ability to create a scalable, peer-to-peer Fibre Channel/IP storage network.

How iFCP works Fibre Channel devices (e.g., switches, disk arrays, and HBAs) connect to an iFCP gateway or switch. Each Fibre Channel session is terminated at the local gateway and converted to a TCP/IP session via iFCP. A second gateway or switch receives the iFCP session and initiates a Fibre Channel session. In iFCP, TCP/IP switching and routing elements complement and enhance, or replace, Fibre Channel SAN fabric components. The protocol enables existing Fibre Channel storage devices or SANs to attach to an IP network. Sessions include device-to-device, device-to-SAN, and SAN-to-SAN communications.

Considerations for iFCP deployment Centralized consolidation of Fibre Channel SANs via iFCP is a consideration for those environments where there is a heavy investment in both Fibre Channel SANs and an enterprise-wide IP network backbone. The driving force behind iFCP is the expansion of IP-based network services to interconnect Fibre Channel devices and SANs. The increased port density and lower cost of Gigabit Ethernet switches vs. Fibre Channel switches enables these environments to scale and expand without increasing overall cost of ownership. Like FCIP, applications developed for Fibre Channel SAN environments are supported over iFCP. iFCP's peer-to-peer storage networking benefits enable broad access to, and consolidation of, storage resources to be used by a number of enterprise-class applications.

Even with the differences in transport mechanisms and deployment strategies, the one common factor that makes iSCSI, FCIP, and iFCP worth considering is the ease of deployment, management, and support associated with IP networking. All three transports will continue to be put through their paces with SNIA-supported interoperability testing and demonstrations.

For more information, refer to the following white papers on the SNIA IP Storage Forum Website (www.ipstorage.org): The Benefits of Internet Fibre Channel Protocol (iFCP) for Enterprise Storage Networks The Emerging FCIP Standard for Storage Area Network Connectivity Across TCP/IP Networks Basic Concepts of Internet SCSI.