blog.virtualtacit.com

As a follow up to the post, Unified I/O-FCoEasy does it….Successor to FC?, I thought it would be fair to drop some info on Infiniband and its place in the storage ecosystem.

IB from a raw throughput perspective is no doubt head of the class. Low latency, high bandwidth, and the ability to collapse both FC and Ethernet on a single transport are all contributing factors to IB’s push toward the de facto standard for high speed I/O interconnects. Although most notably a main staple in HPC (High Performance Computing) environments it quickly is targeting its efforts toward virtualization. And why shouldn’t it, the inevitable build up or build out of pure physical cabling to support even the most simplest designs makes this consolidated technology a perfect fit. VMware has already announced support for Mellanox HCA’s in its ESX 3.5 release, signaling that the tides are now split between IB and 10GigE as viable Unified options–but wait a minute, lets look under the covers shall we..

Comparison of both competing technologies above– (notice FCoE and IBoE are not listed but both are defining technologies in development)

So where do we stand from both IB’s perspective and Ethernet as a transport; the good, the bad, the ugly–noted here–http://www.nowmicro.com/NM_PDF/wp-cisco-interconnects.pdf

Advantages of IB

• Reduced CPU and memory overheads by utilizing specialized HCA hardware
• Remote Direct Memory Access (RDMA) allows for the placement of information from one computer system into the memory of another computer
  system, reducing latency and minimizing the processing overhead demands at the host system
• Low end-to-end system latency (from 5 to 10 microseconds, depending on the application)
• Multiple vendors supporting and shipping products
• Consolidated I/O with network, management, and storage all on one interface
• Implements various protocols, such as general-purpose remote-procedure call (RPC), direct-access transports (as part of RDMA), sockets direct protocol for TCP/IP, and more

Disadvantages of IB

• There is only one provider of an InfiniBand core chipset (Mellanox), constituting a risk of supply disruption. (and this is huge in my book, free market in effect-not good)
• This lack of diversity also generally reduces innovation and over time drives prices higher.
• There is limited expertise in the field with the protocol.
• It is a new networking protocol and relies heavily on gateway functionality (storage area network and IP) to get the full benefits of a unified fabric.
• Limited diagnostic and troubleshooting tools are available in shipping products.
• It has had limited use in enterprise environments, which is the true test for sustaining long-term viability of the protocol. (here we are talking about significant up front cost, you are essentially up fitting you entire environment with gear to support this technology)

Ok, not nearly as sexy as first discussed…how about Ethernet..

Advantages of Ethernet

• Long networking history
• Knowledge base of the protocol and tools available–come on who doesnt know how to configure a NIC..what about an HCA? —that’s what I thought
• Based on standards and available from multiple vendors

Downside of Ethernet

• RDMA offload network interface cards are new and relatively expensive
• 10 Gigabit Ethernet is still relatively expensive….and dropping as we speak
• End-to-end MPI latency may be longer in Ethernet networks (testing is still in progress)
• Other interconnects are more competitive in price performance at this time….give it time

Are we looking at both hardware and soft costs with IB when comparing price/performance per port to Ethernet? I am not so sure…this is definitely a topic worth revisting…thanks to JN for kicking up this topic

There has been a lot of news over the last several months about FCoE and its applicability in the data center. With the advent of 10GbE and the premise of data center unification I can understand why this technology has gotten legs. Joining the ranks of iSCSI, iFCP, and FCIP, FCoE offers the consumer yet another means to shove block based data traffic over an Ethernet network.  But unlike the former protocols,  FCoE is purely touted as a “Data Center” protocol, non-routable and connection oriented much like FC.

The practicality of this protocol comes into play with the use of a CNA or Converged Network Adapter, ultimately replacing the need for Ethernet NICs and FC HBAs. Less adapters equates to less cabling, less power and less complexity. Paradisiacal, huh???

Well hold on…undeniably the landscape of Ethernet will need to change to support this convergence of data and storage traffic. To further this unification, Converged Enhanced Ethernet  (or DCE, Data Center Ethernet) is under development to allow for FC to pipe over Ethernet networks.

What follows are the modifications that need to take place to Ethernet to support Fiber Channel.

• Encapsulation of native fiber channel frames within an Ethernet frame.
• Extensions to the Ethernet protocol itself to enable a lossless Ethernet fabric
• Replacing the fiber channel link with MAC addresses in a lossless Ethernet

Fiber Channel, it seems, is architecturally  dammed. With Ethernet extending within the 10G range and beyond (40G and 100G under development) how can FC ever compete? By switching transports to Ethernet is how. Protecting FC investments is the key to FCoE’s adoption and all vendors with business interests in FC know that.  Cisco, of particular interest, is leading this march. With its buy-in to Nuova Systems in 2006 and most recent buy-out in early 2008, it is well positioned to capitalize on future FC ecosystem upheavals. The Nexus 5000 series of DC class switches, released in April ‘08, is the first real step toward Unified I/O and you better believe it will support FCoE, with a few license buy-in’s of course.

With that said, the adoption of FCoE will be slow going despite its backing. Companies in general will not be quick to uproot there existing native FC environments for the “new kid” technology on the block until it is tested and proven from soup to nuts.

What’s exciting about the introduction of FCoE to the market is that it will force enhancements to Ethernet that will ultimately benefit iSCSI and network storage protocols in general, further driving the server consolidation and virtualization cause. Its proliferation will be built on the back of DCE and its adoption will be in the hands of you and me. So educate yourself, its coming…

For more info on FCoE and the developments that are happening in this space, please visit http://www.fcoe.com/

Also be sure to check out the compelling video describing the hurdles of explosive data growth at CNBC and how they will tackle their challenges with FCoE, http://video.computerworld.com/services/link/bcpid1351827287/bctid1410385428.

****Updated****

Thanks to Omar for commenting and correcting my inaccuracies, as FCoE does not require an upheaval but is complementary and non-disruptive to existing FC environments. He also noted, the Nexus 5000 is currently being certified in EMC E-Labs and should be customer implemented by year’s end. I may add that the current MDS product line does support FCoE in their third generation modules, further protecting your current investment.

Comments are always welcome.

It was always time consuming for me to decipher Cisco documentation that related to port group bandwidth and module overscription rates for their Gen-2 FC Modules. So what follows is a brief interpretation of how those ratios are determined and why they matter. Currently the modules available are 12 port, 24 port, 48 port and the 4 port 10G cards. The 12 port card offers a full 4G line rate per port, the 24 and 48 port cards share their bandwidth and operate according to the following subscription ratio’s:

So what this means is that if all ports were active on a 24 port module they would operate at the full line rate of 2Gbps. Broken down this equates to four 6 port port groups, each group capable of 12.8 Gbps. The 48 port modules consist of four 12 port port groups, each group again capable of 12.8 Gbps. That being said the full line rate of each port on a 48 port module if all ports were active would be 1Gbps. The maximum bandwidth per port per module, minus the 10G mod, is 4Gbps. So why does all this matter? Well its extremely important for host placement on those fiber switches. Ideally if you had a pleothra of modules in your director, you could seperate your traffic as follows:

12-port 1/2/4 FC Module-high perfomance servers, front end array ports, ISL’s, and tape drives/libraries

24-port 1/2/4 FC Module-server connections, front end array ports

48-port 1/2/4 FC Module-tier storage solutions (low,medium and high), server connections

4-port 10Gbps FC Module-ISL consolidation, high performance metro connections

These recommendations are highly dependent on your environment as there is nothing preventing you from putting a front end array port on a 48 port card. But being aware of what other devices exist on that module can save you trouble in the long run, not to mention it will ease you down the path of scalability. In addition, you should not only be cognizant of per module placement but also per port group placement. Remember you only have 12.8Gbps of bandwidth per port group so loading up all you front end ports, high performance servers, etc. into one port group is a recipe for a poor design.