blog.virtualtacit.com

As a follow up to the post, Unified I/O-FCoEasy does it….Successor to FC?, I thought it would be fair to drop some info on Infiniband and its place in the storage ecosystem.

IB from a raw throughput perspective is no doubt head of the class. Low latency, high bandwidth, and the ability to collapse both FC and Ethernet on a single transport are all contributing factors to IB’s push toward the de facto standard for high speed I/O interconnects. Although most notably a main staple in HPC (High Performance Computing) environments it quickly is targeting its efforts toward virtualization. And why shouldn’t it, the inevitable build up or build out of pure physical cabling to support even the most simplest designs makes this consolidated technology a perfect fit. VMware has already announced support for Mellanox HCA’s in its ESX 3.5 release, signaling that the tides are now split between IB and 10GigE as viable Unified options–but wait a minute, lets look under the covers shall we..

Comparison of both competing technologies above– (notice FCoE and IBoE are not listed but both are defining technologies in development)

So where do we stand from both IB’s perspective and Ethernet as a transport; the good, the bad, the ugly–noted here–http://www.nowmicro.com/NM_PDF/wp-cisco-interconnects.pdf

Advantages of IB

• Reduced CPU and memory overheads by utilizing specialized HCA hardware
• Remote Direct Memory Access (RDMA) allows for the placement of information from one computer system into the memory of another computer
  system, reducing latency and minimizing the processing overhead demands at the host system
• Low end-to-end system latency (from 5 to 10 microseconds, depending on the application)
• Multiple vendors supporting and shipping products
• Consolidated I/O with network, management, and storage all on one interface
• Implements various protocols, such as general-purpose remote-procedure call (RPC), direct-access transports (as part of RDMA), sockets direct protocol for TCP/IP, and more

Disadvantages of IB

• There is only one provider of an InfiniBand core chipset (Mellanox), constituting a risk of supply disruption. (and this is huge in my book, free market in effect-not good)
• This lack of diversity also generally reduces innovation and over time drives prices higher.
• There is limited expertise in the field with the protocol.
• It is a new networking protocol and relies heavily on gateway functionality (storage area network and IP) to get the full benefits of a unified fabric.
• Limited diagnostic and troubleshooting tools are available in shipping products.
• It has had limited use in enterprise environments, which is the true test for sustaining long-term viability of the protocol. (here we are talking about significant up front cost, you are essentially up fitting you entire environment with gear to support this technology)

Ok, not nearly as sexy as first discussed…how about Ethernet..

Advantages of Ethernet

• Long networking history
• Knowledge base of the protocol and tools available–come on who doesnt know how to configure a NIC..what about an HCA? —that’s what I thought
• Based on standards and available from multiple vendors

Downside of Ethernet

• RDMA offload network interface cards are new and relatively expensive
• 10 Gigabit Ethernet is still relatively expensive….and dropping as we speak
• End-to-end MPI latency may be longer in Ethernet networks (testing is still in progress)
• Other interconnects are more competitive in price performance at this time….give it time

Are we looking at both hardware and soft costs with IB when comparing price/performance per port to Ethernet? I am not so sure…this is definitely a topic worth revisting…thanks to JN for kicking up this topic