Discussions abound the web on the ambition of FCoE and whether or not its birth is iSCSI’s demise. Yeah it makes for a great debate, a chance for industry pundits to smack the proverbial knuckles of their competitors. But is it necessary, and who can argue or foresee that either block level transport will trump the other. Additionally, where do this competing technologies fit in the grand enablement of UF or do they? For those not familiar with the advent of FCoE, check out my post Unified I/O-FCoEasy does it…Successor to FC? for a brief description of this technology and all of the Supplemental and References below.

But the main purpose of this post is mostly interest and some knowledge sling-shotting to some great intellectual literature within the community. Not to mention to raise awareness of an up and coming transport that in a few short years will exist, in some capacity, in your friendly neighborhood data center. So lets start with what FCoE is and isn’t…

What FCoE isn’t?

• An I/O virtualization technology, any doubt ask Mr. Lowe. I tend to agree with him on this, there should be no mistaking what this was developed for.
• El decapitator of iSCSI, killer it isn’t…grappler of market share? sure..
• Enabler of the unified fabric..Um no..Lossless Ethernet or Converged Enhanced Ethernet, CNA’s (Converged Network Adaptor), these are enabler’s…
• TCP based, you didn’t know? and with that, non-routable.

What FCoE is?

• Protector of FC investments
• Proliferater of 10GE, no doubt iSCSI definitely is a healthy contributor to this as well.
• Protractor of Ethernet and its convergent qualities among other things, with assistance from Ether2.0 or CEE
• Completely stateless
• Minimizer of cables, adaptors, switch ports and power, oh my..

The true Enabler’s of the Unified Fabric (2 of many)

• As I mentioned earlier the true enabler’s of a Unified fabric are CEE and CNA’s among others but without these none of this is possible.
  •  
  • CNA’s- These are your FC and IP handlers, multiple ASICs on board to address encapsulation, 10GbE, and FC. Here is a graphic representing the encap ASIC known as Menlo (check out page 17). This handles your FC frame to Ethernet frame encapsulation, host-side.

CEE-Its complete existence is to handhold FCoE’s adoption upfront. Block I/O, as you know, is not real receptive to frame loss. Hence the need for a revamping of Ethernet and its support of a Lossless network.

Other Enabler’s-PCI-E, DCBX, PFC and Pause, FCF, did I mention 10GbE, seems so obvious?

Let’s remember FCoE is an extension of the FC fabric so the following are still employed. Not only that host to storage zoning, LUN masking, and general storage related functions will still exist.

• FSPF-Fabric Shortest Path First
• RSCN-Registered State Change Notification
• Name Server Database

Some subtle port differences

1. N_Ports are now Virtual N_Ports or VN_Ports within the Ethernet world
2. F_Ports are VF_Ports
3. And E_Ports are VE_Ports
4. Addressing in the FCoE world takes on two forms:
   1. MAC addressing-Used for hop addressing from the Virtual ports above (VN_P to VF_P, VE_P to VE_P, etc)
   2. FC addressing-Used for end to end mapping of your Virtual or Non-Virtual N_Ports to a host or target port. These are your FC_IDs of the bunch.

Supplemental’s and References

• The definitive piece of literature on FCoE is “Data Center Networks and Fibre Channel over Ethernet (FCoE)” by Silvano Gai from Nuova Systems, Cisco owned now. This is where a lot of my understanding (if you could call it that) and notational referencing of the technology came from.
• Check out this Cisco video as well, with no other than Mr. Gai himself.
• FCoE overview video by Stu MiniMan at EMC, additionally here is a 10 pager in close to layman’s terms about said technology as possible. Thanks to Dave Graham for pointing this out and thanks to Stu for his great commentary.

• Speaking of Dave Graham, always thought provoking content, here are a couple of posts to contemplate.
  • Moving a Fabric forward: FCoE Adoption and other Questions
  • Fibre Channel over Ethernet or Infiniband: a Response
• And from the Grand Puba’s of Puba’s, you know who
• EMC Fibre Channel over Ethernet (FCoE),great doc on the in’s and out’s of the Nexus 5020, configuring, management and serviceability.
• Cisco Whitepaper, Fibre Channel over Ethernet Storage Networking Evolution. Discusses the noted evolution of this technology and descriptions of the phased approach to a Unified Fabric.

During the course of my experiences with Site Recovery Manager I came across some timeout issues relating to OS Heartbeat delays that make even a successful test recovery look down right error ridden. Specifically it is the portion of your recovery plan that checks for VMware tools heartbeat. This is a configurable value, and by default in SRM 1.0 was 300 seconds and in Update 1 is 600 seconds. Although any value set, decremented or incremented, changed the outcome.

So after talking to the extremely helpful, Lee Dilworth over at the blog, Uptime (VMware and Business Continuity), a couple of items were brought to my attention. One, and I don’t want to mislead you, most of the time this is more of an annoyance than anything else. It doesn’t imply that your test recovery went down in a ball of flames. But to keep these recovery steps visually accurate it is necessary in my opinion to make this change. Now that being said, it poses the question to why the OS heartbeat is not part of JUST the actual recovery. As I had thought the VMware tools heartbeat check was a network operation.

Here are the steps to curb this outcome, again thanks to Lee and please keep the great info coming our way. Link to original post, http://communities.vmware.com/message/1130385.

• Edit you hostd config.xml on all recovery site hosts, the path is, /etc/vmware/hostd/config.xml. The <vmsvc> section should look as follows:
  • <vmsvc>
    <enabled>true</enabled>
    <heartbeatDelayInSecs>40</heartbeatDelayInSecs>
    </vmsvc>
  • Note: As Lee suggests the 40 is a configurable value, 20 was the previous value prior to ESX 3.5 U3.
• Restart hostd or the ESX host, issue the following command..
  • service mgmt-vmware restart
• Reset your original heartbeat timeout values to the default, 300 seconds for 1.0, 600 seconds for Update 1.

First things first, if you are an EMC shop (or not, that’s cool you don’t have to be) and you aren’t using Recoverpoint you might as well be flying blind through the DR brume. This technology, in cahoots with SRM, makes ye failover and failback a dream specifically in regards to the storage prepatory piece. But with any replication technology there are gotchas, so with out further a do here are couple I have found..remember helping you…help yourself…

To begin, virtual image access (as opposed to physical logged image access) is a requirement if you want to test you failover plans within SRM, which is a given right? superfluous question I know. So what’s the difference…

—————Logged access (physical)—————–
Logged access rolls backwards (or forwards) to the snapshot (point in time) you select to access. There will be a delay until the system rolls to the specified snapshot. The length of delay depends on how far the selected snapshot is from the snapshot currently being distributed to storage.
Once the access is enabled, hosts on the SAN will have direct access to the replica volumes, and the RPA will not have access; that is, distribution of snapshots from the journal to storage will be paused. If you disable image access, the writes to the volume while image access was enabled will be rolled back (undone). Then distribution to storage will continue from the current snapshot forward.

————–Virtual access (instant)———————

Virtual access creates the required parts of the image you wish to access in a separate virtual LUN (or in memory). Access is very fast, as the system does not actually roll to the image in storage. You can use virtual access in the same way as logged access, however, it is not suitable if you need to run many commands or if you need data from large areas of the replica. During Virtual access, distribution from the journal to storage continues. When you disable image access, the virtual LUN and all writes to it are discarded.

So whether you purchase RP/SE or the full blown version makes no difference, each come with VIA without additional cost. Now this feature is tied to your license so if it was not included as part of your original activated license (which can happen) then you will need to contact licensekeys@emc.com to rectify. Notice the output of  the “get_account_settings” command (login with admin not boxmgmt). “Instant Recovery Support” otherwise known as virtual image access is how we need to roll..

And here is what you will see in the history tab for your tested recovery plan…

Ok that was pretty easy, here is another one, a two-fer if you will. If you receive the following error message within you test recovery plan then patch that CLARiiON splitter folks. This occurs in the recovery plan during the recovery of the protected VM’s phase. I am sure there are a number of other situations that can throw up this error so don’t take this as a postulate…

I was seeing situations were virtual image access was given per say to the ESX host but it couldn’t read or write to the LUN. Manual physical image access worked great so relationally it was just a VIA problem. After involving EMC support, they pointed me to the following primus, emc201145. Although it was not exactly my problem it did resolve my issue…this of course is an NDU patch so no worries…and most certainly implies that the RPSplitter Enabler is installed as well.

Use of Target Side Processing in virtual (virtual access) or physical (logged access) mode

Problem:
Target side CX splitter may appear down (via RP GUI)–Wasn’t a problem in my case—

Problem:
Single SP panic due to memory corruption (panic code may vary)—Again wasn’t a problem in my case—

Problem:
The following statements will be seen in the std ktrace buffer:

03:33:36.285    8791488 84570320 Assertion Failed:  (!TcdIrp->InUse)
03:33:36.285    23 84570320 at catmerge\host\Tdd\Driver\dca.cpp:211

Fix:
Escalate suspected instances of this issue to RecoverPoint Sustaining Engineering to verify before taking any action.

Workaround :

When confirmed, update to the latest version that has the issue, i.e. for CX3 that would be 03.26.xxx.5.020 and for CX4 that would be 04.28.000.5.008, and then install the appropriate type 3 patch on top of that FLARE version.

• For Release 26 (CX3) - the patch is 03.26.003.6.002 and should be installed on top of 03.26.xxx.5.020
• For Release 28 (CX4) - the patch is 04.28.006.6.001 and should be installed on top of 04.28.000.5.008

To download go to : http://www.cs.isus.emc.com/csweb2/dgweb/software/index.asp#Navisphere:

RPSplitter Engine

NOTE:  These require RPSplitter Enabler
RPSplitter Engine (CX3 Variant) 03.26.003.6.002
RPSplitter Engine (CX4 Variant) 04.28.006.6.001

Now go forth and spread the good news::::

Proper Datastore design and VM placement upfront can save you and your predecessors from self induced product failure. What do I mean? Listen up…SRM, among other things gives you the luxury of failing over or testing failover for certain applications or VM’s within your environment. By creating protection groups that point to specific VM’s that point to specific replicated Datastores  (referred to as Datastore Groups) allows for granular failover control. Albeit granular in the sense of all the VMs that exist on that Datastore, but you get my point.

I dare to say this is pretty straightforward, but introduce an environment where VM’s extend across multiple Datastores and your granular failover control goes out the door. Remember by default protection groups are mapped one to one to a Datastore group, so if a VM has multiple vmdk’s across multiple Datastores then you limit how many protection groups and ultimately how many recover plans you can create. Now this limitation is not software based..no, no these are more exciting these are self induced limitations. Case and point, lets look at this specific example from the SRM Evaluator Guide….

VM1 is contained in Replicated LUN A (Datastore Group 1) so VM1 is a protected VM. VM2 is contained on a non replicated LUN and therefore VM2 is not protected by SRM. Datastore Group 2 consists of Datastore C and Datastore DE which contains protected VMs 3, 4, and 5. It is worth noting that even though VM4 spans two Datastores, it is completely contained within Datastore Group 2 and as a result is fully protected by SRM.

In SRM an entire VM needs to be located within a Datastore group which has a one to one mapping to a protection group. Datastore groups are automatically discovered by SRM, a Datastore group is defined by the configuration of the virtual machines selected for protection by SRM.

Remember once a Datastore Group is mapped to a protection group it has now been benched, meaning you cant use that DS within any other PG. Ok that’s cool…well hold up, it’s cool when you don’t have only one Datastore Group that in essence encompasses all of your replicated Datastores. And this my friends is what can happen if you have multiple VM’s spread across multiple replicated Datastores. ONE DS GROUP = ONE PROTECTION GROUP…Failover is now all or none, you have in effect taken a product with endless possibilities and made it a shoe horn..

Policy driven information storage and distribution. EMC Atmos is a multi-petabyte offering for information storage and distribution. It combines massive scalability with automated data placement to deliver content efficiently anywhere in the world.

That my friends is where we are headed, globally available, policy focused storage, sick..I love this meaningful example that Steve Todd gives, this put things in perspective for me….>this is more for me than you I need a place to tie this content together, sorry if there isn’t proper flow here<

The Atmos architecture also allows for extensible policies. Customers that want to specify policies outside of those natively offered by Atmos can develop their own.  For example, picture a customer that wants to add “Cheap Power” as a policy; Atmos can be programmed to globally move content to a location with the cheapest power rates.

Additionally here is some other delightful tacit knowledge from King Hollis…

So, as a starting point, let’s imagine a global object repository, formed by multiple storage nodes scattered around the Internet, all seen as a logical whole. Not really a file system in the traditional sense, although it could be presented as one if needed.

Applications load content from anywhere. They’re now part of the ‘global object pool’. When you load the content, you specify a policy, such as “gold” or “free” or “secure” or “pay per view” or “we think this is gonna be really popular” or “keep a certain minimum number of copies around for redundancy purposes”.

That policy specification is dynamically interpreted by the Atmos environment. If something gets very popular, and access times elongate, Atmos can make multiple copies based on where the interest is coming from. And when the demand storm subsides, go back to a more cost-effective scheme.

So Atmos is capable of predictive and real time traffic analysis and can route or replicate content accordingly based on geo-presence or pre-configured policies attached to this content, am I under standing that right?

Lets look at the hardware behind this offering, Dave Graham has done all of the leg work for me so lets lay it out..

Hardware-Atmos.

Compute WS1-120 configuration: 8 servers plus 8 disk enclosures Server to disk ratio: 1:15 120X 1 TB SATA II drives = 120 TB (15 per disk enclosure)

Compute WS1-240 configuration: 16 servers plus 16 disk enclosures Server to disk ratio: 1:15 240 x 1 TB SATA II drives = 240 TB (15 per disk enclosure)

Capacity WS1–360 configuration: 6 servers plus 24 disk enclosures Server to disk ratio: 1:60 360 x 1 TB SATA II drives = 360 TB total capacity

 Additionally here are the hardware configurations at length

 
<<<Yawwnnn>>>Dell-ishis commodity servers, been there done that ala Recoverpoint, so what..I agree with Dave hardware is hardware.

Software-Atmos.

Being the building block for  COS or Cloud Optimized Storage, Atmos has the propensity to take on many forms. Is this a natural storage evolution as Todd suggests? I think so, I would even go as far as to say that COS has broken the traditional mold and long line of relatives and formed Storage 2.0. But whatever form it does take Atmos is assured to be at the head of the class, but aren’t all EMC’s products : ) <Bias Alert>

Features as noted by ESG..

• Object metadata: Metadata, a core cloud service enabler, allows users to tag content and use those data tags to apply policies, improve searches, or build custom queries.
• Policy-based information management: Policies leverage metadata to determine where and how Atmos stores data so it can meet information performance and availability requirements, such as how data is protected, for how long, with how many copies, and where it can be stored geographically to best meet performance/availability requirements.
• Service-oriented: Services can be applied to data—based on metadata and policy—such as replication, versioning, compression, deduplication, and spin-down.
• Unified namespace: A unified namespace enables Atmos to scale massively while still operating as a single entity. Massively scalable: Atmos can theoretically scale performance and capacity infinitely within the unified namespace.
• Globally distributed: Atmos uses a distributed object architecture that places data in the cloud closest geographically to where it is required.
• Operationally efficient: auto-healing, auto-managing, auto configuring.
• Multiprotocol support: Supports web services via REST and SOAP, as well as legacy protocols such as CIFS, NFS, and IFS.
• Multi-tenancy: Serves multiple applications from the same infrastructure. Each application is securely fenced off and cannot access another application’s data.

This niche play by EMC is no doubt focused on media and entertainment companies, telecoms, Web 2.0 and Internet providers. Now I won’t even pretend to know what all Atmos brings to the table but just the shear introduction of it to the market has peeked a lot of interest in the direction of COS, the support of the storage vendors and where their interest lie going forth.