Battle Ponderings - A Year in Review 2012

For the last two years I like to recap the year with events that have been instrumental in Varrow’s  success. Although there are many, I document only what comes to me in my stream of consciousness (mostly recapping Varrow.com). Hope you enjoy, and have time to reflect on this whirling dervish we call Varrow…Here’s to a successful, and customer focused 2013!

Varrow Madness 2012 – Certainly the largest partner event in the Carolinas, #vm never disappoints. Labs, sessions, keynotes, beer and basketball, what’s not to like? Whats got me most excited about this year is more engineering participation for the sessions. I like to think ToT was a catalyst for this but maybe its the new blood pushing the old blood : )

Company Support of Home Labs – As eddy would say from Christmas Vacation, this truly is the gift that keeps on giving. It’s a fact engineers are naturally more hands on, to this point home labs have excelled the learning process here at Varrow. Investing in training has been a huge focus this year, and it shows. Engineers are more confident, customers are more comfortable, trust is built, lifers they become. Its that simple.

More vExperts added – VMware vExperts that is, and its VMware’s way of showing support for those in the community that contribute, promote and evangelize their products. Jason Nash and I were the lone rangers for the last two years but 2012 introduced Martin Valencia, Andrew Miller and Phillip Jones (by acquisition) to the list. Remember, anyone can apply, you can nominate yourself! Just contribute and be prepared to document. 2013 nominations will be here before you know it…

TrainSignal Exposure – I don’t know how he has time for it but Nash has produced both the vSphere Advanced Network Training and the soon to be released vCenter Operations Manager course over the last year. The latter of which I served as tech reviewer. Not only do you get to experience Jason’s witty knack for teaching but both are jammed pack with useful, practical and detailed technical goodies. Huge exposure for Jason and Varrow!

VMware View focused partner of the Year – For one that delivered the first View implementations at Varrow, I couldn’t be happy about this. VMware has provided funds to develop and grow our View business. For many years we were on the fence as to whether or not VDI would be core to growing our business. With these funds, the right leadership and engineers in place, we’ve shown that VDI indeed is a pillar of this organization.

EMC Velocity Services Quality Award again – This award from EMC is based on what every services organization should strive for, Customer Satisfaction. We’ve accomplished this year over year. Blood, sweat and tears, it ain’t easy but its what drives us to do better.

Top Storage Practice in Nation so says CRN and me : ) - I’m not even sure what this really means or what its based on, but CRN gave us an award for it. I always knew this, I’m glad others have recognized the overwhelming talent in this group.

4 Presenters at VMworld 2012 - Brian Boyd, Jason Nash, Greg Camp and Myself, if you didn’t know. This was a big deal for me, really pushing my comfort zone. But I learned a lot and realized the obvious. Public speaking is easy if you practice, have a mastery of your subject, and memorize your starting and ending statements. Everything in between is all you. And that is what everyone comes to see and listen to, you!

Inc 500 awarded number 447, CRN’s Fast Growth 100, #1 in Triad FAST 50 again, and CRN’s 2012 List of Tech Elite 250 – Yep there are more, these are but a few. These will come and go and ultimately are a byproduct of our rocket growth. At days end, the service quality awards are what gives our foundation.

2 office moves – GSO and CLT have moved entirely and the Raleigh office expanded. I, being 100% remote have not seen any of the spaces in person, but I understand this was a much needed expansion. Varrow has dressed up and taking it to a new level. I’m so proud of what we have built.

10’s of new faces – I’m not even sure how many Varrow-ites (39 it seems) we have added this year, but just remember I’m number 5 : ) And if I haven’t said it, Welcome!

Teachers of Tech (ToT) introduced – Really glad this is getting traction in 2012. My suspicions are that we were all thinking about this avenue. Sometimes it just takes someone to own it. I for one, look forward to many great sessions over 2013. For those outsiders, ToT is a weekly internal webinar on a particular process or technology that the engineers host. You simply sign up for a slot and speak for an hour on whatever “technical” topic you want. Their recorded and posted on online for those that can’t attend. We hope to expand this externally, so look for more info to come…

I can only imagine what we’ll accomplish together next year. 2012 was big, but 2013 will be bigger…

The 10,000 Year Clock

In 01989 Danny Hillis dreamed the flight of the 10,000 year clock. Seen as a symbol for long term thinking, it sparked the question, Are we being the best ancestral beings we can be?  With this purpose in mind, Danny’s intent was “..to build a clock that ticks once a year.The century hand advances once every 100 years and the cuckoo comes out on the millennium..” Dissatisfied with society’s short term attention span, Danny with the help of Brian Eno, established The Long Now Foundation dedicated to seeding a “..long-term cultural institution”. The clock and their other symbols of innovation set out to promote what was once a passion of us humans. The future. 

Approximately 200 feet in size, the clock itself is imbedded in the bowels of a mountain near Van Horn, TX. Funded both geographically (he owns the land) and monetarily, Jeff Bezo’s (Co-founder and CEO of Amazon) has also played an active role in seeing this come to light. The clock itself is designed to conserve energy, updating the face only when visitors, “wind” it via a winding station. Sunlight allowed in from the top of the mountain heats a chamber of air, powering a cylinder that keeps the pendulum moving. Ceramics, geneva wheels, dials, governors and titanium are among some of the other bits and pieces that encompass this clock. All were intimately architected and engineered with longevity in mind. 

Besides my general interest in this project of which you can explore more here, it made me think how little I dream or ponder beyond say the next 100 years (ie. my lifetime and my kids lifetime). Perhaps its a sense of helplessness that my actions today won’t have any bearing on a world 10,000 years from now. And besides what do I care, I’ll be gone. But I argue we should care. The fact that most of us brought children into this world implies that we do in fact care. So my question to you is what are you doing now with the future in mind? What are you doing to promote and encourage long term thinking?

vCOPs - On Data Pruning and Kahadb

A couple of issues I ran into this week with vCOPs that are worth mentioning. One has to do with the infamous grey question mark. This replaces a badge when data collection is broke. The other has to do with pruning data collection in an environment where you are licensed for less per VM than what is being managed by a single vCenter instance. Yes a bit long winded, but let me explain…

First off the most noticeable tip off to vCOPs not collecting data is the grey question mark. In times of normalcy, each badge should be colored with a number indicating, visually and numerically, that it’s reading and making sense of the data. If it’s not there are a couple things that could contribute to this. First, disk space on the analytics VM, lack of..breaks ActiveMQ. VMware uses conservative guidelines as follows in regards to disk sizing, but keep in mind you can add disk space to the Analytics VM, albeit with a shutdown, at anytime.

A bit of background, ActiveMQ is an open source message broker that writes to a database called kahadb. Here it stores and forwards to other services. If the service can’t accept the messages, it writes it to this database. If the disk underflowth ActiveMQ is unable to write a complete file, there in potentially corrupting the database. To resolve this you can follow the steps in VMware KB 2013266. The process is pretty straight forward which haves you rename the kahadb and create a new one. However, it doesn’t state which default account to use to do so. I mistakenly used root (as opposed to admin) to make the changes which prevented the ActiveMQ service (which uses admin) from writing to the kahadb. Simply changing the permissions ( chown -R admin:admin /data/activemq/kahadb/)on the database resolved the issue.

Secondarily, unexpected shutdowns. As with any vApp make sure to shutdown, or restart the VM’s within the vApp at the vApp level. This will save you a lot of trouble down the line and keep that kahadb in working order.

Finally, data collection pruning is something I have heard numerous times from customers. How do I only report on 100 VM’s (because that is all I am licensed for) even though my vCenter instance see’s 300? Very simple. vCOPs collects data from vCenter using the vCenter enterprise adapter. Using the vCOPs admin UI, you establish this pull communication by registering vCOPs with your vCenter instance(s). Within this setup it asks for the IP address, Display name and the registration user for the vCenter plug-in. An optional setting is the collection user. This is the account it uses, based on its visibility within vCenter, to pull metrics. By default it will use the registration user. If you want to only collect on one datacenter or one cluster, for example, setup your collection user with just visibility to that datacenter or that cluster within vCenter. Perhaps in future versions this will be a little more stream lined, doable within the vSphere UI. But for now it’s the recommended way of pruning collections. Visit KB 1036195 for more information.

vCOPs - On Data Collection..

Understanding data collection in vCOPs tends to be a mind bender. There are multiple associated terms that are used to illustrate this process. Terms like attributes, metrics, super metrics, thresholds, and KPI’s is enough to make you question your own self worth. But with a few helpful explanations maybe we can break new ground together…

First off we all know what resources are in a data center environment. These are items such as datastores, VM’s, vSS’s, vDS’s, vCPU’s, pCPU’s, etc. They are items we consume or that have value. Data that is ingested within vCOPs has an associated attribute(s) for individual resources. By default vCOPs groups together multiple attributes for resources into attribute packages. By doing so you are telling vCOPs to collect only these attributes for this resource.

A metric is a point in time instance of an attribute. If a single metric doesn’t clearly convey what you are looking for, then perhaps a super metric would. A super metric takes this metric, that metric and that other metric applies a mathematical operation (formula) to give you a broader more scalable metric. For instance, if you wanted to capture the average CPU utilization across 50 web servers you would use a super metric to do so.

Thresholds mark the boundary between what’s normal and what’s abnormal for a single metric or super metric. When either boundary is crossed an anomaly is logged. Both hard and dynamic thresholds exist, with the latter being the default. Dynamic thresholds are new to vCOPS and are based on incoming and historical data. It formulates a pattern of normalcy, there in decreasing the number of alerts that are generated. Hard thresholds model themselves after vCenter thresholds. Static in nature, changing only when you change it.

Finally, KPI’s or Key Performance Indicators are attributes you deem important. By defining an upper and/or lower threshold violation on a attribute or super metric package, you are in essence establishing a KPI for those attributes. Once an attribute is defined as KPI, new rules are set forth in motion. For instance, alerting is treated differently than non-KPI attributes. But more importantly a feature called predictive alerting is utilized. Once a violation occurs on a KPI for a application or tier, vCOPs examines the events prior to the violation and marks that as a fingerprint. If it finds similar events in the future it can alert you prior to the thresholds being broken. How sick is that!

Here is an evolving graphic to illustrate the relationships. It’s not pretty, sorry…More to come… 

Deploying an Active/Active Datacenter with VMware SRM 5 - VMworld 2012 Session Voting

There is nothing better than customer testimonials when it comes to preaching the power of cutting edge technology. So when Michael Bailess from American National Bank came to me with interest in presenting at VMworld 2012, I realized we had something special. Almost a year ago now Mike and I engaged on a journey to make their data center portable across their two locations. The business was looking to move workloads between sites at any given time but didn’t need the cost and complexity of a true active-active, “always up” distributed data centers. Down time was acceptable (off hours of course) to gracefully shutdown his almost 100% virtualized environment at Site A and move bits and pieces or all to Site B. So a few things came out of this design that are notable…

  1. The ability to constantly test their DR plan. This is in affect a rolling DR plan, weekly and monthly. Simply not feasible without technologies like VMware Site Recovery Manager, EMC RecoverPoint and VNX. 
  2. For those company’s that can withstand minimal service interruptions, moving a VM from one site to another is as easy as a reboot. Not every environment needs vMotion capabilities between sites. Although sexy, you have other options, American National Bank is proof of that. Everything comes at a cost, ask yourself what is acceptable down time (if any) based on your business’s defined RPO’s and RTO’s. VM/DC portability is and has been here for years and its more affordable than you think. 
  3. Stretched layer 2 between both sites was accomplished with common Cisco Networking Technologies. That’s it. There was not one Nexus 7K on the PO, I promise : )
  4. VMware SRM 5 is truly leaps and bounds over the 1.0 days and the spitfire behind this design. With the Planned Migration Recovery option data is safely synced, consistent and available before any VM’s are moved. This ensure’s your VM’s come up as they would during a normal reboot. 

Interested? Of course you are. AMNB is continuing a new trend of simplicity and elegance. Through this engagement we have formed a great partnership and assimilated a simple highly functional design. The ability for AMNB to protect their business and properly utilize their resources, is what all companies should strive for. So please vote for our session below, Mike and I truly want to provide you with the right information to protect your business. We have done the groundwork so you don’t have to. Thanks for your vote!

Session:  #1883 –  Deploying an Active/Active Datacenter with SRM 5

Abstract:  In this session we will discuss the challenges that face a single physical datacenter as well as how these challenges can be resolved with SRM 5. We will explain the design that we deployed at American National Bank which did not include more expensive technologies that would have been needed to run a stretched cluster environment. American National Bank has implemented their own private cloud that is not linked to any physical site. This solution saved American National Bank from long late night maintenance windows to correct facility issues, spread the load between to geographic regions, and created a DR plan that can be fully tested daily.

Other sessions brought to you by Varrow brethren are as follows, voting is key so show your support!

Brian Boyd (blog:  http://www.thesangeek.com) submitted:

Session: #2239 Deciphering the Mystical World of Storage Performance

Abstract:  An introduction to SAN and NAS attached volume types within VMware. The purpose of this presentation is to help VMware and Storage administrators that are new to the virtualization scene to appreciate the implications of RAID types, drive performance, throughput and bandwidth in an easy to understand and fun way.

This session will break down the differences between Network (iSCSI, NFS), and Fiber Channel storage but not be vendor specific. In addition the usage of certain drive types, what they are good for and what they are not good for will be discussed. Example production environments will be shown to include VM and Storage as a Service, archival, and application delivery.

Tom Cornwell (blog:  http://blog.piratesjade.com/) submitted:

Session: #2848 Cloud Continuity: How Does the Cloud Fit into Your Business Continuity Plan?

Abstract:  Today cloud is becoming ubiquitous. However, it can be utilized in many different was. Whether you utilize cloud for your production environments or not, cloud can play a part in your business continuity plan. This session describes different business continuity strategies utilizing cloud. Topics include: cloud as a source, cloud as a target, and cloud to cloud DR. In addition, we will discuss mixed strategies and different types of cloud implementations including Platform-as-a-Service, Infrastructure-as-a-Service, Storage-as-a-Service, and DR-as-a-Service.

Jason Nash (blog: http://jasonnash.com/) submitted:

Session: #2181 – Cisco Nexus 1000v: Architecture, Deployment, and Management

Abstract:  This session will walk attendees through the architecture, deployment, and management of the Cisco Nexus 1000v virtual distributed switch. The information in this session is based on experience with numerous production deployments of the Nexus 1000v and the audience will benefit from many lessons learned from the field. The session begins with a high-level overview of the 1000v’s design and components followed by several considerations and preparations that anyone looking to deploy this new distributed switch should consider. This includes items such as network design, existing network infrastructure, and overall integration in to the organization’s IT processes. Next, the focus will shift to technical details of the configuration process and several examples will be shown covering many of the common scenarios seen in the field. Day-to-day operations and management of the switch will be covered in detail and include routine items such as adds, moves, and changes, as well as upgrades and maintenance. Finally, a section on troubleshooting processes and information will give attendees the tools they need to support their new virtual switch.

Session:  #2197 - A Deep Dive on Virtual Distributed Switching & Cisco Nexus 1000v

Abstract:  This session will provide an in-depth look at the distributed virtual switching technologies available in VMware vSphere. The discussion will start with an overview of both the integrated Distributed Virtual Switch as well as Cisco’s Nexus 1000v. We will compare and contrast the options highlighting features, functionality, management, complexity, and operational considerations. Each available option provides its own set of features, functions, challenges, and design and deployment considerations. During the session the components, design and implementation considerations, as well as troubleshooting recommendations will be covered in depth. Attendees should expect to walk away with the knowledge they need to decide which of these technologies fit the requirements for their environment as well as the understanding to deploy them.

Session:  #2207 - vSphere Distributed Switch – Technical Deep Dive

Abstract:  While the vSphere Distributed Switch (vDS) has been around since vSphere 4, vSphere 4.1 and 5.0 have added a number of enhancements. This session will provide a technical deep dive in to the vSphere Distributed Switch. This includes design and deployment considerations, configuration, migration steps, tuning, and troubleshooting. Special attention will be paid to migrating an existing production environment from the standard vSwitch to the vDS with no or very minimal disruption. Extended features such as Network I/O Control (NIOC), Network Resource Pools, and Load-Based Teaming (LBT) will be discussed in depth with use cases and recommendations given. Finally, methods and tools for troubleshooting network connectivity and performance problems will also be highlighted. The inclusion of accessing a live lab environment will make for a very interactive session.

Session:  #2463 - vSphere Physical Connectivity – Deep Dive & Best Practices

Abstract:  This session will provide an in-depth look at your options for physically connecting vSphere hosts to the network. The discussion will center around common question areas that come up during knowledge workshops and customer design sessions. Throughout the session videos and animations will be used to help attendees easily see the expected result from many of these configuration options. The presentation will focus heavily on the different hashing types and traffic control, especially the more advanced options such as Load-Based Teaming and Network I/O Control. Other areas of focus include physical separation of traffic, networks of differing security requirements such as DMZs, and suggested NIC configurations for both 1Gb and 10Gb environments. Finally, recommendations for physical switch configurations will also be covered. Throughout the session best practices, recommendations, and lessons learned from many production deployments will be shared. Attendees should expect to walk away with a deep understanding of the physical connectivity options available with vSphere, how they can be utilized in their environment, and the best methods for deploying them.