What on earth does Jeff actually do?!

I have been wanting to sit down for a while now and write about my job and what exactly I do for forty hours each week, mainly because I feel so bad for my wife and family whenever they get popped the question and have to fumble around to try to explain it.  Even I have a hard time trying to describe it to people when asked, and usually have a few different explanations based on the technical understanding of the person asking the question.  Some of the responses I'll often give are....

• "I work in the IT industry."
• "I work in maintaining data centers."
• "I work with 'The Cloud' and data storage."
• "I am a tech support engineer."
• "I support Network-Attached Storage arrays."

That's just a small sample of the different replies I give to people who want to know what I do for a living.  But in reality, my job is pretty complex to explain.  In fact, even my entire title is an acronym:  FAST VP SME, which stands for Fully Automated Storage Tiering for Virtual Pools Subject Matter Expert.

That's certainly a mouthful, so I'll try to start at the beginning.

I work at EMC Corporation in Draper, which is an IT company that is most widely known for its storage arrays and its acquisitions of VMware and RSA Security.  Not being able to afford multi-million dollar computer equipment, I'd never really heard of EMC in terms of their data storage systems, but knew who they were because of VMware and RSA, which I was exposed to quite a bit while studying at UVU.

If there was one word I could use to describe EMC, it would be "HUGE!"  Especially after coming from a small business of maybe 80 employees, I was incredibly shocked to be surrounded by several hundred employees just in Utah, as well as hundreds if not thousands more worldwide, all supporting a large variety of products.

I was hired on as a Technical Support Engineer for the Symmetrix department.  I don't usually like to refer to myself as being in "tech support" because of the call-center stigma that is associated with it, because even though I work with customers in resolving break/fix issues, I think the term "technical support" doesn't do my role very much justice.

But I'm getting ahead of myself.  I think I should first explain a little bit about what this Symmetrix thing is.  The Symmetrix storage array is the first product created by EMC, and is what is called a Network-Attached Storage (NAS) array.  These arrays look like giant refrigerators that line up beside each other like a football team on the scrimmage line, and are filled with hundreds or thousands of hard drives.  

Why on earth would someone need so many drives?  Well, just think about it.  If you're reading this post, then chances are you have a Facebook account, unless somehow I'm way more famous that I think I am...which is not very.  How many photos and videos would you say you've uploaded to your account?  Hundreds?  Thousands?  Ok, now ask yourself how many users exist on Facebook right now?  I sparked my own interested just now and Googled the answer, and it turns out that Facebook has about 1.19 BILLION active users as of October, 2013.  So multiply that number by the number of photos and videos you have on just your account and see what you get.  I'm not sure my calculator even goes that high.

Well, all those pictures have to be saved somewhere, and that's a LOT of storage space.  That's where storage arrays like the Symmetrix come into play.  These powerful machines can hold petabytes and petabytes of data, and make sure that it's ready and accessible every time someone needs to use it.  These arrays have loads of features to make sure that the data remains intact no matter what by having several backup power supplies, spare replacement hard drives that will kick into gear the moment one starts to fail, and even the ability to have a mirrored system set up hundreds of miles away just in case a major disaster occurred in the area like an earthquake or tornado and the primary data center went offline.

I was trained to support these arrays for all of our customers (which are pretty much all Fortune 500 companies) and make sure that there are no problems accessing their data at any time and at the speed that they want.

The Symmetrix has a really interesting system in place where it actually complains automatically to EMC whenever it's having any problems.  This "call home" feature allows the array to automatically dial home to an EMC server with any issues, which in turn automatically opens a support ticket for my team to investigate.  These "dial home errors" that dial home arrive in the form of a four-digit hexadecimal number with a two-digit modifer number which refers to a specific error.  Some examples of these errors are 24AF.78, 01B3.C0, 0471.02, and 033E.38.

This was probably one of the hardest parts of the job, learning how to decipher these codes to know what they are referring to.  One may refer to a failed drive that needs to be replaced, whereas one may refer to a backup power supply that won't hold a charge, and another could indicate that a cable was plugged into the wrong jack.  I look at the list of errors now and am still surprised when I know what they mean, when they looked completely Greek to me months before.

This is why I say that we are not the "traditional" type of technical support center, because in our case we will receive a report of an issue and have it corrected or a local representative onsite with a replacement part before the customer even knew there was a problem in the first place.

The way we do this is by remotely accessing a laptop built into one of the arrays, which is called the service processor.  The same way that I'll patch into my dad's computer to help him get rid of a virus or install a program, I can connect to an array in Sweden or Australia or anywhere else and remotely troubleshoot issues.

My primary role when I was hired was to investigate and resolve these dial home errors, which I performed until I was enrolled in a two-week course taught by a colleague from our Hopkinton, MA, office.  James Nigrelli was a Technical Support Engineer (TSE) like I was, but he had been flown out to Utah to teach me--along with a small group of people--about a new feature in our top-of-the-line Symmetrix VMAX arrays called FAST VP, or Fully Automated Storage Tiering for Virtual Pools.

This feature is surprisingly exactly what it sounds like, although it probably makes no sense to anyone reading this at the moment.  But in a nutshell, here's how it works.  There are essentially three types of drive technologies that make up a storage array.  The first type is called SATA, which is the hard disk that most of us are used to, with the spinning metal plate touched by a spindle used to read and write data, making it essentially look like a tiny vinyl player if you were able to see inside its chassis.  Next comes the Fibre Channel (FC) drive type, which is essentially a SATA drive but with a much faster, optical connection and a faster speed.  Third is the Enterprise Flash Drive (EFD) type, which is an Enterprise version of the Flash or Solid State technology, which is what you'd find in a USB thumb drive, your iPhone, or even some computers like the Macbook Air.

Obviously the faster, better drives are going to be more expensive than the slower ones, and customers want to make sure they're getting their money's worth out of them.  FAST VP is a completely automatic system built into the VMAX arrays that collect performance statistics on the data and then uses those statistics to make sure the data that is "hot" and being used all the time is being kept on the faster drives so they can be accessed quickly and that the "cold" data that isn't being used very much stays on the slower, less-expensive drives.

Did I lose you?  Well, then think of if this way.  Let's imagine you are a fashion extraordinaire and have a pretty extensive wardrobe, but just moved into a small apartment with only one closet.  With all of the clothes you have, you're going to need to keep them in three different locations: your closet, the garage, and at your parents' house.  Naturally you're going to want the clothes you wear all the time in the closet, the clothes you might wear every now and again in the garage, and the clothes you own but really should have been given to Goodwill a long time ago at your parents' house.  It's the same thing with FAST VP, but with data.

I really enjoyed learning about FAST VP and, after James returned home, I soon became the "go-to guy" for any FAST VP-related cases that were opened.  I had such an aptitude for it that, one year ago, my manager approached me and informed me that he wanted to send me out to the Hopkinton lab with another TSE named Richard Kimball (no relation to Harrison Ford) to study FAST VP "from the masters" in order to become a Subject Matter Expert.

The trip was incredible, but also difficult because it was one week before my wedding.  But I learned a ton, studying alongside James and his mentor Rob Tasker.  They taught me so much, and I was able to experience the busy life of those in Boston for a while, which actually made me pretty grateful that the Utah lab is a bit more mellow.  I also had the opportunity to meet the Symmetrix developers, including the programmer who invented FAST VP, who shockingly is actually a pretty young guy.

Upon returning from my trip, I was bestowed the title of FAST VP SME, or FAST SME for short.  Richard and I then became the first escalation point for Utah TSEs before having to escalate to the Hopkinton lab for help with FAST VP-related issues.  

A few months later, James announced that he had taken a job in the development team and would be leaving the support lab.  Shortly after that, Rob announced that he was being promoted to the L2 Support Team, which is the elite escalation team for all cases.  While he would still be a resource for troubleshooting FAST cases, I soon found myself to be the primary escalation point on a global scale, as I have a Monday through Friday shift and have more of a full exposure to the feature than Richard, who works on a rotating shift and will often spend up to a week at a time away from the lab.

Being one of two FAST SMEs on shift globally during the US business hours, I had a lot of responsibilities on my hands.  Not only was I expected to take all the FAST-related cases that appeared in the support queues and act as an escalation point to others, but I also participated in weekly and bi-weekly conference calls with the developers (referred to as Engineering) that I met in Hopkinton to discuss current and new issues that affected the feature, participating in a group called the FAST Task Force.

I have truly learned a lot while in this position.  Not only have I had to stretch my technical understanding to great lengths in order to understand the abstract concepts behind the product I support, but I have also had to hone my customer-facing skills as I have grown accustomed to sending dozens of emails each day and participating in several impromptu conference calls throughout the week.  I also became very adept at writing knowledgebase articles and other technical documents to help my colleagues in diagnosing various issues relating to FAST VP and its counterpart technology known as Virtual Provisioning.  (I'm not even going to attempt to explain this concept, as it's much more complex than even FAST VP!)

This job has been really wonderful, and yet very stressful at the same time.  More than occasionally would I go twelve hours without eating anything because I had to work through my lunch, and would come home with throbbing headaches from swiveling between my four workstations.  But on the whole, I really loved my job and felt like I lived up to expectations.

You may have noticed that I have been using the past tense when describing my job.  That is because I announced today to my team that I have taken on a new role as a TSE II in the RSA Security department, supporting the Security Analytics (also known as the NetWitness) platform, which is a tool for identifying and putting a stop to hacker infiltration attempts.

As I obtained my degree in IT Security, this has been pretty much what I would call my "dream job."  I always wanted to work in the IT Security field, keeping hackers at bay and fighting the virtual battle on the cutting-edge of technology.  I now hold a position in one of the leading IT Security companies worldwide.

This role is going to be much different than supporting the Symmetrix storage arrays, as I will be forgetting about hexadecimal error codes and proprietary software commands, and will instead be returning to my Linux roots, which I couldn't be more excited for.  I'm also being sent on a two-week business trip to Washington DC and Virginia in a few weeks for extensive onsite training to learn more about the product that I'll be supporting.

To say that I am excited would be an understatement.  This job opportunity also came at the perfect time, as Whitney and I will be having our first child at the end of March, and have had a lot to do in order to prepare for his arrival.  It really was a blessing that I discovered the offer when I did, so that I could take advantage of it and make this huge change in my life.

While I'm afraid of leaving the Symmetrix team, especially as I will be putting the burden of my role onto others' shoulders, I definitely feel like this is the right move for me at this time, and that I am doing what is in the best interest of my growing family.  But I am also so relieved and grateful that it includes starting a role in the job I've always dreamed of having.

Here's to the future!