rss feedWe Don’t Want to Leave You in the Dark
…like our night shift system administrators were on March 30th.
I should probably start this post by introducing you to our VP of Systems Operations, Rick. This is a person who is not only passionate about his job, but also highly intelligent and creative.
OK, enough of the positive. He’s a little crazy. His insanity does happen to be almost wholly attached to the safety of the data center (he’s a really cool guy otherwise), but that insanity is there and it is something worth poking fun at.
Super, High-Tech Tea and Coffee Station
Danger: Tea and Coffee May Cost You More Here Than it Does at Starbucks
For example, when our team decided to create a coffee and tea station in the kitchen (it’s not even as elaborate as I just made it sound…see photo), Rick immediately felt the need to establish regulations. We are not allowed to leave the “station” until whatever process we initiated is complete, and then when it is complete, we must unplug every surrounding appliance. If we use the microwave, we must watch it like a hawk—watch it like that microwave could steal our livelihood at any moment. You know why? Because it can. At least, that’s what Rick thinks. Any of these appliances can start a fire, which could spread if our fire-suppression system doesn’t kick in, which could enable that fire to somehow reach the servers, which could melt our customer data, which could make our customers very angry with us, which would make us very angry with ourselves, which could sink our business.
All for a plate of bland, re-heated Chinese food.
If Rick is this way about appliances that lie hundreds of yards away from the actual servers, imagine how he careful he is about the servers themselves. Our data center is about as redundant as they come—millions of dollars went into making it this way.
Well, in the middle of the night on March the 30th, a power line went down in our corporate park. All buildings in the complex (from one end of Dividend Dr to the other) lost power. No problem, right? We have backup generators, backup batteries…we even have the energy Rick’s undying concern for the data center would provide if we could only find a way to harness it. Unfortunately, one branch of our backup setup failed (namely, a generator) and a good percentage of the machinery we need for operations was left powerless (namely, our network). Fortunately, our network was back up fairly rapidly (within minutes) and it didn’t take long for the servers to catch-up.
For those of you that experienced outages, however short, we are unbelievably sorry. We were able to quickly return to status quo, but many of our customers still experienced downtime and we know that this is unacceptable. We are really, really sorry.
The issue that we had with the power loss is being investigated with the utmost priority because we don’t want anything like this happening again… and neither does Rick. I came late onto the scene (at about 2 AM, when the power had been long-restored and the servers were coming back up from their temporary, power-starvation induced sleep), but I knew Rick and was still inside because his car was parked parallel to the building, nearly on the sidewalk entryway and I could still hear the echo of his tires screeching against pavement. When I got inside, the root of the problem had been taken care of, but we had another problem. Our main System Administrator/Network Engineer had clearly arrived straight out of bed, wearing MC Hammer pants… and he was doing so without shame.
While this event that took place never should have happened, we are going through great pains to make sure that it doesn’t happen again. We, again, are terribly sorry for any downtime that was caused by this downed line, though we were fortunate that it happened at such an odd hour of the night. If we find out that the line went down because it was hit and knocked over by a clumsy driver, we’ll make sure to find and then glare at them on behalf of all of us.
We’re sorry, we’re sorry, we’re sorry! (I can be redundant too!)
Thanks for reading!
Michelle Schroeder
P.S. If you want to look like one of our very own sysadmins, you can get your own set of updated, metallic MC Hammer pants for one payment of only $19.95 from this website. Oh wait, I’m sorry. I put the decimal where I thought it belonged, not where it actually was…these happen to be only slightly more at $1,995.00.
That site will even suggest what to pair with them, though I personally think that this other site does a better job in helping you achieve the whole “MC Hammer” package: http://www.ehow.com/how_2058695_dress-like-mc-hammer.html.
