Monday, June 02, 2008

The Planet is on fire!!

From some place called the planet, where they are holding Webby's server hostage:

"This evening (May 31) at 4:55pm CDT in our H1 data center, electrical gear shorted, creating an explosion and fire that knocked down three walls surrounding our electrical equipment room. Thankfully, no one was injured. In addition, no customer servers were damaged or lost. We have just been allowed into the building to physically inspect the damage. Early indications are that the short was in a high-volume wire conduit. We were not allowed to activate our backup generator plan based on instructions from the fire department. This is a significant outage, impacting approximately 9,000 servers and 7,500 customers. All members of our support team are in, and all vendors who supply us with data center equipment are on site. Our initial assessment, although early, points to being able to have some service restored by mid-afternoon on Sunday. Rest assured we are working around the clock. We are in the process of communicating with all affected customers. we are planning to post updates every hour via our forum and in our customer portal. Our interactive voice response system is updating customers as well.There is no impact in any of our other five data centers.I am sorry that this accident has occurred and I apologize for the impact. "

Hmph! "Around the clock" isn't good enuff!!

And last night they write:

"As you have read, we have begun receiving some of the equipment required to start repairs. While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.There is some good news, however. We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity. We will be powering up the air conditioning system and other necessary equipment within the next few hours. Once these systems are tested, we will begin bringing the 6,000 servers online. It will take four to five hours to get them all running.We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems. The call center has also brought in double staff to handle the increase in tickets we're expecting. Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers. The news is not as good, and we were not as lucky. The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary. We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do."

Oh suuuurrre. How much you wanna bet Webby's stuff is on the first floor? :(

(It's actually quite hilarious reading the updates on the situation, because the guy's avatar is his own smiley face, and you just know they're going berserck, running around with cables and wires all tangled around them. *snort*

I'm evil, I know.

And today, after whining about working through the night, they say:

"We now have 90% of servers located on the second floor of H1 online. Support technicians are on location to manually bring the remaining 10% online. "

They say the first floor will be coming up tonight sometime.

I'm not sure we can blame the whole thing on Bill Clinton frying french fries on the stove. "Electrical gear shorted" sounds a lot like a squirrel issue. We get them squirrel issues up here a lot in Canada.


At 11:45 AM, Blogger Puffy said...

And we're baaaaaaack! Just as I have to leave for work, of course, but RTVW is up and running.

At 10:58 PM, Blogger Maroonclown said...

Huh? Webby's not just some guy sitting in his dark basement with a little Microsoft Exchange Server, and a bag of Oreos? :)

At 11:51 AM, Blogger ~Nutz said...


Forum is currently offline for maintainance. Please try again later


Post a Comment

<< Home