Up in flames; down on the Web

Two of our six Web sites were down yesterday. They are hosted by two different companies in Baltimore. The power grid in that area hadn't failed since 1982, but an explosion and underground fire in downtown Baltimore Monday morning caused the grid to fail. Power was cut to approximately 50 city blocks. City Hall was without power. 2400 city and state employees took the day off. The traffic light system went out. And several Internet providers, including AT&T and Qwest, were left without power.

The problem manifested itself in interesting ways. We could surf the Web using Verizon DSL but could not use any of the features of our landmark.org site (now used primarily for testing, maintenance, and privileged features for some clients). We could get to the Postini Web site, but couldn't log into our spam filter. We could send email but couldn't receive it.

So an underground fire 50 miles away created problems for our Web sites -- and thousands of others.

Our primary Web host's data center has a dual power feed from the so-called "redundant grid with three power generators", with a transfer switch between the two feeds. But when the grid itself failed, the additional power feed could not operate. The data center is installing yet another generator to deal with the highly unlikely event of another grid failure.

In a perfect world this kind of problem would not happen. Makes you kind of wonder just where our soft spots are, how many there are, and how we can protect ourselves from random events and purposeful attacks. In the worst case, if the servers had all fried or all of Baltimore had burned down, we could have moved our Web sites to a different host. We would have been down for 24-48 hours, but our Web sites would survive. At least we were backed up.

More information on the Baltimore fire: http://wbal.com/stories/templates/news.asp?articleid=22926