Examples of Disaster Recovery

One of my first tasks when I was hired ten years ago was to
investigate the creation of a disaster recovery site for our
mainframe computer systems. I had already had some experiences
with disasters and recovery. Here are some examples of a few
that we included in our plans.

Major Earthquake - Those of us who live in California understand
earthquakes. I've personally been through at least five
significant quakes (6.8 or greater) without suffering any
damage at all. In many people's mind, a major earthquake is
the disaster scenario.

One day the "big one" will come (in California) and who knows
what will happen at that time. In fact, my boss and I were able
to convince the CEO of our company to create a "hot site" (a
duplicate site which is already ready to take over in the event
of a disaster) because of a recent significant earthquake.

One of the first things that we did is contact Caltech (the
experts on earthquakes) to commission a study to determine
where we should place our disaster site. The primary criteria
was that the site be relatively close (within 50 miles) but on
a different geologic plate so the earthquake would not flatten
both locations.

As we studied the possibility of this disaster, we realized
that the building and computers might emerge from the earthquake
entirely intact, but the infrastructure (power, phone lines
and so forth) might be destroyed. In addition, a major
earthquake is a unique disaster because it's more likely that
your people will be in complete shock and more interested in
their families and homes than in restoring your computer
operations.

The thing to do here is be sure you've got the infrastructure
issues covered cold. This includes phones, power and the
network. Make sure you have a disaster site (or very good
backups kept off-site) ready to go. Rehearse your disaster
plan, and make sure your people know what to do.

Minor Earthquake - A minor earthquake might be easy to survive
(we've been through several of them with no issues) and it
might introduce some interesting quirks on it's own. The power
might be out, phone lines might be down and take weeks to
repair, and the general infrastructure (roads, food shipments
and so on) might be disabled. In addition, earthquakes tend to
put people into a state of shock, so it might be difficult to
get people to recover and get back to work.

Biological Event - When envelops fill of anthrax started
appearing on the news, we were suddenly confronted with a new
type of disaster. What if a biological attack or event occurred
in our building? What if the receptionist opened an envelop
contaminated with anthrax? We would then be confronted with a
unique situation. The building would be sealed and off-limited
for days, weeks or even months; and we would not be allowed
back in under any conditions for any reason for that time.

This situation is probably one of the worst disaster scenarios
of all. The old building and equipment is intact but completely
unreachable. Tantalizing and frustrating. Sigh. What to do?

Hope and pray that you've got an excellent disaster recovery
plan and a very competent team, that's what you do. In this
case, you'd better have a hot site or, at the very least some
excellent backups. Not only that, you're business continuity
plan had better be totally finished and rehearsed. You see, in
this case you will not even be able to look through the ruble
or burned building for papers, disks, CDs or anything else.
Nothing, absolutely nothing, will be available for your use
for a long, long time. In fact, if any of your people are in
the building you may find them unavailable (as in quarantined)
as well.

Lightening - One day a few years ago, when I was just beginning
in the computer field, I was working late, way past midnight.
I was just sitting in the computer room (freezing my butt off),
listening to one of the heaviest rain storms that I'd heard in
a long time. It was kind of eerie, listening to the rain drops
on the roof and the thunder in the distance.

Suddenly, the room lit up and I was blinded for a few seconds.
I instinctively pulled back from the keyboard, and I believe
that saved my life. I felt heat on my face and body, and when
I opened my eyes the computer room was dark.

I soon learned that lightening has struck the power pole just
outside the building. The computer simply melted down - no
circuit breaker in existence could have protected it. I was
lucky to be alive - wow, what a rush that was!

Unfortunately, this company did NOT have a disaster recovery
plan. We had to purchase a new computer system and build it
back up from scratch. Fortunately, we did keep backups off-site,
and within a week or so we were back up and running.

And that's the reason why, now, I have a complete belief and
focus on disaster recovery - if you have a plan and have
prepared well, you will recover just fine (even with the
unexpected bumps and such). If you don't, then you basically
leave it up to fate or whatever else you believe in. Personally,
I would rather be in control of the situation. I find that
makes things much easier.