Posts Tagged ‘Earthquakes’

Be Prepared

It was January 2012, not quite a year ago, when The Economist published an article entitled, “The rising cost of catastrophes.” The article contained an important bit of guidance:

For their part companies need to operate on the assumption that a disaster will strike at some point.

In 32 states and the District of Columbia, “some point” happened this year. According to the Federal Emergency Management Agency (FEMA), the number of “Major Disaster Declarations” reached 46 in 2012. And while that is far fewer than the 99 that occurred in 2011, the decline provides no relief for those that were impacted by the disasters that did occur, especially the ten states that declared disasters due to Hurricane Sandy.

The author of the article also wrote:

Disasters are inevitable; their consequences need not be.

For the most part, we agree with the premise. In a pursuit of efficiency, some organizations have over-optimized their infrastructure at the expense of resilience. One data center, for example, can be operated more efficiently than two. But having two data centers provides more resilience, especially if they are geographically separated.

When it is not possible to operate outside of a high-risk area, the infrastructure, itself, can be made more resilient. In Japan, for example, in both buildings and data center racks, the use of shock absorbers is common place. This enables organizations in Tokyo to survive the frequent, moderately-severe earthquakes that occur.

Certainly, we can’t eliminate the risk of every event. Few could survive a magnitude 9.0 earthquake, like the one that occurred near the east coast of Honshu in 2011. But the risk can, just as certainly, be reduced. There are two flawed assumptions that occur all too frequently in organizations. They are:

1. A major disaster will never happen to us.
2. If a major disaster happens, there’s nothing we can do about it.

As we approach the New Year, if we are going assume anything, it should be that we need to assume more personal responsibility.

We wish you all a Happy New Year; one in which you are well prepared for whatever comes your way.

This week I wanted to share with you Dr. DR., a monthly comic on disaster recovery from Rich Parsons.  We hope you like it as much as we do. Please share it with your friends.

Can We Predict the Future?

According to Nate Silver, author of The Signal and The Noise, weather forecasting has dramatically improved over the past 30 years. So why is it that the U.S. National Ocean and Atmospheric Administration (NOAA) could be so wrong in forecasting the 2012 hurricane season? As I wrote in my post, Dodging Bullets in Disaster Recovery, NOAA in May of this year was stating that “Conditions in the atmosphere and the ocean favor a near-normal hurricane season in the Atlantic Basin this season.” According to NOAA, that translates to “12 named storms with six hurricanes, including three major hurricanes.” Instead, assuming no more tropical storms and hurricanes, the U.S. will end the season with 19 named storms and 10 hurricanes, tying with 1887, 1995, 2010, and 2011 as the ‘third most active Atlantic hurricane season in recorded history” according to a Wikipedia post. Whether this is a long-term trend remains to be seen, but we seem, at least for now, to be in a dangerous weather trend.

Trends are not the same as forecasts, and weather forecasts have improved. But, not surprisingly, Mr. Silver reports, “the further out in time these models go, the less accurate they turn out to be.”  Forecasts for a season may not be very reliable, but organizations and individuals should closely attend to near-term forecasts for a specific event. As an example, forecasts for when and where a specific hurricane will make landfall have dramatically improved.  Mr.  Silver wrote, “Just twenty-five years ago, when the National Hurricane Center tried to forecast where a hurricane would hit three days in advance of landfall, it missed by an average of 350 miles.”  ”Today, however, the average miss is only about one hundred miles.”

Even better, predictions can be highly accurate when making a probabilistic estimate over longer periods of time, such as:

• What’s the probability of a magnitude 6.0 earthquake in the eastern United States in the next 100 years?
• What’s the probability that the Lincoln Tunnel will flood again in the next 50 years?

These probabilistic predictions are even further improved, if you look at conditional probabilities such as:

• What’s the probability of a magnitude 6.0 aftershock, within 2 days of a magnitude 7.0 earthquake?
• What’s the probability of the Lincoln Tunnel flooding, if ocean temperatures increase 2 degrees?

These conditional probabilities enable us to evaluate scenarios and to plan and prepare. As organizations, we need to spend more time evaluating scenarios and looking at approaches that will mitigate the impact of dangerous events. I’ll write more on this in a later post, but, in the meantime, I’ll say it once again, “The greater the distance between your primary and disaster recovery data centers, the greater the probability that your organization can survive a catastrophic event.”

Using Your Business Continuity Data Center for Disaster Recovery is Like a Tandem Dive

Too often, business continuity planning and disaster recovery planning are treated as the same functions. Unfortunately, they are not. Business continuity planning  helps organizations insure that applications and processes continue through the myriad of day-to-day disruptions that might occur. These include IT component failures, such as disk-drive failures, a server failure, a dropped network link, or an application bug. Disaster recovery planning helps organizations recover operations after less frequent, but far more devastating events, such as fires, floods, hurricanes, earthquakes, and a variety of man-made disasters. While the data center strategy is only one component of business continuity and disaster recovery planning, it is a key component.  And while business continuity and disaster recovery planning are different functions, they must often be considered together, because of budget limitations.

There are plenty of advantages to having a business continuity data center in region, a very short distance from the production data center. If the data centers are very close, there will be little impact on transaction latency for the always-important two-phase database commit.  Failover times from the production data center to the business continuity data center can be very short. Staff that normally work at the primary data center can easily show up for work at the in-region business continuity data center.  WAN charges between the primary and business continuity data centers will be relatively low.

The  problem with an in-region business continuity data center is that it can’t replace an out-of-region disaster recovery data center. The two are simply too close for comfort. And few organizations can afford three data centers. Following are a few of the types of disasters that can prevent an in-region business continuity data center from acting as a disaster recovery data center:

Natural disasters

• Earthquakes
• Floods
• Hurricanes
• Tornadoes

Man-made disasters

• Electrical-grid failure
• Telecommunications failure
• Transportation systems failure
• Chemical spills
• Radiation leaks
• War, terrorism, and civil unrest

For these types disasters, it is much more likely that both in-region data centers will be affected and much more challenging to recover applications and data. One of the trade-offs organizations must make is between how quickly they recover and how certain they are that they can recover from the range of disasters that could strike them. We believe that a slight increase in recovery time is well worth the additional assurance that you can actually recover applications after a disaster. Using an in-region business continuity data center as a disaster recovery data center is a little like doing a tandem sky dive. It’s fine, as long as nothing goes wrong.

Avoid Locations that Suffer Frequent Natural Disasters

I found a very  good article written by Tom Deaderick called “10 Places You Don’t Want a Data Center.”  Tom is a Director at OnePartner LLC, which provides high-availability colocation services from the company’s data center in the southwest corner of Virginia. Anyone who is on a site-selection team for a new data center or evaluating new colocation providers should read Tom’s article. OnePartner is doing something right. The company reports having no outages in over 1400 days.

Tom’s #2 place you  don’t want a data center is “in a location that suffers from frequent natural disasters.” He includes some useful data on the annual frequency of tornadoes for each state in the United States.  Based on a quick glance at the data, you might think you should never build a data center in Texas.  The state had an average of 139 tornadoes per year between 1950 and 2004. That’s over 7,600 tornadoes in 55 years. Maryland, on the other hand, had only 6 tornadoes per year over the same period. From a tornado-risk perspective, Maryland is obviously much safer, right? Wrong.

You’ve got to be careful with statistics. Texas, as most Americans know, is the second largest state in the U.S., with an area of almost 270,000 square miles. Maryland is #42 and covers only 10,455 square miles. So if you calculate the tornado-rate per square mile, Maryland ranks 8th in annual tornado frequency at 5.74 tornadoes per 10,000 square miles, 10% higher than in Texas, which ranks 11th. For the record, Florida is the state with the highest tornadoes-per-10,000 square-miles rate at 9.37.

Tom offers 10 important factors to consider when locating a data center. Read the article to get the list, because I  don’t want to steal his thunder. But, yes, companies should know the frequency of various types of disasters and obviously avoid known flood plains, airplane take-off and landing paths, and the San Andreas Fault.  I wonder if Tom looked at earthquake risk in Virginia.  Based on data from the last century, they are extremely rare. But, in fact, a significant earthquake  occurred in Virginia in August, 2011.  And there was another, less-severe earthquake in the same area just a few days ago. The epicenters for both the August 2011 earthquake and the July 2012 earthquake were almost 350 miles from Tom’s data center. But a much stronger earthquake occurred in southwest Virginia in 1774. I wonder when southwest Virginia will have its next big earthquake. Despite new earthquake prediction techniques, nobody really knows.

That brings me to my last point. Disasters are, by their nature, simple to track, but very difficult to predict. In designing data centers for maximum up-time and minimal data loss, it’s important to protect your data against disasters that you can’t predict.

A Bad Bet

Tim is the VP of IT. His company’s data center is more than a thousand miles from the nearest ocean, so it’s not going to be impacted by a tsunami or a hurricane. It’s in an area that has very little seismic activity, so it’s not likely to be affected by an earthquake.  There are no active volcanoes nearby. It’s not near any rivers or near a flood plain. There are no other major buildings nearby, and, even though his area has experienced a major drought over the past year, the risk from fire, at least from somewhere outside the data center, is very low.

Tim’s disaster recovery plan calls for a full backup of the application and data files of the critical applications once a week and an incremental backup nightly. The backups usually complete without error, but not always. Some applications are considered more critical than others, so some applications are backed up less frequently. Tim does a disaster recovery test twice a year, to make sure that everything in the DR plan is working. Usually it is. There are other risks to his data center. He could lose power or network communication. He could have a fire that starts inside the data center. His area does occasionally have tornadoes, but not very often. There could be a chemical spill that would require the area to be evacuated, but none of these are very likely.

Like every IT director, Tim has a limited budget, and he is constantly under pressure to keep IT costs low. Tim has made a series of guesses, bets really, in developing his disaster recovery plan. He’s bet that he’s covered for most of the risk associated with natural disasters, he’s bet that the applications that he deemed critical are the right priorities, that he’s got all of the program files and data together in the proper groups, and that nothing has changed since he last revisited the plan. He’s betting that the backup process is working, that the tapes are readable and the applications recoverable.

Those are only some of the bets that Tim has made. Each bet has a consequence. Sometimes he’ll win. Sometimes he’ll lose. But what happens, if Tim guesses wrong?

I’m a fan of the movie, “The Princess Bride.”  If you haven’t seen the movie, click on the link below to see a short clip of what can happen when you guess wrong.

The Princess Bride: The Man in Black in a battle of wits with Vizzini.

Actually, like the movie, the story I just told you is fantasy. Tim is real, but I made up the rest. In reality, Tim made a very different bet. He bet on Axxana. With one very good bet, he avoided making hundreds of bad ones.

Local Floods Have Global Impact

This year, 2011, has been a year of tremendous natural disasters. It began with heavy rainfall in January in Queensland, Australia, and Rio de Janeiro Brazil, causing flooding, landslides, and crop losses. An earthquake in New Zealand followed in February, causing building collapses and an estimated $12 Billion in damages. Japan’s earthquake and tsunami in March resulted in the loss of an estimated 20,000 lives, massive destruction of buildings, loss of power and disruptions to transportation systems, a hurricane in the Eastern United States left 7 million people without power for days and many without power for weeks. Floods in Thailand that began in the summer and continued into December, flooded the capital, killed more than 500 residents and disrupted the lives of millions. In August, an earthquake rocked Virginia and shook buildings as far away as Massachusetts. And a rare October snow storm hit the North Eastern United States, leaving millions without power for days.

In all of this tragedy, there are some important observations:

1. Disasters will strike where they are expected, such as the earthquake in Japan, and where they are not, such as the earthquake in Virginia.
2. Disasters will strike when they are expected, such as hurricanes in the late summer, and when they are not, such as massive snow storms in the fall.
3. Localized disasters, such as the floods in Thailand, can have far-reaching effects, such as the global disruption of the supply chain for disk drives.

The science that enables the prediction of the location, the size and the effect of natural disasters is improving, but it is far from perfect. The local impact of natural disasters is increasing, because people and businesses are migrating into a massive urban areas. The global impact of natural disasters is increasing, because the supply chain is highly specialized into centers of expertise, but at the same time is globally interconnected and interdependent. Because of this specialization, a flood in a relatively small country can impact the global availability and price of products for which the country provides a single, but critical component.

Perhaps the most valuable lesson in all of this tragedy is that a highly efficient global operation that concentrates capabilities into unique centers of  expertise, leaves itself exposed to massive disruption from localized disasters and their impact on infrastructure and the workforce. One of our customers has reduced this risk by creating dual centers of expertise, separated not by hundreds of miles, but by half the globe. With the help of Axxana, these dual centers will operate not only highly efficiently, but 100% in synch. Perhaps it is time to re-think your strategy as well.

Disaster Protection with a Very Pleasant Surprise

There was so much good information in the VansonBourne European Disaster Recovery Survey, that I thought it was worth writing about it again. Twenty-five percent (25%) of the 250 European IT decision makers that VansonBourne interviewed reported having a data loss in the previous twelve months. The causes for the data losses included:

• Hardware Failure
• Data Corruption
• Loss of Power
• Software Failure
• Security Breach
• User Error
• Loss of Backup Power
• Physical Security
• Employee Sabotage
• Natural Disaster

When you think about Axxana, you probably think about protecting data from natural disasters, such as earthquakes, tornadoes, hurricanes, tsunamis, floods, and man-made disasters, such as bombings, explosions, and fires. And that’s exactly what we do. We’re fire-proof, smoke-proof, water-proof, vibration and shock-proof. Throw a javelin at us, and your data will be protected. But these kinds of disasters are near the bottom of the list of causes for data loss, so why do we even talk about it?  Good question.  Because, if you look up the list at all of the other causes of data loss, and you think about what we do, when combined with the snap-shot technology, asynchronous replication technology, and roll-back, roll-forward DVR-like capabilities of our partner, you will realize that we not only protect you from the really bad, but infrequent natural and made-made disasters, but most of the other causes as well. It’s a pretty good deal, and, for some, an unexpected additional benefit.

80% of Your Parts Have Arrived

Imagine you make cars, and 80 percent of your parts come from 20 percent of your suppliers.  The parts are packed in containers and delivered to your manufacturing location on ships. Imagine there was a disaster, like an earthquake. Your biggest suppliers have great contingency plans that ensure a seamless flow of components, so you can make cars. But one of your suppliers, not one of the big 20%, was affected, and couldn’t ship parts for several months.  Oh, well, it’s not that important.  Just apply the 80/20 rule.

The 80/20 rule, which is also known as the Pareto Principle or Juran’s Pareto Principle, doesn’t always work. The rule originated from an analysis of wealth distribution by  Italian economist, Vilfredo Pareto, who estimated that 80% of the wealth in his country was controlled by 20% of the people. Dr. Joseph Juran, who was a pioneer in quality management, applied Pareto’s analysis to quality management challenges, determining that 20% of the factors account for 80% of an outcome. In manufacturing, this might mean that 20% of your suppliers account for 80% of your output potential, so in supply chain disaster preparedness, companies logically place the bulk of their focus on the 20% of companies that supply 80% of the parts. Unfortunately, according to Patrick Brennan, in his article, Lessons Learned from the Japan Earthquake, published this summer in the Disaster Recovery Journal, Lesson 1 was “Don’t Apply the 80/20 Rule to Supply Chain Disaster Preparedness.” The 80/20 rule doesn’t work.

When the lack of availability of a $1 part prevents a company from making a $30,000 product, something needs to change.

Patrick Brennan

The same error occurs when attempting to apply the 80/20 rule to the value of data. While it might be convenient to believe that 20% of your data accounts for 80% of the value, the loss of even a small amount of data, can have an enormous effect on the output of an analytical process or on the reputation of an organization. Imagine, for example, that a disaster destroys the last 3 minutes of data, and one of those pieces of data was an email that provided critical evidence to defend against a shareholder claim, or it was a buy order for fuel in a rising fuel market, or it was a change to a medication order for a critically ill patient.

You can’t always determine in advance, which data will be valuable.  Therefore, it is best to provide complete protection to all data.  If it’s important enough to keep, it’s important enough to protect.  Fortunately, we make complete data protection both possible and affordable.

What is the ROI of a Fire Extinguisher?

There’s a LinkedIn group called BCMIX – Business Continuity Management Information eXchange. There are over 7,000 members of this group, which I think shows just how important Business Continuity Management is in organizations today.  Members can post questions to the community and get advice from other professionals who are struggling with the same issues.  I’m paraphrasing here, but some of the recent topics were:

• Can you develop a profile for what types of individuals are able to manage disasters?
• How do you determine the RTO for critical systems and applications?
• What is the ROI from a Business Continuity Management Program?

In response to that last question, Peter Morris, who is a Business Continuity Coordinator for Debenhams, wrote something I really liked:

I’m always interested in the calculation of an ROI on an intangible such as a BCM program, because the true value of it, like insurance, is not really calculable until after the event. I mean what is the ROI on a fire extinguisher?

There’s really no ROI on a fire extinguisher until you need it, which, hopefully is never.  But, if you do have a fire, you want the fire extinguisher that works well with the type of fire you have.  There are different types of fires and different types of fire extinguishers for each type of fire. There are also combination fire extinguishers that work with more than one type of fire. For those of you who want a quick tutorial on fires and fire extinguishers, here’s a helpful website: Fire Extinguisher: 101.

Once you’ve decided what risks you want to reduce, then you should get the best possible protection at the lowest possible cost. And that’s where the ROI comes in. Our Phoenix System is like a combination fire extinguisher, because we protect data through a wide variety of disasters: floods, fires, earthquakes, bombings, hurricanes, building collapse. But we have something else going for us. We actually lower the cost of data protection, by reducing data communications costs when replicating data over distance.

Maybe there’s no way to determine the return on a Business Continuity Management plan, but once you’ve made the decision to put a plan in place, you might as well have the best possible coverage at the lowest possible cost.  To help you understand the savings that an Axxana Phoenix System investment can provide, we developed an ROI white paper.  I hope you find it helpful.