Everyone wants a highly available data center specifying 9s of availability or Tier levels, but what is harder to find is advice on approaching the availability problem. Part of the problem is there aren't that many good writers who focus on data centers.
So, let's switch over to the Harvard Business Review's senior editor, Diane Coutu and her article on "How Resilience Works." Please read the full article on HBR to get the full set of ideas I am about to share below when applied to the data center availability problem.
The article starts describing a great newsman who had resilience. This description could fit what you want in a data center - one that can endure an environment that at time can be hostile.
a quintessential survivor, someone who had endured in an environment often hostile
So you want your data center to be resilient. What makes something/someone resilient? Diane asked this question and this is what her article answers.
What exactly is that quality of resilience that carries people through life?
...
I have considered both the nature of individual resilience and what makes some organizations as a whole more resilient than others. Why do some people and some companies buckle under pressure? And what makes others bend and ultimately bounce back?
Resilience is so popular that even college graduates are saying they are resilient, but as the author points out resilience comes after you experience the things like in operations like an outage.
Candidates even tell us they’re resilient; they volunteer the information. But frankly, they’re just too young to know that about themselves. Resilience is something you realize you have after the fact.
...
“More than education, more than experience, more than training, a person’s level of resilience will determine who succeeds and who fails. That’s true in the cancer ward, it’s true in the Olympics, and it’s true in the boardroom.”
Surviving stressful conditions like concentration camps can provide insight to the psychology of someone who is resilient.
Looking at Holocaust victims, Maurice Vanderpol, a former president of the Boston Psychoanalytic Society and Institute, found that many of the healthy survivors of concentration camps had what he calls a “plastic shield.” The shield was comprised of several factors, including a sense of humor. Often the humor was black, but nonetheless it provided a critical sense of perspective. Other core characteristics that helped included the ability to form attachments to others and the possession of an inner psychological space that protected the survivors from the intrusions of abusive others.
So let's get to the core principle explained. Three characteristics that resilient people and organizations exhibit.
Most of the resilience theories I encountered in my research make good common sense. But I also observed that almost all the theories overlap in three ways. Resilient people, they posit, possess three characteristics: a staunch acceptance of reality; a deep belief, often buttressed by strongly held values, that life is meaningful; and an uncanny ability to improvise. You can bounce back from hardship with just one or two of these qualities, but you will only be truly resilient with all three. These three characteristics hold true for resilient organizations as well.
Do you see the reality of the situation is the first characteristic. This fits well with monitoring systems and assessments of the current state of operations.
“Do I truly understand—and accept—the reality of my situation? Does my organization?” Those are good questions, particularly because research suggests most people slip into denial as a coping mechanism. Facing reality, really facing it, is grueling work.
...
The fact is, when we truly stare down reality, we prepare ourselves to act in ways that allow us to endure and survive extraordinary hardship.
The second characteristic builds on facing the reality. Your reason, your meaning for what you do. What is your value system?
Strong values infuse an environment with meaning because they offer ways to interpret and shape events.
...
immutable set of values. Businesses that survive also have their creeds, which give them purposes beyond just making money. Strikingly, many companies describe their value systems in religious terms. Pharmaceutical giant Johnson & Johnson, for instance, calls its value system, set out in a document given to every new employee at orientation, the Credo. Parcel company UPS talks constantly about its Noble Purpose.
For those of you who think this is BS and you just hire resilient people consider this point made that values are more important at an organizational level than people.
Values, positive or negative, are actually more important for organizational resilience than having resilient people on the payroll. If resilient employees are all interpreting reality in different ways, their decisions and actions may well conflict, calling into doubt the survival of their organization. And as the weakness of an organization becomes apparent, highly resilient individuals are more likely to jettison the organization than to imperil their own survival.
When there is an outage speed of resolution is critical. Which means you ideally are going to make do with what you have. Placing an order for an item and having it FedEx will not be acceptable.
The third building block of resilience is the ability to make do with whatever is at hand. Psychologists follow the lead of French anthropologist Claude Levi-Strauss in calling this skill bricolage.
I have shared the HBR article with a few other people and they have all enjoyed it. Give it a read. Then read it again. There are some good ideas, well written on what it means to be resilient.