On Target

A quarterly publication of Society of Workforce Planning Professionals

Saving the Day!  Developing a Disaster Recovery Plan for Your Contact Center

By Maggie Klenke

Disaster recovery is a topic on many of our minds today.  If you do not have a comprehensive, well-tested plan for your contact center in place, now’s the time to get started!

We find that most organizations have a carefully devised and tested plan to address the data processing and IT risks and recovery process.  But less than half of these same organizations have thought about the unique risks involved in the contact center – the front door to the company!  We also find that many plans deal with only the recovery from a catastrophic disaster and do not address the greater likelihood of partial failures, or the prevention techniques that can prevent them.

This article addresses the steps of creating a comprehensive contingency and disaster recovery plan for the contact center with some examples of the types of issues to consider along the way.

Step 1: Assembling the Team

The team you will need to develop and implement a disaster recovery/contingency plan should include representatives from every major discipline in the business.  Obvious members are IT, telecom, and call center management.  And don’t forget to reserve a seat in your meeting for the folks from marketing, purchasing, legal, human resources, and others who influence call volume or depend on the call center. It probably makes sense to put at least one frontline agent on the team as well.  And if you currently use an outsourcer, or plan to contract with one in the event of an emergency, make sure they are included in the planning for situations that will affect both of you.

As your specific plans develop, it will be crucial to bring in your call center technology vendors to help you understand the full capabilities and options of the equipment and systems in place.  It’s important you fully understand how far the systems can be “stretched” in the event of a disaster, and what the costs would be to add backup or duplicate systems to keep you operating in the event of an outage.

Step 2:  Identifying the Risks

A disaster plan needs to start with a focus on prevention.  Every risk must be identified along with what steps would be necessary to prevent it from happening in the first place, since avoiding a problem is more than likely cheaper than fixing it after the fact.

Another set of steps must be in place to minimize the impact when prevention fails.  The plan must define the precise steps to take to recover as quickly as possible.  Various studies show that companies that experience a disaster have a high likelihood of going out of business if a major outage lasts longer than a few days.  A Gartner Group study suggests 2 out of 5 companies that experience a disaster of that magnitude will be out of business in 5 years.  A similar University of Minnesota study suggests only 6% of such companies would survive.  These studies focused primarily on IT and data system problems.  Far less research has been done on what the business impact would be if the contact center as the main customer access point were to experience a significant outage.

What are the risks you face in the contact center?  While terrorist attacks and bomb threats may be the incidents that you think about first in light of the terrorist activities in New York and Washington, they’re actually not the most likely events you need to worry about in your planning effort.  Studies by Contingency Planning Research suggest that power outages/surges and weather are far more likely to cause damage than man-made disasters that are so much in our thoughts since September 11. 

Identifying the potential risks involves taking a comprehensive inventory of all the components of your contact center.  Then begin to “think negatively” about all the things that could fail and what the impact would be if they did. 

Consider each of the following components and what could go wrong:  your physical facility, data systems, call center systems and applications, communications networks, electrical service, customer access points, partners, procedures, and staff. 

As an example, let’s take a look at just one risk in each of these areas:

Facility: It does not take a major hurricane or tornado to cause a weather disaster. Several areas of the country often experience soaking rains and many businesses can be affected.  A simple heavy rain can cause flooding in a localized or widespread area.  What can you do to prevent damage to your facility from a storm?  Can you detect water seeping in the equipment rooms before it is a major factor?  Do you have tarps readily available to cover equipment and flashlights to aid the process that must often take place in the dark? Is the main driving route to your center include a flood plain area or an underpass where a street may be inaccessible? If so, are alternative routes identified for staff?

Data Systems: September 11 wiped out major data centers and huge amounts of end-user equipment.  Do you have a hot standby center with a current set of customer data?  Of course, the more recent event that forced centers to move most if not all staff into their homes required a large quantity of equipment that might not have been needed in the normal office environment.  Can you shift your personnel to laptops or other systems, or do you have a large quantity replacement plan in place with your vendors?  Does your vendor agreement include prioritization when there are multiple requests at the same time?  Writing down the detailed procedures for backup restoration is equally important since the backups may have to be installed by someone unfamiliar with normal operations.

Call Center Systems and Applications:  If your IVR goes down, could you provide a backup quickly or deal with the increased influx of calls to your call center staff? Or in the event of an ACD failure, how would calls be routed to the staff? Is your system capable of routing calls to the planned backup location(s)? How will you deal with the sorting of calls by skills to match up to only the agents trained to handle them?  Keep in mind that what you will be missing during the downtime is not just call routing but valuable management reports.  In any disaster, you will want to get the ACD up and working as soon as possible.  Often ACD reports serve as evidence that is submitted for “loss of business” insurance.  Chat functions and handling web-based interactions is another challenge to be considered.

Communications Networks: What if major storms or a construction mishap leave your building unaffected, but take out local telephone lines or a cellular tower?  What about your Internet connection?  Do you have an alternative service provider or a secondary cable route to back up your primary service?   Some centers have chosen to have cable to a second local telephone central office (CO) in their city in case of cable cut or CO failure.  In most cases, this back up is enough to ensure continued communications.

Note:  Sometimes the backup plan needs a backup.  In the September 11 catastrophe, Verizon lost five of the company’s COs around the World Trade Center.  And companies that chose to back up their wire-line service with cellular saw 14 cell sites vanish from the area.  One company dependent upon web communications had 9 internet providers and lost 8 of them!  A backup that is in close proximity to your main service provider may experience down time too if the cause is weather or highly localized trauma.

Electrical: Lightning is one of the most serious threats to electrical equipment.  But lightning can enter through a variety of pathways.  Have you identified all the paths and put the necessary protection in place?  We have seen thousands of dollars of damage to ACDs from an indirect hit, and a direct hit can destroy it or another call center system completely.  Grounding of equipment is essential to its proper operation during normal periods but may spell the difference between a failure and a crisis averted in another.  Have a qualified electrician ensure that your systems are properly connected to the electrical feed and back-up systems, and make sure the appropriate levels of lightning protection are in place and are grounded thoroughly.  This is probably the lowest cost and most effective preventative measure you can take to guard your expensive equipment.  Be sure batteries are continuously charged and there is fuel available for generators.

Customer Access Points:  The percentage of customer interactions handled over the Internet is growing.  If your web site is down, can you handle the calls that may come into your call center staff as an alternative?   Any point where customers access your business is a potential risk and could “dump” increased workload on the other channels.  But more importantly, can you effectively communicate with your customers who normally use the affected channel to offer them other ways to contact you rather than calling your competitor?

Partners: Do you have a contract with another party, either another call center in the area or an outsourcer, to take overflow contacts?  More and more call center executives are identifying other call centers in their area that could serve as a temporary “home” should their own facility or systems become unavailable.  It is critical to make sure your network provider has an updated call routing plan in place should this event occur, and that the agreements are updated regularly to reflect the growth in both your organizations.  And if your partner center has a disaster of its own, what is your backup option?

Procedures:  Several recent disasters have forced call centers to do things “the old fashioned way” when their computer system was down for an extended period.   Some were able to conduct business by manually taking orders and looking up information in paper-based documentation. Others simply could not function without automated systems in place.  Does your center have a procedure in place for carrying on business as usual if the computers are down?  Does your staff know how to handle manual operations?

Staff:  Have you considered what would happen to service if a large percentage of your staff suddenly became unavailable — something more serious than typical Monday morning absenteeism?  What if a major flu epidemic hit the community that affected half your workforce or their families?  How would you handle the calls? Is there any risk of an employee action such as a walkout?  Do you have a plan to recruit additional staff if there is a sudden surge as well as a training plan that may need to be conducted remotely?

The above list outlines just one risk in each of the categories.  It’s critical that you and your team brainstorm all the negative possibilities in each of the areas, determine what can be done to prevent the bad events from occurring, and formulate a plan of recovery if the worst happens. 

Step 3: Calculating the Costs

Calculating costs will be a big part of developing your plan. It is important that you understand and can attribute a value to each contact so that you can calculate the cost of being out of service.  Even in those centers that do not generate revenue directly, it is important to agree on a value for each answered contact to serve as a base.  Then you will want to calculate the cost of each measure of prevention to see if you can realistically afford each one.  Sometimes the cost of prevention is much higher than what the cost of recovery or lost business would be.  But more than likely, the cost of an ounce of prevention is less than the pound of cure you pay for in the end.

Step 4: Selling and Writing the Plan

Once you have identified all risks and the costs of prevention and cures, it is now time to present the plan to senior management for approval.  Hopefully, you have had representation from senior management on your planning team.  Before you can begin to implement the plan and put in place the equipment, systems, and procedures, you will obviously need approval and budget dollars from the top to make it all happen.

In most cases, call center professionals find themselves with a limited budget and cannot implement everything in the plan at once.  So be sure that you have ranked the risks, options, and preventive measures in their order of importance to carrying on effective operations.

Step 5: Testing the Plan

A plan that has not been tested is a paper tiger.  You must regularly schedule a test and run through various elements of the plan to ensure that the plan is current, workable, and that the measures in it work.  This is painful for the company and requires the backing of senior management, but without it, the planning effort is wasted.  As time passes without any significant crisis, the commitment to the plan tends to waiver.  Regular testing will help prevent this from happening and ensure ongoing buy-in with the process.

Unforeseen Opportunities

Finally, do not forget to look at the bright side!  Contingency plans are not just about planning for negative events.  Any good plan will outline what to do and how to take maximum advantage when an unforeseen opportunity comes your way.  What if your marketing department was unbelievably successful with a new ad campaign, or your competitor suffered a significant problem that gave you a short window to gather in their customers?  Could you react in time with the right resources to make the most of it?

Setting Priorities

Keep in mind the most critical part of any disaster plan:  take care of the people first.  Your staff and customers are your most important assets – make sure they are both well cared for regardless of the circumstance.  Set up a communications plan and keep the information coming.  Quite simply, put people first!

Maggie Klenke co-founded The Call Center School and has written numerous books and articles related to call center and workforce management.A semi-retired industry consultant, Maggie provides training programs and serves as an Educational Advisor for SWPP.She may be reached at Maggie.klenke@mindspring.com.

Saving the Day!  Developing a Disaster Recovery Plan for Your Contact Center

By Maggie Klenke

Disaster recovery is a topic on many of our minds today.  If you do not have a comprehensive, well-tested plan for your contact center in place, now’s the time to get started!

We find that most organizations have a carefully devised and tested plan to address the data processing and IT risks and recovery process.  But less than half of these same organizations have thought about the unique risks involved in the contact center – the front door to the company!  We also find that many plans deal with only the recovery from a catastrophic disaster and do not address the greater likelihood of partial failures, or the prevention techniques that can prevent them.

This article addresses the steps of creating a comprehensive contingency and disaster recovery plan for the contact center with some examples of the types of issues to consider along the way.

Step 1: Assembling the Team

The team you will need to develop and implement a disaster recovery/contingency plan should include representatives from every major discipline in the business.  Obvious members are IT, telecom, and call center management.  And don’t forget to reserve a seat in your meeting for the folks from marketing, purchasing, legal, human resources, and others who influence call volume or depend on the call center. It probably makes sense to put at least one frontline agent on the team as well.  And if you currently use an outsourcer, or plan to contract with one in the event of an emergency, make sure they are included in the planning for situations that will affect both of you.

As your specific plans develop, it will be crucial to bring in your call center technology vendors to help you understand the full capabilities and options of the equipment and systems in place.  It’s important you fully understand how far the systems can be “stretched” in the event of a disaster, and what the costs would be to add backup or duplicate systems to keep you operating in the event of an outage.

Step 2:  Identifying the Risks

A disaster plan needs to start with a focus on prevention.  Every risk must be identified along with what steps would be necessary to prevent it from happening in the first place, since avoiding a problem is more than likely cheaper than fixing it after the fact.

Another set of steps must be in place to minimize the impact when prevention fails.  The plan must define the precise steps to take to recover as quickly as possible.  Various studies show that companies that experience a disaster have a high likelihood of going out of business if a major outage lasts longer than a few days.  A Gartner Group study suggests 2 out of 5 companies that experience a disaster of that magnitude will be out of business in 5 years.  A similar University of Minnesota study suggests only 6% of such companies would survive.  These studies focused primarily on IT and data system problems.  Far less research has been done on what the business impact would be if the contact center as the main customer access point were to experience a significant outage.

What are the risks you face in the contact center?  While terrorist attacks and bomb threats may be the incidents that you think about first in light of the terrorist activities in New York and Washington, they’re actually not the most likely events you need to worry about in your planning effort.  Studies by Contingency Planning Research suggest that power outages/surges and weather are far more likely to cause damage than man-made disasters that are so much in our thoughts since September 11. 

Identifying the potential risks involves taking a comprehensive inventory of all the components of your contact center.  Then begin to “think negatively” about all the things that could fail and what the impact would be if they did. 

Consider each of the following components and what could go wrong:  your physical facility, data systems, call center systems and applications, communications networks, electrical service, customer access points, partners, procedures, and staff. 

As an example, let’s take a look at just one risk in each of these areas:

Facility: It does not take a major hurricane or tornado to cause a weather disaster. Several areas of the country often experience soaking rains and many businesses can be affected.  A simple heavy rain can cause flooding in a localized or widespread area.  What can you do to prevent damage to your facility from a storm?  Can you detect water seeping in the equipment rooms before it is a major factor?  Do you have tarps readily available to cover equipment and flashlights to aid the process that must often take place in the dark? Is the main driving route to your center include a flood plain area or an underpass where a street may be inaccessible? If so, are alternative routes identified for staff?

Data Systems: September 11 wiped out major data centers and huge amounts of end-user equipment.  Do you have a hot standby center with a current set of customer data?  Of course, the more recent event that forced centers to move most if not all staff into their homes required a large quantity of equipment that might not have been needed in the normal office environment.  Can you shift your personnel to laptops or other systems, or do you have a large quantity replacement plan in place with your vendors?  Does your vendor agreement include prioritization when there are multiple requests at the same time?  Writing down the detailed procedures for backup restoration is equally important since the backups may have to be installed by someone unfamiliar with normal operations.

Call Center Systems and Applications:  If your IVR goes down, could you provide a backup quickly or deal with the increased influx of calls to your call center staff? Or in the event of an ACD failure, how would calls be routed to the staff? Is your system capable of routing calls to the planned backup location(s)? How will you deal with the sorting of calls by skills to match up to only the agents trained to handle them?  Keep in mind that what you will be missing during the downtime is not just call routing but valuable management reports.  In any disaster, you will want to get the ACD up and working as soon as possible.  Often ACD reports serve as evidence that is submitted for “loss of business” insurance.  Chat functions and handling web-based interactions is another challenge to be considered.

Communications Networks: What if major storms or a construction mishap leave your building unaffected, but take out local telephone lines or a cellular tower?  What about your Internet connection?  Do you have an alternative service provider or a secondary cable route to back up your primary service?   Some centers have chosen to have cable to a second local telephone central office (CO) in their city in case of cable cut or CO failure.  In most cases, this back up is enough to ensure continued communications.

Note:  Sometimes the backup plan needs a backup.  In the September 11 catastrophe, Verizon lost five of the company’s COs around the World Trade Center.  And companies that chose to back up their wire-line service with cellular saw 14 cell sites vanish from the area.  One company dependent upon web communications had 9 internet providers and lost 8 of them!  A backup that is in close proximity to your main service provider may experience down time too if the cause is weather or highly localized trauma.

Electrical: Lightning is one of the most serious threats to electrical equipment.  But lightning can enter through a variety of pathways.  Have you identified all the paths and put the necessary protection in place?  We have seen thousands of dollars of damage to ACDs from an indirect hit, and a direct hit can destroy it or another call center system completely.  Grounding of equipment is essential to its proper operation during normal periods but may spell the difference between a failure and a crisis averted in another.  Have a qualified electrician ensure that your systems are properly connected to the electrical feed and back-up systems, and make sure the appropriate levels of lightning protection are in place and are grounded thoroughly.  This is probably the lowest cost and most effective preventative measure you can take to guard your expensive equipment.  Be sure batteries are continuously charged and there is fuel available for generators.

Customer Access Points:  The percentage of customer interactions handled over the Internet is growing.  If your web site is down, can you handle the calls that may come into your call center staff as an alternative?   Any point where customers access your business is a potential risk and could “dump” increased workload on the other channels.  But more importantly, can you effectively communicate with your customers who normally use the affected channel to offer them other ways to contact you rather than calling your competitor?

Partners: Do you have a contract with another party, either another call center in the area or an outsourcer, to take overflow contacts?  More and more call center executives are identifying other call centers in their area that could serve as a temporary “home” should their own facility or systems become unavailable.  It is critical to make sure your network provider has an updated call routing plan in place should this event occur, and that the agreements are updated regularly to reflect the growth in both your organizations.  And if your partner center has a disaster of its own, what is your backup option?

Procedures:  Several recent disasters have forced call centers to do things “the old fashioned way” when their computer system was down for an extended period.   Some were able to conduct business by manually taking orders and looking up information in paper-based documentation. Others simply could not function without automated systems in place.  Does your center have a procedure in place for carrying on business as usual if the computers are down?  Does your staff know how to handle manual operations?

Staff:  Have you considered what would happen to service if a large percentage of your staff suddenly became unavailable — something more serious than typical Monday morning absenteeism?  What if a major flu epidemic hit the community that affected half your workforce or their families?  How would you handle the calls? Is there any risk of an employee action such as a walkout?  Do you have a plan to recruit additional staff if there is a sudden surge as well as a training plan that may need to be conducted remotely?

The above list outlines just one risk in each of the categories.  It’s critical that you and your team brainstorm all the negative possibilities in each of the areas, determine what can be done to prevent the bad events from occurring, and formulate a plan of recovery if the worst happens. 

Step 3: Calculating the Costs

Calculating costs will be a big part of developing your plan. It is important that you understand and can attribute a value to each contact so that you can calculate the cost of being out of service.  Even in those centers that do not generate revenue directly, it is important to agree on a value for each answered contact to serve as a base.  Then you will want to calculate the cost of each measure of prevention to see if you can realistically afford each one.  Sometimes the cost of prevention is much higher than what the cost of recovery or lost business would be.  But more than likely, the cost of an ounce of prevention is less than the pound of cure you pay for in the end.

Step 4: Selling and Writing the Plan

Once you have identified all risks and the costs of prevention and cures, it is now time to present the plan to senior management for approval.  Hopefully, you have had representation from senior management on your planning team.  Before you can begin to implement the plan and put in place the equipment, systems, and procedures, you will obviously need approval and budget dollars from the top to make it all happen.

In most cases, call center professionals find themselves with a limited budget and cannot implement everything in the plan at once.  So be sure that you have ranked the risks, options, and preventive measures in their order of importance to carrying on effective operations.

Step 5: Testing the Plan

A plan that has not been tested is a paper tiger.  You must regularly schedule a test and run through various elements of the plan to ensure that the plan is current, workable, and that the measures in it work.  This is painful for the company and requires the backing of senior management, but without it, the planning effort is wasted.  As time passes without any significant crisis, the commitment to the plan tends to waiver.  Regular testing will help prevent this from happening and ensure ongoing buy-in with the process.

Unforeseen Opportunities

Finally, do not forget to look at the bright side!  Contingency plans are not just about planning for negative events.  Any good plan will outline what to do and how to take maximum advantage when an unforeseen opportunity comes your way.  What if your marketing department was unbelievably successful with a new ad campaign, or your competitor suffered a significant problem that gave you a short window to gather in their customers?  Could you react in time with the right resources to make the most of it?

Setting Priorities

Keep in mind the most critical part of any disaster plan:  take care of the people first.  Your staff and customers are your most important assets – make sure they are both well cared for regardless of the circumstance.  Set up a communications plan and keep the information coming.  Quite simply, put people first!

Maggie Klenke co-founded The Call Center School and has written numerous books and articles related to call center and workforce management.A semi-retired industry consultant, Maggie provides training programs and serves as an Educational Advisor for SWPP.  She may be reached at Maggie.klenke@mindspring.com.