Planning Ahead for Data Disaster
"Data availability is the name of the game anymore. Without the data, you don't have a customer relationship," says Jim Baker, senior product manager for EMC of Hopkinton, Mass., which develops and markets enterprise data storage hardware and software products.
As e-commerce grows, 24/7 operation is becoming the norm, not the exception, according to Roger Peters, managing director for RSM McGladrey, Minneapolis. Even the idea of planned outages for maintenance, upgrades or other reasons is a thing of the past because in an increasingly global economy, a slow time in the U.S. is prime time in Asia, Europe or somewhere else on the globe. So a company's systems need to operate at peak efficiency around the clock.
The redefinition of business also means a redefinition of a business disaster, according to Bob Burns, president and CEO of Evergreen Data Continuity, Newbury, Mass. The list of potential threats isn't limited to natural occurrences like floods and hurricanes, or man-made disasters wrought by arsonists and computer hackers. Today a business disaster--meaning lost sales--can result from anything that cuts a company off from access to its customers or data for even a short period of time.
Interruptions to e-commerce sites cost companies more than $20 billion worldwide last year, according to industry estimates. That amount easily could be surpassed in 2000. In late January alone, hackers shut down eBay, Yahoo and E*Trade, as well as several other Web sites, for 90 minutes to three hours each with "denial of service" attacks, which involved several "hijacked" computers sending multiple e-mails to the attacked sites. The disruptions resulted in millions in lost sales. As a result, Evergreen and other firms that help companies prevent or recover from disaster no longer call their services "disaster recovery," but rather "business continuity."
Businesses providing these services range from long-term disaster recovery firms, like Rosemont, Ill.-based Comdisco to newer companies like Evergreen, with hardware, software and other companies in between.
Similarly, recovery or continuity strategies are becoming more complicated: There are more systems, more third parties and more interdependencies among disparate systems. The systems themselves are also more complex due to the increasing use of multitier client/server architecture and increased network dependence.
By establishing a disaster recovery plan, a company will reduce its potential for service interruption, the financial ramification and any potential security risks and litigation from system down times, while also increasing the recovery time, Peters says.
Evaluating Systems
Know the risks of system outages and determine the ones that are most critical, Peters urges. From there, companies can identify where they get the most effective risk reduction for their investment. The failure of a computer system is the most likely type of disaster, Peters says. Contingency Planning Research studies show that power outages cause nearly 28 percent of all computer-related disasters. That's why hospitals and some other companies have their own generators that are powerful enough to run the enterprise in the event of an outage. But most companies, even those with generators, don't have enough power capacity to run long or to run all systems until traditional power supply is restored.
storm damage (11.7 percent) and flood (9.6 percent) are the other most likely causes of computer-related disasters. "You're more likely to have a problem with a backhoe cutting your telephone line than with a natural disaster," says Dave Palermo, vice president of marketing for the business continuity and Internet division of Sungard, Wayne, Pa.
Customers don't care if the problem is a cut telephone line or a natural disaster, if they don't receive the service they want, they're likely to go elsewhere for their needs, experts agree. "Cybercustomers want what they want when they want it--and they want it now," says Sioban McCarthy, product manager for Comdisco InTouch Services.
Beyond the immediate loss of customer service and potential loss of customer data, a computer failure or other technical disaster could also lead to lawsuits from disgruntled shareholders and customers, Peters cautions.
Peters recommends system redundancy for all systems, including power supplies, processor systems, RAID disks, tape backups and communication systems--including routers, modems and communication lines. But redundant systems in and of themselves aren't enough. For example, Peters says, Northwest Airlines had a backup in place before a systems failure, but the backup and primary system both used the same switch. When the switch failed, so did the backup system.
So the business continuity plan needs to trace the workings of all systems to their origin, whether they are directly under a company's control or not. Companies can choose from a variety of resources to protect their data and to continue their businesses in the event of a disaster. Solutions range from backing up information on-site manually or through automated software programs to remote off-site storage to Internet storage solutions and off-site resources owned by third parties.
On-site backup programs, while the least expensive solution, are also the most susceptible to their own problems, experts agree. If power goes out or if there's a natural disaster, any on-site backups will be subject to the same problems as the primary systems. That's why the backup systems for many Wall street firms are located in nearby New Jersey, where there's a separate power grid, according to Baker. Backup systems that are too far away aren't good for immediate backups, Baker adds, because transmission time prevents full real-time mirroring.
"You still have to factor in the speed of light," Baker says. Even if a backup location is physically close enough for real-time backup, the routing of the communications may not be direct. If the routing distance is too far, the backup can't be in real time.
Mirroring
Data should be mirrored rather than copied, Baker adds, because the data can be re-integrated easily with the primary system. Copies are good up until the time the copy is created, but it's difficult to combine the copy with data collected on backup systems during down times--scheduled or unscheduled.
EMC's Symmetrix remote data facilities mirror data using Internet Protocol and a virtual private network, which is separate from the Internet and from other communications lines, to prevent a failure to one causing a failure to both. Another mirroring product enables companies to have local mirroring of active production volumes that can run simultaneous tasks in parallel with one another.
The company also offers a data manager that features centralized management through an intuitive graphical user interface and is scalable to handle increasing storage needs.
The mirroring technology offers another advantage as well, according to Baker, because the same technology can be used to immediately update a primary site from a remote site once the disaster has passed.
Web Solutions
Internet-enabled recovery planning software provides companies with off-site data backup, security through encryption, fast updates and easy access to help reduce recovery times, according to Peters. SunGard, Comdisco and other companies are starting to provide Internet-based solutions. For example, SunGard's eSourcing provides co-location, Web hosting, Internet access, dedicated or on-demand private network access and outsourcing for production data centers.
Earlier this year, SunGard teamed with GE Access, Boulder, Colo., to offer a comprehensive package of business recovery service via the Internet to accommodate the business recovery and continuity needs of middle market users. GE Access is a distributor of complex computer products, solutions and services and is also Sun Microsystems' largest distributor.
Remote Operations
Because the customer contact center has become such an integral part of customer relationship management, companies need to provide contact center services, even if their contact center goes down, McCarthy says. SunGard and Comdisco are among the companies that offer remote facilities for companies to use in the event of a disaster or for seasonal needs. Companies can choose various levels of service, from full availability and functionality to streamlined services.
Comdisco's new InTouch Continuity Service offers voice, interactive voice response (IVR), Web chat and e-mail support, along with planning and technical expertise to ensure availability for a company's customer interactive center. Comdisco also offers the service as a way for businesses to handle peak period (for example, a retailer for Christmas sales) contact center needs without investing in full facilities that would be idle much of the time.
Comdisco offers its InTouch service in tiers and with a variety of add-on options. For example, companies with basic call center operations can combine a variety of voice services, including PBX and automated call distribution (ACD) support. Companies with more advanced needs can add support for IVR and computer-telephone integration, as well as recovery solutions for e-mail and Web chat environments.
Comdisco provides alternate facilities with the same capabilities as in the customer's "home" facilities. Similarly, Comdisco's Web hosting service offers the same functionality as the client company's own Web page. Most other Web hosting services will provide the viewer with the home page, but not all of the functionality. So a viewer at most Web hosting pages may be able to select items to buy, but not complete the purchase. The Comdisco service includes the functionality for the customer to complete the purchase.
For companies just embarking on e-commerce initiatives or with Web applications that are currently less time-sensitive, Comdisco also is offering its Rapid Recovery and Web Disaster Recovery services, which enable customers to restore their Web presence within a pre-defined time frame using redundant technology, standby operations and alternate sites.
Human resources need to be considered along with the technical systems themselves, Palermo points out. For example, if there's a regional flood, people are going to try to save their homes first and won't be available to help with a company's disaster prevention and recovery. So remote facilities are needed in the event of a regional disaster, Palermo says.
storage Area Networks
By clustering servers and data, storage area networks offer the "five 9s" (up 99.999 percent reliability) that many businesses require, says Jill Kaplan, director of storage solutions marketing for IBM. Distributed servers will give companies 90 to 96 percent up time; centralized servers will provide 96 to 97 percent up time; and clustered servers, 99 to 99.5 percent, according to Kaplan. Clustering the servers reaches the five 9s. IBM works with companies to help configure servers in the most equitable manner.
In a storage area network environment, storage management can be scaled to accommodate changes in server and storage configurations rapidly, and storage system selection is based on storage needs only. The storage area networks have the fast capability required by network users today and in the future. Storage area networks employ high-speed Fibre Channel transmission technology that can reach speeds of more than one gigabit per second--particularly useful for the growing need for real-time transfer of audio and video files.
As fast as storage area networks are today, they had their beginnings in older mainframe computers, which also used centralized storage systems. In 1991, IBM introduced ESCON, which was the industry's first-generation storage area network. Today's storage area networks can be tested at IBM's facilities to make sure they work properly before going into everyday use. Use of multiple paths ensures a disaster doesn't disrupt the business.
ASP Services
Many smaller companies are even more vulnerable than larger firms in the event of a disaster, yet they can't afford the services offered by EMC, IBM and other companies, according to Burns. So Evergreen is offering an ASP service to enable Internet B2B, e-commerce enablers, ISPs and similar small businesses to have a business continuity program.
Evergreen is offering the service on a monthly subscription basis. The plan gives customers, giving customers online access to current information via database links, including the important ability to access online plans remotely. Wireless access will be added soon.
Equipment Replacement
Compaq, which is partnering with Evergreen on some of the company's business recovery projects, enables customers to have all of their equipment replaced in the event of a disaster through the Compaq Recover-All asset replacement program.
Recover-All customers receive an online link to get recovery procedures for their equipment. Though they differ on what the best solution or combination of solutions is, the experts agree that the need for disaster recovery systems will become even more critical in the future.