In June 2007, NYSERNet opened its Syracuse-based Business Continuity Center, signaling the culmination of a project that engaged NYSERNet's staff and Board of Directors, technical staff representing NYSERNet's members, and two nationally recognized data center design firms in a two-year quest to enhance the disaster recovery capabilities of NYSERNet's members.
The center's concept originated in a meeting of the NYSERNet Board in Buffalo in June 2005, two months after NYSERNet's latest statewide network came online. Confident the new network, with its aggregate capacity of 320 Gbps, would satisfy the foreseeable networking needs of New York's research and education community and emboldened by the project's success, the Board turned attention to the topics of disaster recovery and business continuity.
The ideal disaster recovery solution requires duplication of mission critical information systems at a location sufficiently removed from campus that a disaster affecting the primary data center spares the backup. Systems so mirrored are available instantaneously should they be called upon, without loss of continuity.
However, while ideal, mirroring is typically expensive and difficult to accomplish. It relies on duplicate computing systems and software, and expensive remote data center and telecommunication services, all of which require remote management. Due to the complexity and cost of mirroring, most NYSERNet members were, until June 2007, relying exclusively on cold-site services and offsite backups as their primary disaster recovery resources.
The discussion in Buffalo focused on the concept of a centrally located data center, one that might leverage the competencies of NYSERNet staff and the capabilities of the new NYSERNet network to satisfy institutional disaster recovery needs and preference for mirror-based business continuity solutions. The Board asked NYSERNet staff to form a working group to explore the concept's feasibility.
A two-year odyssey to design and construct the facility ensued.
The center has been operational for six months now (as of December 2007). NYSERNet's members are using it to support a wide range of applications, from systems that enable critical communications in the event of emergency to duplication of multiple institutional administrative computing systems. Anticipated applications include high-performance research computing and backups using massive network storage devices.
These applications and their demanding requirements were incorporated early on in our thinking on the design of the center. NYSERNet's Board and members of our working group challenged NYSERNet staff to develop a facility that would meet their immediate needs without compromising future utility. The center's feature set and services are a direct consequence of this member-centered approach to design:
- The center is convenient to member institutions but sufficiently remote to satisfy their disaster recovery plan requirements. Located in Central New York State - home of NYSERNet's operational headquarters, a region not historically prone to natural disaster - two major interstates, Hancock International Airport and an Amtrak terminal are minutes away by car.
- The center offers flexible, affordable colocation options. NYSERNet offers full cabinets, half cabinets and one-third cabinets, all of which are lockable. Members may provide their own cabinets, also. Power is available in a variety of flavors, from 120-volt single-feed to 208-volt three-phase dual-feed.
- The center's proximity to a NYSERNet POP offers participants affordable network access via infrastructure that is robust and flexible. Participants can connect to their equipment via GigE, 10 Gigabit Ethernet, Fibre Channel SAN and the NYSERNet R&E network. The center has a total burstable capacity of 1.0 Gb of Tier 1 commercial Internet service.
- The center's critical infrastructure is concurrently maintainable. Multiple distribution paths for cooling and power, with redundant (N+1) capacity, mean that planned maintenance does not require shutting down the data center and inconveniencing systems administrators and users.
- The center is accessible 24/7/365 by authorized personnel, but strictly controlled via a card-key access system. The facility is under 24/7 video surveillance via fourteen strategically placed high definition video cameras.
- Facility intrusion, fire detection and HVAC systems are monitored 24/7/365. An early-warning fire detection system, including smoke and high temperature detectors, backed up by both dry-pipe and FM-200 fire suppression systems provides fire protection.
- A 750 kW diesel generator provides sufficient power to operate the entire facility in the event that utility power is lost. On-site fuel reserves are sufficient to operate the generator for 24 hours at full load. Contracts with multiple fuel suppliers ensure continuous operation via hot refueling.
- Members of NYSERNet's NOC - individuals keenly aware of the unique needs of our member institutions - are available to provide on-site assistance. Participants receive an annual allowance for remote hands support based upon the number of cabinets they purchase. Individuals they know and trust provide the service.
Our initial quest complete, our attention turns to assisting members in the design and implementation of their disaster recovery strategies. As we expected, there is only a little commonality among the solutions they are considering and employing. Two things are certain, however. The first is that our members are blending the center's services in ways we never anticipated, but which the flexibility of our design supports. The second is that the compendium of best practices gleaned from these early adopters will pay dividends to future participants.
[ PDF version ]