Early Friday morning, network services to Burke, the Dartmouth-Hitchcock Medical Center, the Dartmouth Medical School, Fairchild, Steele, Sudikoff and Wilder went down when construction workers behind Baker Library destroyed buried network cables.
Service was restored to all buildings except the DHMC and Sudikoff by that afternoon. The College repaired service to the DHMC and Sudikoff by midnight.
The ruptured cables disabled Nynex and Dartmouth telephone services, Appletalk services and Internet Protocol services. Appletalk services include connections to BlitzMail, said Computing Services System Administrator Stephen Campbell.
"I think this is about the biggest cable break we have had in quite a while," Campbell said. "It was essentially repaired in four hours. The rest was repaired in 12 to 15 hours, which is not bad considering how much damage was done."
At 9:30 a.m., a construction contractor on the steam tunnel being constructed behind Baker Library mistakenly cut a buried conduit that contained telephone circuits, the data network for the campus and other control circuits, Campbell said.
"They are building the steam tunnel and these conduits run through that area," Campbell said. "The conduits were misidentified as abandoned or unused."
"The conduits are encased in concrete when installed to prevent damage from things like a backhoe but in this case, the casing had been exposed and somebody thought they were not in use and thought it was safe to remove them," Campbell added.
Campbell said it was fortunate the cut circuits were not power circuits. These cut circuits were telephone circuits and parts of the data network. They did not use electricity of sufficient voltage to hurt the workers.
Administrators at Kiewit recognized the problem very quickly, Campbell said.
"We realized it because our displays immediately began to show buildings down," he said. "Within 15 minutes, some people from computing services were out there."
Kiewit has network displays that show the status of different parts of campus and our Internet connection, Campbell said. These diagnostic displays continuously inform Kiewit about the status of the network so that if a part of the network suddenly goes down, Kiewit can take appropriate action can be taken.
Several types of cables were cut, Campbell said. The two computer cables cut were coax cables, or coaxial cables, which carry Appletalk services and fiber optic cables. Fiber optic cables connect Dartmouth to the DHMC and carry Internet Protocols, which transfer World Wide Web information and link UNIX workstations to each other.
"Some of the cables were relatively easy to repair," Campbell said. "The ones that took the longest were the fiber optic cables that were some of the ones used in the data networks. Those take special equipment and special training to splice."
Kiewit reconnected Appletalk network in minimal form by 11:00 a.m., he said.
"The Appletalk network had some more redundancy in it," Campbell said. "It was restored but the backup circuits are slow circuits. Although Appletalk was up, it was up in a reduced level until other circuits came up, too."
The coax cables were fixed early in the afternoon, which restored all Appletalk services to the affected buildings, Campbell said.
"BlitzMail was then back to normal which is the thing that is most important to most people," he added.
But network services to the DHMC and Sudikoff remained disrupted.
Because of the difficulty in repairing fiber optic cables, the portion of the network that used Internet Protocols was not repaired until late in the evening, Campbell said.
"This affected people in buildings that used workstations," he said. "Workstations tend to use [Internet Protocols] while Macintosh computers use Appletalk."
In addition to the workstations in Sudikoff and in several science buildings being affected, all connection to DHMC was affected as well.
"The only connection to DHMC is through the fiber optic cables," Campbell said.
The break in the fiber optic cables created a backlog of e-mail that was stored in Kiewit for later delivery to the DHMC and to Sudikoff.
"When the circuits were down, there was electronic mail that could not get through," Campbell said. "That was queued up on servers on Kiewit until the fiber optic cables were repaired."
A specialty company had to be called on an emergency basis to repair the fiber optic cables, Campbell said.
"It takes equipment and techniques that nobody in the College is able to do," he said.
By 10 p.m., workers started to repair the fiber optic cable and finished midnight, after which the displays in Kiewit showed things coming back to normal, Campbell said.
"By morning, things were okay," he said. "At 10 p.m., there were about 2,500 e-mail messages waiting to be delivered to Sudikoff and the Medical Center. By Saturday morning it was all delivered."
"The Medical Center reported a backlog as well," he added. "Outgoing mail was queued up on their side."
The DHMC has its own BlitzMail system so their internal mail was not affected by the cable break. It was only outgoing mail between the College and the Medical Center that was affected, Campbell said.
