Munson Healthcare officials are trying to figure out how to avoid a repeat of a four-plus-hour data systems crash and “resultant chaos” that gripped local hospitals and clinics this week.
A system failure Tuesday morning shut down computers, telephones, pagers and other telecommunications systems at Munson Medical Center and its Munson Healthcare affiliates in Frankfort and Kalkaska, an incident that administrators described as “unacceptable.”
The systems failure at Munson and other medical sites occurred when Traverse City Light & Power crews, as planned and scheduled, severed a fiber optic line during utility work near Hagerty Insurance’s compound in Traverse City.
Munson officials still aren’t sure why a back-up fiber optic circuit failed during a planned outage that started Tuesday at 7:30 a.m., and said their technical staff and outside consultants are trying to determine what happened.
“You can rest assured we’re looking very carefully at that,” Munson Medical Center CEO Kathleen McManus said. “Of course, we need to know what happened.”
McManus said no patients were adversely affected during the outage that wasn’t fully rectified until after noon on Tuesday, a stretch of more than 4.5 hours.
Munson’s data system runs on two high-speed fiber optic circuits connected to Traverse City Light & Power. Light & Power executive director Ed Rice said utility crews cut one of two fiber optic loops that serve Munson as part of a project near Hagerty’s expansion in Old Town.
The other loop was tested before the first was cut, Rice said, and it was expected that the first loop would be enough for Munson’s needs.
Munson knew for several weeks the loop would be cut on Tuesday, Rice said.
“They told us it was OK,” Rice said. “From what I understand, there was a problem on their side.”
Both loops are now back in service, Rice said.
A hospital email memo sent Wednesday afternoon by Chris Podges, vice president of information services for Munson Healthcare, stated that Light & Power’s planned outage left the hospital on one fiber optic circuit.
The remaining circuit “became confused,” the memo said, and crashed the data network, including various computer applications, paging systems and other wireless devices.
McManus called the shutdown frustrating, because for years the hospital operated on one fiber optic line, and installed a back-up line as a safety precaution. The back-up line is routinely tested and there haven’t been problems until Tuesday’s problems, she said.
“That’s what was disappointing about it,” McManus said. “I can’t remember the last time we had unexpected downtime.”
The abrupt shutdown meant that it took about 4.5 hours to fully bring the system back on line. Systems at Paul Oliver Memorial Hospital in Frankfort and Kalkaska Memorial Health Center were restored in about 2.5 hours, McManus said.
Officials at Kalkaska Memorial referred questions to Munson, and several local physicians declined to comment.
Podges’ email said the network crash “had widespread operational and clinical implications” and created “resultant chaos,” and described the incident as “unacceptable.”
McManus said hospital staffers train for unexpected computer shutdowns and used paper records to track patient information and other data. A command center was established and staffers shuttled messages and used the hospital’s traditional intercom system.
Staffers also used personal cell phones to communicate, she said. No surgeries were delayed and McManus said other medical procedures went on as scheduled.
“There were no patients that were impacted because of this,” McManus said. “We have paper back-up for everything; that’s the part we practice routinely.”
The hospital doesn’t have a formal system to inform patients and families of telecommunications breakdowns, McManus said. Nurses inform patients as necessary when written records are being utilized in case of computer problems, she said.
Another outage was scheduled for this morning at 2 a.m. to put the secondary back-up system back on line, and hospital officials expect things to go more smoothly.
“Everything looks good and there shouldn’t be any problem with it,” McManus said.
Staff writer Art Bukowski contributed to this report.