State grappling with frequent phone problems
SALEM — In the wake of Tuesday's outage phone at state offices in the mid-Willamette Valley, two legislators say they're eager to see state phone functionality improve.
Between early 2016 and late May, state government phone systems have experienced 22 outages of some kind, according to the Office of the State Chief Information Officer. Tuesday's outage, which affected state offices in Salem, Keizer, Woodburn and Dallas, was attributed to a problem with CenturyLink.
"My office lives and dies by the phone," said Sen. Betsy Johnson, D-Scappoose, on Wednesday. "I mean, we pride ourselves on being available to our constituents."
Eight of the 22 outages were associated with IBM, the vendor on a troubled effort to overhaul the state phone system, Project MUSIC. Six were associated with CenturyLink; and eight were associated with power failures and leased network circuits.
An outage doesn't always translate to a complete failure of the phone system, DAS says.
An outage could mean problems with call features such as voicemail and voicemail notification, or lower call quality, a DAS spokeswoman wrote in an email to the EO/Pamplin Capital Bureau on Wednesday.
Tuesday's outage occurred because there was a problem with a piece of hardware from CenturyLink, which provides network services to a slew of state field offices.
After learning of the issues, the Oregon Department of Administrative Services contacted CenturyLink to send a technician to fix the problem. Service was restored Tuesday evening, and DAS has asked CenturyLink to conduct a review of the incident.
Meanwhile, the state is working to remedy issues with Project MUSIC. Problems with the system became evident in January, when the system's "failover," or backup plan, failed after an outage, Liz Craig, a spokeswoman for DAs, wrote in an email Wednesday.
"If IBM's failover had been successful, it would have quickly resolved the outage and restored all voice services and voice-related applications," Craig wrote. "And, this happened after IBM had conducted a failover test in April 2017 and reported to the Office of the State CIO that the test was successful."
That's what prompted the state to issue a notice of default to IBM that month.
The state told IBM to come up with a plan to fix the issues. By early May, 82 percent of the remediation measures IBM said it would undertake had been completed, according to a May 22 presentation by the CIO's office to lawmakers.
Gartner, a third-party contractor, has done a "risk assessment" of the Project MUSIC system, and DAS has already mitigated some of those risks, Nikki Fisher, community outreach and events strategist for Oregon Gov. Kate Brown, wrote in an email Wednesday.
The total budget for Project MUSIC was $45.8 million. Financing costs are expected to set the state back $7 million every two years through mid-2023.
IBM is planning a disaster recovery test later this month, Craig said.
"We're feeling pretty good about how things are running with the project," Craig said Tuesday of Project MUSIC. "We should know more ... after the vendor runs this test later this month."
Sen. Johnson says she wants more, and clearer, information about the Project MUSIC issues.
"It would be so refreshing to have an explanation of the totality of the problems and have it described in a straightforward, unambiguous way that shares the scope of the issue with the legislature," Johnson said.
State Rep. Greg Smith, R-Heppner, the chair of the legislative budget subcommittee on general government issues, said he was concerned as well, but thinks the governor recognizes the importance of remedying the technical problems.
"In all fairness to the State of Oregon, we don't always control internet connectivity or cloud activity, and so unfortunately the Governor's Office has been taking beatings sometimes when acts outside of her control occur," Smith said. "To me, this isn't a very partisan issue."
Smith says that when lawmakers convene for a formal session early next year, there will be more opportunities to discuss the problems with the Governor's Office and state CIO.
"When we get into session, we need to have a collective sit-down and ask what is going on, and how we can correct those outages," Smith said.