Approximately 9:00 am on Friday, November 9, the MPW Administrative Building suffered a power outage. The outage was caused due to an error that occurred during the attempted commissioning of a new Uninterruptable Power Supply (UPS) system. The main breaker for the UPS tripped; the UPS serves all of our technology areas. The resultant hard crash of MPW’s technology-based systems caused issues throughout our business, phone and communications systems. Internet, TV, Fiber Phone, MachLink email, our main phone line, as well as our internal business applications were down – our building was eerily quiet aside from the rush of people reporting to our electrical power area to do a damage assessment. After first restoring power to our building, our teams began assessing our servers and equipment. Our first priority was restoring internet service and our main phone line. Delivery of Internet signal and Fiber phone was restored around 9:30 am, although some customers’ connections and equipment took additional time to come back online. Our main business phone line was restored around 1:30 pm.
After Internet service was restored, our next focus was TV service. We determined the hard crash caused a complete failure of the network switch that passes all video signals to both the new IPTV and legacy TV systems. Fortunately, we had new equipment on hand for an upgrade that was planned for later this year. Our team responded to configure and put the new equipment in place as quickly as possible; this was a fantastic effort given that the multiple pieces of equipment were still boxes and had to be configured, mounted and cabled. Restoration of TV services started approximately 3:15 pm with the first networks coming online. We continued restoring networks through the night and all but two networks were on by 11:00 pm. We continued making IPTV video quality corrections and correcting a time issue on the legacy TV guide and DVR boxes into Saturday.
Our team’s resources were then focused on the restoration of MachLink email services. As of 8:00 pm Friday, MachLink email was delivering mail, and later that night most customers were able to send mail.
Our technical teams are lean, therefore, we had to prioritize restoration efforts. We enlisted additional help from other MPW team members, as well as outside resources to work to resolve the issues as quickly as possible. We are very proud of the response of our technical team, many which worked tirelessly for 19 hours. Our HelpDesk and customer support personnel also worked long hours to answer your calls and respond to your voicemails and emails. This was a challenging restoration. It is hard to describe and appreciate the number of issues that a hard crash of technical equipment can cause; fundamentally, that is why these systems are powered by a UPS system.
We know our track record the last couple of weeks may leave doubts in your minds, unfortunately those both occurred partially as a result of projects to ensure ongoing reliable service. We want to reassure you that we will be reviewing the outages of the last two weeks and documenting lessons learned to implement resolutions/processes that will ensure better project execution, ongoing reliability, and optimal restoration processes.
Due to the extended time to fully resolve TV service, I have authorized a credit to our video customers, which you will see on your next utility bill. We very much appreciate the support we’ve received through these events and your patience.
Our pledge to you, as it has always been, is to provide reliable Utility services with outstanding customer service. As we work to finish executing reliability projects and complete the FTTH system, MPW and you, our customer-owners, will have an excellent system that will reliably provide service for years into the future.
If you experience service or reliability issues, please call our HelpDesk. Our legacy system and wiring in many homes is now 20 years old – that is one of the primary drivers in the decision to upgrade the system to Fiber to the Home. Our HelpDesk and field technicians are trained to resolve service and reliability issues. Many times, issues are a result of identifying in-home cabling that has been damaged or chewed into and in need of replacement, fittings that have become loose or damaged, or routers that can’t handle the amount of internet traffic that now travels through your home network. As I mentioned earlier, we’re committed to providing outstanding service; so please call us and give us the opportunity to provide that great experience.
While the focus of this communication addresses the restoration of Internet, TV, Fiber Phone, email services, and our main phone line, many other MPW employees were deployed at the same time restoring business applications we use to conduct our business, and were dispatched throughout the community to substations and wellfields to ensure our loss of multiple business applications and monitoring systems did not further impact our customers’ power and water services.
Rest assured, whether in the Electric, Water, or Communications Utilities, we, your friends and neighbors, work tirelessly “behind the scenes and around the clock” for you.